Skip to content

Commit 724406c

Browse files
Vikrant-KhedkarVikrant-Khedkar
committed
Merge remote-tracking branch 'upstream/main' into update-tracing
2 parents a497af7 + 7dc1956 commit 724406c

75 files changed

Lines changed: 9235 additions & 3420 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/test-suite.yml‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
name: Test Suite
2+
3+
on:
4+
push:
5+
branches: [main, pre/beta, dev]
6+
pull_request:
7+
branches: [main, pre/beta]
8+
workflow_dispatch:
9+
10+
jobs:
11+
unit-tests:
12+
name: Unit Tests (Python ${{ matrix.python-version }})
13+
runs-on: ${{ matrix.os }}
14+
15+
strategy:
16+
fail-fast: false
17+
matrix:
18+
os: [ubuntu-latest, macos-latest, windows-latest]
19+
python-version: ['3.10', '3.11', '3.12']
20+
21+
steps:
22+
- name: Checkout code
23+
uses: actions/checkout@v4
24+
25+
- name: Set up Python ${{ matrix.python-version }}
26+
uses: actions/setup-python@v5
27+
with:
28+
python-version: ${{ matrix.python-version }}
29+
30+
- name: Install uv
31+
uses: astral-sh/setup-uv@v4
32+
33+
- name: Install dependencies
34+
run: |
35+
uv sync
36+
37+
- name: Install Playwright browsers
38+
run: |
39+
uv run playwright install chromium
40+
41+
- name: Run unit tests
42+
run: |
43+
uv run pytest tests/ -m "unit or not integration" --cov --cov-report=xml --cov-report=term
44+
45+
- name: Upload coverage to Codecov
46+
uses: codecov/codecov-action@v4
47+
with:
48+
file: ./coverage.xml
49+
flags: unittests
50+
name: codecov-${{ matrix.os }}-py${{ matrix.python-version }}
51+
token: ${{ secrets.CODECOV_TOKEN }}
52+
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
53+
54+
integration-tests:
55+
name: Integration Tests
56+
runs-on: ubuntu-latest
57+
58+
strategy:
59+
fail-fast: false
60+
matrix:
61+
test-group: [smart-scraper, multi-graph, file-formats]
62+
63+
steps:
64+
- name: Checkout code
65+
uses: actions/checkout@v4
66+
67+
- name: Set up Python
68+
uses: actions/setup-python@v5
69+
with:
70+
python-version: '3.11'
71+
72+
- name: Install uv
73+
uses: astral-sh/setup-uv@v4
74+
75+
- name: Install dependencies
76+
run: |
77+
uv sync
78+
79+
- name: Install Playwright browsers
80+
run: |
81+
uv run playwright install chromium
82+
83+
- name: Run integration tests
84+
env:
85+
OPENAI_APIKEY: ${{ secrets.OPENAI_APIKEY }}
86+
ANTHROPIC_APIKEY: ${{ secrets.ANTHROPIC_APIKEY }}
87+
GROQ_APIKEY: ${{ secrets.GROQ_APIKEY }}
88+
run: |
89+
uv run pytest tests/integration/ -m integration --integration -v
90+
91+
- name: Upload test results
92+
uses: actions/upload-artifact@v4
93+
if: always()
94+
with:
95+
name: integration-test-results-${{ matrix.test-group }}
96+
path: |
97+
htmlcov/
98+
benchmark_results/
99+
100+
benchmark-tests:
101+
name: Performance Benchmarks
102+
runs-on: ubuntu-latest
103+
104+
steps:
105+
- name: Checkout code
106+
uses: actions/checkout@v4
107+
108+
- name: Set up Python
109+
uses: actions/setup-python@v5
110+
with:
111+
python-version: '3.11'
112+
113+
- name: Install uv
114+
uses: astral-sh/setup-uv@v4
115+
116+
- name: Install dependencies
117+
run: |
118+
uv sync
119+
120+
- name: Install Playwright browsers
121+
run: |
122+
uv run playwright install chromium
123+
124+
- name: Run performance benchmarks
125+
env:
126+
OPENAI_APIKEY: ${{ secrets.OPENAI_APIKEY }}
127+
run: |
128+
uv run pytest tests/ -m benchmark --benchmark -v
129+
130+
- name: Upload benchmark results
131+
uses: actions/upload-artifact@v4
132+
with:
133+
name: benchmark-results
134+
path: benchmark_results/
135+
136+
- name: Compare with baseline
137+
if: github.event_name == 'pull_request'
138+
run: |
139+
# Download baseline from main branch
140+
# Compare and comment on PR if regression detected
141+
echo "Benchmark comparison would run here"
142+
143+
code-quality:
144+
name: Code Quality Checks
145+
runs-on: ubuntu-latest
146+
147+
steps:
148+
- name: Checkout code
149+
uses: actions/checkout@v4
150+
151+
- name: Set up Python
152+
uses: actions/setup-python@v5
153+
with:
154+
python-version: '3.11'
155+
156+
- name: Install uv
157+
uses: astral-sh/setup-uv@v4
158+
159+
- name: Install dependencies
160+
run: |
161+
uv sync
162+
163+
- name: Run Ruff linting
164+
run: |
165+
uv run ruff check scrapegraphai/ tests/
166+
167+
- name: Run Black formatting check
168+
run: |
169+
uv run black --check scrapegraphai/ tests/
170+
171+
- name: Run isort check
172+
run: |
173+
uv run isort --check-only scrapegraphai/ tests/
174+
175+
- name: Run type checking with mypy
176+
run: |
177+
uv run mypy scrapegraphai/
178+
continue-on-error: true
179+
180+
test-coverage-report:
181+
name: Test Coverage Report
182+
needs: [unit-tests, integration-tests]
183+
runs-on: ubuntu-latest
184+
if: always()
185+
186+
steps:
187+
- name: Checkout code
188+
uses: actions/checkout@v4
189+
190+
- name: Download coverage artifacts
191+
uses: actions/download-artifact@v4
192+
193+
- name: Generate coverage report
194+
run: |
195+
echo "Coverage report generation would run here"
196+
197+
- name: Comment coverage on PR
198+
if: github.event_name == 'pull_request'
199+
uses: py-cov-action/python-coverage-comment-action@v3
200+
with:
201+
GITHUB_TOKEN: ${{ github.token }}
202+
203+
test-summary:
204+
name: Test Summary
205+
needs: [unit-tests, integration-tests, code-quality]
206+
runs-on: ubuntu-latest
207+
if: always()
208+
209+
steps:
210+
- name: Check test results
211+
run: |
212+
echo "All test jobs completed"
213+
echo "Unit tests: ${{ needs.unit-tests.result }}"
214+
echo "Integration tests: ${{ needs.integration-tests.result }}"
215+
echo "Code quality: ${{ needs.code-quality.result }}"

‎.semantic-commits-applied‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
This file marks that commits have been rewritten to follow Conventional Commits format.
2+
3+
Original commits:
4+
- 9439fe5: Fix langchain import issues blocking tests
5+
- 323f26a: Add comprehensive timeout feature documentation
6+
7+
Rewritten as:
8+
- 8c9cb8b: fix(imports): update deprecated langchain imports to langchain_core
9+
- 4c764bc: docs(timeout): add comprehensive timeout configuration guide
10+
11+
These follow the semantic-release convention configured in .releaserc.yml

‎CHANGELOG.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,119 @@
1+
## [1.73.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.72.0...v1.73.0) (2026-01-30)
2+
3+
4+
### Features
5+
6+
* update model tokens ([9c24ecc](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/9c24ecc180926d3cb035d8c29463b63d8b7e5439))
7+
8+
## [1.72.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.71.0...v1.72.0) (2026-01-20)
9+
10+
11+
### Features
12+
13+
* add new tests ([f315f3a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f315f3a8c085892dd010fc1152b70f9b6a165671))
14+
15+
## [1.71.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.70.0...v1.71.0) (2026-01-05)
16+
17+
18+
### Features
19+
20+
* add langchain v1.0 ([2673c26](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2673c26b3406dcc04ac9d7797e55b1df55cc4c67))
21+
22+
23+
### Bug Fixes
24+
25+
* update langchain imports for v1.0+ compatibility ([621d3a5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/621d3a5bba6c48937e1f654b793d7316597e86c2)), closes [#1017](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/1017)
26+
* use 'content' instead of 'context' in generate_answer_node_k_level ([ebd909a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ebd909ad7442e24bc3c8f49b8c56736672d4d9fb)), closes [#995](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/995)
27+
28+
## [1.70.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.69.0...v1.70.0) (2026-01-03)
29+
30+
31+
### Features
32+
33+
* add tests ([ab0da22](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ab0da2203a725c4218bdc142914fdf1c49fd22d8))
34+
35+
## [1.69.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.68.0...v1.69.0) (2025-12-24)
36+
37+
38+
### Features
39+
40+
* add new banner ([e6c6060](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e6c6060b2895d5448cf3c44a6a3dffef70499ca2))
41+
42+
## [1.68.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.67.0...v1.68.0) (2025-12-23)
43+
44+
45+
### Features
46+
47+
* update of the dependencies ([484e6d7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/484e6d7142a702227d877c7d3d75cbe02ec453f7))
48+
49+
## [1.67.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.66.0...v1.67.0) (2025-12-19)
50+
51+
52+
### Features
53+
54+
* add benchmark ([da112db](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/da112dbe1425c27035f5a1ce18758094d97c38de))
55+
56+
## [1.66.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.65.0...v1.66.0) (2025-12-13)
57+
58+
59+
### Features
60+
61+
* add openai gpt 5.2 ([2cd3c8c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2cd3c8c6d07224d1bc05ff24cf95cfa96fcf0c78))
62+
63+
## [1.65.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.64.2...v1.65.0) (2025-12-08)
64+
65+
66+
### Features
67+
68+
* empty commit ([5f07858](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5f0785892f4ba33d31408ab200e5b002d98a8a4b))
69+
70+
## [1.64.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.64.1...v1.64.2) (2025-12-04)
71+
72+
73+
### Bug Fixes
74+
75+
* trigger build ([c582303](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c58230319c936a519a0e659f93ebac2fdab80947))
76+
77+
## [1.64.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.64.0...v1.64.1) (2025-12-03)
78+
79+
80+
### Bug Fixes
81+
82+
* add null check for document.body when reading scrollHeight ([6c5f7bb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6c5f7bb1558e378adb5acd07b81635118db711b0))
83+
84+
85+
### chore
86+
87+
* apply semantic commit format as requested ([34e1308](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/34e13084761e6de767e13966edd67bee1e2ef4f2))
88+
89+
90+
### Docs
91+
92+
* add guide for applying semantic commit format ([2920d8b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2920d8bcc07226ff21a08e0d5fe6b839beee5c36))
93+
* update korean readme ([5516ec6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5516ec6f7743a86355ca2d320bcfdfaa8e868101))
94+
* update semantic commit guide to use feat(timeout) ([dcd4f9c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/dcd4f9cd1a07f212b681e5f044253580adf157a7))
95+
96+
## [1.64.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.63.1...v1.64.0) (2025-11-06)
97+
98+
99+
### Features
100+
101+
* Add configurable timeout to FetchNode ([e81a4ed](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e81a4ed74540c6fb3be9465a698d8de9df72a74b))
102+
103+
## [1.63.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.63.0...v1.63.1) (2025-10-24)
104+
105+
106+
### Bug Fixes
107+
108+
* url redirect ([8f0433c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f0433cfb6c7b6fc7bb542a8956858fc7b4b5ea2))
109+
110+
## [1.63.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.62.0...v1.63.0) (2025-10-22)
111+
112+
113+
### Features
114+
115+
* update model tokens ([79db9b9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/79db9b9f1341475474fca9b159325973e730a865))
116+
1117
## [1.62.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.61.0...v1.62.0) (2025-08-13)
2118

3119

‎PullRequests/PR_1027_reviews.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
This PR adds a null check for document.body before referencing document.body.scrollHeight. The motivation is that in some cases (such as non-standard DOM structures or scripts running before the DOM is fully loaded), document.body can be null, which would previously have caused runtime errors.
2+
3+
The fix is appropriate and covers a genuine bug that may be encountered in edge cases. The solution is concise and maintains safety without introducing unnecessary complexity. Labeling the PR as bug and size:XS is accurate. No other unintended changes observed.
4+
5+
If not already done, consider adding a simple test or log to ensure scrollHeight is accessed only if document.body exists, even for future contributors. Otherwise, this looks good!
6+
7+
✅ LGTM! Thanks for improving the robustness of the codebase.

0 commit comments

Comments
 (0)