Repository navigation
feat: standard tags, reporters and retries - #18
Merged
Merged
Conversation
- Register standard markers (p0-p3, positive/negative/boundary, llm, api) and auto-register feature markers so 'pytest -m <feature>' needs no setup - JUnit XML + HTML reports, trace retain-on-failure, screenshot only-on-failure into results/ - Document tag/report/retry conventions in docs/USAGE.md; tag existing tests - CI uploads results/ artifacts
|
✅ All smoke tests passed! View report |
philmcneely
force-pushed
the
feat/tags-reporters-retry
branch
from
October 4, 2026 16:59
5819f48 to
76b19f6
Compare
|
✅ All smoke tests passed! View report |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
BrowserStack and setup failures miss artifacts, while sequential CI runs overwrite earlier browser results.
Review effort: Balanced
Findings: 3
Open (4)
What changed in this PR
Adds standardized pytest tagging, reporting artifacts, retries, and CI artifact uploads.
Changes:
- Registers and applies standard test markers.
- Adds HTML/JUnit reports, screenshots, traces, and retry documentation.
- Uploads generated results from CI jobs.
| File | Description |
|---|---|
conftest.py |
Implements marker discovery and artifact capture. |
pytest.ini |
Configures markers and default reporters. |
docs/USAGE.md |
Documents tags, reports, and retries. |
.github/workflows/smoke_full.yml |
Uploads smoke and full-suite results. |
.gitignore |
Ignores generated results. |
tests/api/test_api_mocking.py |
Adds API/regression markers. |
tests/login/test_login.py |
Adds priority and type markers. |
tests/login/test_login_attacks.py |
Classifies security tests. |
tests/login/test_login_error_conditions.py |
Classifies negative and boundary tests. |
tests/visual_regression/test_visual_regression.py |
Adds visual regression markers. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+231
to
+236
| - name: Upload standard test results (JUnit XML, HTML, traces) | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: results-smoke | ||
| path: results/ |
Comment on lines
+258
to
+260
| def _test_failed(request): | ||
| rep = getattr(request.node, "rep_call", None) | ||
| return bool(rep and rep.failed) |
Comment on lines
+353
to
+355
| tracing_mode = _tracing_mode(request) | ||
| if tracing_mode != "off": | ||
| await context.tracing.start(screenshots=True, snapshots=True, sources=True) |
Comment on lines
+137
to
+138
| | Dependency | `llm` (needs an LLM / AI service) | | ||
| | **Feature** | anything you like: `login`, `cart`, `checkout`, ... | |
|
✅ All chromium login tests passed! View report |
|
✅ All smoke tests passed! View report |
|
✅ All firefox login tests passed! View report |
|
✅ All chromium login tests passed! View report |
|
✅ All chromium login tests passed! View report |
|
✅ All firefox login tests passed! View report |
|
✅ All firefox login tests passed! View report |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Standard test conventions: tag filtering, reporters, retries.
pytest.ini(smoke,regression,p0-p3,positive,negative,boundary,llm,api).conftest.pyalso scans test paths at startup and auto-registers anypytest.mark.<name>, so feature markers (login,cart, ...) work with-m cart/-m "smoke and cart"with no unknown-marker warning.--strict-markersremoved for this reason (trade-off: typos in marker names are no longer errors).results/, git-ignored):--junitxml, pytest-html (self-contained),--tracing=retain-on-failure,--screenshot=only-on-failure,--output=results/artifacts. The framework uses its ownpagefixture (not pytest-playwright's), so the fixture now honors those options itself. Allure unchanged.--reruns N,--reruns-delay,@pytest.mark.flaky). Documented.llmon the AI-healing test,api/visualfeatures).results/as an artifact for smoke and full jobs. Deps (pytest-html, pytest-rerunfailures, pytest-playwright) were already in requirements.docs/USAGE.md.Verification (fresh Python 3.11 venv from requirements)
-m smoke: 4,-m login: 26,-m "smoke and login": 4,-m "login and negative": 23,-m "p0 or p1": 14,-m llm: 1,-m "api and regression": 16 selected; no unknown-marker warnings.tests/api(browser): 16 passed. ruff clean.@pytest.mark.carttest (unregistered):-m cartselected it; failure produced.png+.trace.ziponly for the failing test;--reruns 1reported "1 rerun".Added in follow-up commit
Allure: default
--allurediris nowresults/allure(CLI optional, only to render; CI still passes its own--alluredir). Generation documented.Data setup/teardown (
utils/data_seeding.py, env-driven, no hardcoded creds):cleanup/session_cleanup(LIFO registry, runs all callbacks, reports failures),api_seed(httpx client;SEED_API_BASE_URL/SEED_API_TOKEN; create auto-schedules delete; skips if unset),db_seed(SEED_DB_DSN; sqlite built in, postgres via optional psycopg; no-op if unset). Example testtests/test_data_seeding_example.py(-m seeding, skipped without env); unit tests intests/unit/test_data_seeding.py. Documented under "Data setup & teardown".Verified: 68 passed + 1 skipped (example, no env); against a local stub API the example passed and the server log showed POST -> GET -> DELETE.
CI: chromium/firefox now write per-browser JUnit, HTML and artifact paths (results/junit__.xml etc.), so they no longer overwrite each other. Commits carry no attribution trailers.