Skip to content

fix(ci): fail tests-ok when workspace discovery fails - #19

Draft
MakerFlyFly wants to merge 4 commits into
mainfrom
fix/tests-ok-members-failure
Draft

MakerFlyFly wants to merge 4 commits into
mainfrom
fix/tests-ok-members-failure

Conversation

@MakerFlyFly

@MakerFlyFly MakerFlyFly commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Require successful workspace member discovery before tests-ok may pass. A skipped pytest matrix additionally requires discovery output to be exactly [], supplied through the step environment.
  • Exercise the actual gate shell with exact exit-code and diagnostic assertions. Probe Bash using the real flags and the BATCHES environment input so a launcher that starts but drops the gate input is skipped explicitly.
  • Keep the bounded field reader and its syntax contract. Remove redundant synthetic binding cases and replace probe-argument assertions with a real-shell environment-loss regression.

Type of change

  • Python package (tools/dev/ regression tests)
  • CI / dev loop (the original workflow gate and explanatory comments)

Fixed revision and scope

Current head: a301d0df4b4b2d524285a87ef4e83bc55c14f0be.
Base: upstream deb35662.

The original two commits remain intact. Two test-only commits were appended after reviewed 2c0beae4: the environment-probe revision and a minimal control-flow clarification required by the first real CodeQL result. Relative to 2c0beae4, one test file changes by +15/-19 lines, retaining mode 100644. Workflow logic, selector, dependencies and the required check name are unchanged by these follow-ups. The complete PR still includes the original workflow gate and .asf.yaml comment changes.

Latest review disposition

Item Disposition Result and evidence boundary
A: Bash may start without forwarding gate input Accepted Probe [ "${BATCHES:-}" = "[]" ] with BATCHES=[] in the child environment. The real-shell regression removes that variable and requires an explicit skip. Restoring the old start-only probe makes exactly this regression fail. No global environment or WSL configuration is changed. An actual Linux WSL distribution was not tested.
B: Test-helper coverage is excessive Partially accepted Remove the three synthetic expression cases. Replace the argument/timeout test with a behavioral environment-loss regression. Keep the actual workflow binding check, parser regressions preventing previously observed false rejections, and unavailable-launcher cases. The module has 47 cases and remains smaller than the reviewed version.
C: Hosted execution is missing Accepted; verified The authenticated Actions UI revealed fork-wide execution suspension despite enabled/active API metadata. With explicit user authorization, execution was restored while token permissions and the manually disabled workflow stayed unchanged. Both current-head hosted paths below pass.
CodeQL finding from the first hosted run Fixed and reverified On 3987c7ad, the analysis workflow succeeded but its independent aggregation check failed with “Potentially uninitialized local variable” at probe.returncode. Move that check into the try-else branch, where assignment is explicit, without relying on CodeQL's model of pytest.skip(). Existing exception/skip behavior remains covered. The aggregation check on current head now succeeds with “No new alerts” and zero annotations. No finding was dismissed and no CodeQL configuration was weakened.

Original review disposition

Item Disposition Verification and resulting change
1 Partially accepted The first fix accepts members=success, pytest=skipped with nonempty or missing batches. Require canonical [] for a skipped matrix. The proposed cancellation timing remains unverified; GitHub's cancellation reference does not establish that exact outcome.
2 Accepted Default PowerShell reproduced 2 failed / 15 passed with the original module because bash resolved to the unusable WSL launcher. The fixture now probes the real arguments, with a five-second timeout, and skips missing, failed, or timed-out launchers.
3 Accepted All 14 negative cases in that original WSL run passed without executing the shell. The revised cases assert exact status and the diagnostic for discovery, invalid selection, matrix failure, or success.
4 Accepted Inline comments, quoted lists, block lists, and expression-wrapped always() were falsely rejected. A bounded standard-library field reader now accepts those forms; its docstring states the unsupported YAML features. Incorrect dependencies, conditions, and literal/misdirected BATCHES bindings still fail.
5 Covered by item 1 Checking the actual output makes the gate independent of why pytest was skipped. No additional assertion freezes the pytest job's condition text.
6 Accepted .asf.yaml now describes successful discovery plus a passing matrix or confirmed empty selection. Only the explanatory comment changes.
7 Accepted Actual selector calls include .sh and .json changes and exclude .md / plugin manifests. The three old Python-only comments now defer to eligible files, linked sources, dependents, and shared-input selection.

Test plan

Current local checks

Check Result
Gate module, Git Bash with Windows Python 47 passed, zero skipped
Gate module, native PowerShell 19 passed, 28 explicit skips because the WSL launcher probe exits 1
Gate module, Linux 47 passed, zero skipped
Complete dev suite, Linux 335 passed
Dev Ruff / formatting / mypy, Linux Passed; 44 files formatted, 34 source files type-checked
Whole-tree non-test hooks Passed, including the offline whole-tree lychee check
Affected workspace pytest hook Passed with the revised test file staged
Real commit hook Passed without bypasses; Generated-by: attribution retained

The PowerShell skips are not executed gate cases. Whole-dev Windows support is not claimed: the earlier full runs on 3987c7ad reported 39 failed / 265 passed / 1 skipped / 30 errors under Git Bash, and 39 failed / 237 passed / 29 skipped / 30 errors under native PowerShell. An isolated native replay of 2c0beae4 had the same 69 FAILED/ERROR node IDs. These include missing Windows symlink privileges and existing environment/tree assumptions outside this module. The gate module was rerun on the final helper revision in both Windows shells; no new whole-dev Windows replay is implied.

Reverse checks against the final 47-case module

Workflow/probe variant Failed Passed Skipped
Original deb35662 workflow 24 23 0
First fix d0e14d18 workflow 8 39 0
Remove discovery-result check 15 32 0
Remove confirmed-empty batches check 7 40 0
Restore old start-only Bash probe 1 46 0

These are local fault-injection results, not hosted cancellation experiments.

Hosted verification on the final revision

The matrix contains optional skips in other workspace members; success is not a claim that every workspace test executed. The documentation-only comparison and actual job results establish the empty-selection case without relying on local shell simulation.

Temporary Draft PR #20 was closed without merging after verification. Its branch and run records are retained. It was aligned to the final base by an appended merge, preserving its earlier history and keeping the comparison documentation-only.

Fork workflow execution was restored with explicit authorization. Default token permissions remain read-only, review-approval permission stays disabled, and the manually disabled npm-pin workflow stays disabled. This PR remains Draft; upstream submission and merge are separate actions.

Historical evidence

The previous 50-case module and 338-pass Linux dev suite belong to 2c0beae4. The first hosted matrix/empty runs on 3987c7ad / a76589f4 passed, but that head's independent CodeQL aggregation check exposed the control-flow diagnostic above. Those records are retained as historical evidence and are not retagged as final-head results.

RFC-AI-0004 compliance

  • HITL / write-access discipline — the user explicitly authorized the revisions, fork execution recovery and temporary validation branch. This PR stays Draft for human audit. The temporary PR was submitted by the user through the required web flow.

Linked issues

Closes #18.

Notes for reviewers

The BATCHES contract matches the producer's existing literal []; no JSON parser or package dependency is added. Public APIs and the required check name are unchanged. Raw logs, mutation copies and local harnesses stay outside the worktree; no public download location is implied.

Generative AI disclosure

AI-generated with Codex (GPT-6). Human review is pending. This draft stays in the contributor's fork before any separately authorized upstream submission.

Require successful member discovery before accepting a successful or
skipped pytest matrix. Exercise the workflow's actual Bash assertion
across all job results and preserve empty-selection behavior.

Generated-by: Codex (GPT-6)
@MakerFlyFly MakerFlyFly added family:ci .github workflows, prek, validators capability:fix Implement code change to resolve an issue labels Oct 24, 2026 — with ChatGPT Codex Connector
Require the discovery output to be [] before accepting a skipped matrix.
Probe Bash before executing gate tests, check failure diagnostics, and
accept equivalent literal dependency declarations without new dependencies.
Align the workflow and branch-protection comments with the selection rules.

Generated-by: Codex (GPT-6)
A Bash launcher can start without forwarding the environment needed by
the tests-ok gate. Probe the real BATCHES input path and reject a shell
that drops it, while keeping unavailable-launcher skips explicit.

Replace implementation-detail assertions with a real-shell environment
loss regression and remove synthetic bindings already checked against
the workflow. Gate logic and workflow configuration are unchanged.

Generated-by: Codex (GPT-6)
Comment thread tools/dev/tests/test_ci_tests_gate.py Fixed
Keep return-code inspection in the try-else branch, where the probe is
known to have completed. This makes assignment explicit for CodeQL
without relying on its model of pytest.skip raising an exception.

Launcher failures still skip the gate tests with the same diagnostics.
The workflow gate, modelled outcomes and environment contract are unchanged.

Generated-by: Codex (GPT-6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:fix Implement code change to resolve an issue family:ci .github workflows, prek, validators family:tools substrate:framework-dev

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests-ok passes when workspace member discovery fails

2 participants