test: slow-test ratchet and speed rules from measured experiments#1099
Merged
Conversation
… from experiments Measured (2026-07-04, full unit suite: 340 files / 3,210 tests / 48s wall): wall clock was bounded by the slowest FILE (44.6s android monolith at ~7x file-level parallelism), and the slowest tests were sleeping through real production budgets (10.8s proving 'times out' by waiting the constant out, 8s emulator polls at 1Hz, real retry backoff). Two config experiments rejected with data: --no-isolate exploded the suite to 205s (module state thrashes across files sharing workers) and --pool=threads changed nothing. - scripts/vitest-slow-test-reporter.ts: the slow-test ratchet. Unit budget 2.5s / integration 15s; failure at 2x budget (the band between reports without failing so host-load variance cannot make the gate cry wolf); 36 pinned offenders, exact keys, ratchet-only pin (tracking #1098). - waitForAndroidEmulatorByAvdName: poll cadence derives from the caller's budget (min 1s, floor 50ms, ~timeout/20) — devices.test.ts 25.6s -> 2.8s (9x) in isolation, and short-budget production calls stop sampling at 1Hz against small budgets. - vitest.config: slowTestThreshold 500 for local visibility; reporter wired; isolation/pool decisions documented with the measurements. - docs/agents/testing.md 'Speed rules' + AGENTS.md testing bullet: the three conversion patterns in preference order (budget-derived cadence, budget-wiring assertion, fake clocks), the no-seam constraint, and the file-granularity Amdahl argument that makes the monolith test split a wall-clock fix, not just navigation.
Size Report
Startup median (7 runs, lower is better):
Top changed chunks:
|
Member
Author
|
CI is blocked on Fallow Code Quality. The failed job reports two actionable items on this branch:
All other visible checks are green, so this looks like the current blocker before review/ready-for-human. |
…orter, unit tests The string-path reporter wiring read as a dead file (fallow cannot see vitest's reporter loading); the config now imports the factory, making the edge real and type-checked. The class shape tripped the unused-class-members rule (framework callbacks are invisible to reference analysis) — converted to a factory returning the Reporter object, with the classification and rendering logic extracted as pure exported functions. Those functions now carry their own unit tests (budget bands, integration budgets, pin matching, warn-vs-fail rendering), which also grounds the CRAP estimate in real references. Canary re-verified: unpinned 5.2s sleeper fails the run with exit 1; clean runs exit 0.
thymikee
added
the
ready-for-human
Member
Author
|
Review status: no actionable blockers found. I rechecked the latest head after the Fallow fix. The slow-test reporter is now wired through Added ready-for-human. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to #1097's test-topology rules: makes the test-speed strategy experiment-backed and enforced. All numbers measured on this machine, full unit suite (340 files / 3,210 tests).
Findings
--no-isolate--pool=threadswaitForAndroidEmulatorByAvdNamedevices.test.ts25.6s → 2.8s (9×) isolatedChanges
scripts/vitest-slow-test-reporter.ts): unit budget 2.5s / integration 15s, enforcement at 2× budget with a report-only band between — a wall-clock gate that flakes under host load trains people to ignore it, so variance is absorbed by design. 36 pinned offenders (exact keys), pin shrinks-only per the gap-list convention, tracking issue Slow-test ratchet: convert real-time waits to injected budgets #1098 groups them by conversion pattern. Canary-verified: fires with exit 1 on an unpinned sleeper, passes clean otherwise.docs/agents/testing.md"Speed rules" + AGENTS.md bullet): conversion patterns in preference order (budget-derived cadence → budget-wiring assertion → fake clocks), the no-test-only-seams constraint, and the isolation/pool decisions recorded with their measurements so the next person doesn't re-run the failed experiments.The 10.8s worst offender stays pinned deliberately: its conversion (budget-wiring assertion) requires mocking the tool layer, which the apple monolith test file can't do cleanly until the topology split — evidence for doing that split soon.