Skip to content

[quality] e2e coverage report records the revision but not the runtime its region counts depend on #1747

Closed

Description

@hivecommons-hive

Finding

The e2e-coverage report is the artifact everyone reads its region percentages
out of — it is written to $GITHUB_STEP_SUMMARY on every run and uploaded as
e2e-coverage with retention-days: 90. It stamps revision, so a reader can
always tell which code the line numbers address.

It does not stamp the runtime, and for this report the runtime is the other half
of the provenance. The region counts are a V8 artifact: the same commit, suite
and fixture builds split into a different number of ranges on a different
runtime, so the denominator moves while the code does not. The repository
already knows this and says so in two places:

  • CONTRIBUTING.md warns that a local render on a newer runtime "reports
    headroom the gate does not have", and measures it — DirectoryFreshness.js at
    11/15 (73.33%) on the pinned major vs 9/10 (90.00%) on v26.10.0.
  • tests/e2e-coverage-gate.test.mjs carries MEASURED_ON_NODE_MAJOR = '22' and
    fails if the job moves off it.

Both of those guard the job. Neither reaches the artifact. So a report read
out of the 90-day upload months later, or pasted into a pull request from a
local render, carries percentages with nothing in them to say which runtime
chose the denominator — and the two are indistinguishable.

Evidence

A full local render on this repository at 3bbac0d (init → build:e2e:coverage
→ test:e2e:coverage, 362 specs passed → seal → report:e2e:coverage, exactly
the recipe in CONTRIBUTING.md) reports:

E2E coverage: ok (run local-1; status passed)
Revision: 3bbac0da081df644da541a2fbde6b9908bae4abf
src files | 100.00 | 97.05 | 657/677 regions

That 97.05% was produced on Node v26.12.0, not the pinned major 22. Nothing in
the report says so. Read beside a CI artifact it looks like a 0.24-point
regression against the 97.29% CONTRIBUTING.md records for the pinned major;
it is runtime drift. This is precisely the mistake the prose exists to prevent,
and the artifact gives a reader no way to avoid it.

To be clear about what this is not: there is no coverage gap behind it. Unit
src coverage is 100.00% lines and 100.00% regions (10892/10892, 2995/2995),
and every one of the 20 residual e2e regions is the documented cross-script
region-union artifact, not a missing test.

Recommendation

Record the runtime in the report envelope beside revision, and render it on
its own line so the step summary and the .txt artifact carry it too:

  • add runtime: { node: process.version, v8: process.versions.v8 } to both the
    success envelope (emptyReport) and the failure envelope in main's
    catch — a tooling error that only reproduces on one runtime is the case
    where knowing it matters most, and that envelope is the only record such a
    run leaves;
  • render Runtime: node <v> (v8 <v>) beneath Revision:, with the same
    unknown fallback revision uses so the line is always present;
  • v8 and not just node, because V8 is what splits the ranges and it moves
    between patch releases of one Node major — which is the difference
    CONTRIBUTING.md warns is "runtime drift, not a code change".

Additive and backward compatible: no existing key changes meaning, no threshold
moves, and no workflow edit is needed — the e2e-coverage job already tees the
rendered text into its summary and uploads the JSON.

  • runtime present on the success envelope
  • runtime present on the failure envelope
  • Runtime: line in the text render, with an unknown fallback
  • tests covering all three

Priority

  • Impact: medium
  • Effort: low

🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5n31 | SHA: 3bbac0d

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

Activity

  1. added
    testingApproved by a Hive merger/owner for auto-merge on green CI
    agent/qualityApproved by a Hive merger/owner for auto-merge on green CI
    stage/010-actionableTriaged; eligible for the hive to pick up
    and removed on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/qualityApproved by a Hive merger/owner for auto-merge on green CIstage/010-actionableTriaged; eligible for the hive to pick uptestingApproved by a Hive merger/owner for auto-merge on green CI

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions