[weiddeng]

Stacked on #130, which now includes #131; I'll retarget it once #130 merges. Refs #129.

A benchmark score counts only if the harness worked. Phase 0 showed how it can fail silently: a verifier that couldn't install its tools reported 0 with no error.

Changes

• integrations/harbor/audit.py checks a Harbor job's trials. It flags any trial that:

  • Harbor didn't finish;
  • had a task image that failed to build;
  • where S-Code never got a model response;
  • was cancelled;
  • where S-Code's cleanup failed;
  • ran commands without the task container's environment (provenance's host_command_env isn't container; see Add a pinned Harbor adapter for S-Code #130);
  • got less than full credit from a pytest verifier with no test summary. Other verifiers keep their reward with a warning, unless their output is empty or shows a known setup failure.

  Run it before using a run's scores: python3 integrations/harbor/audit.py <job-dir>. It exits with status 1 when it flags a trial, and --json prints machine-readable results.

• When S-Code stops at its own time limit, the adapter now raises TimeoutError. Harbor records that as AgentTimeoutError instead of a generic agent error, still runs the verifier, and doesn't retry the trial by default.

Testing

• New audit tests (8) cover genuine results, false zeros, non-pytest verifiers, infrastructure failures, unfinished trials, a model that never answered, cleanup failures, and commands without the container's environment. They run in the benchmarks CI job.
• Adapter tests pass (27), including the new timeout mapping. Bundle (11) and CI-plan (19) tests also pass.
• Across all 22 Phase 0 and fixture trials from before Add a pinned Harbor adapter for S-Code #130's fix, the audit still flags the 9 invalid ones for their original reasons: verifier setup failures, failed image builds, a cancelled trial, unreachable or rejecting models, and unfinished trials. The other 13 are now flagged only for the command environment. Fixture runs on the rebuilt bundle, as root and non-root, pass.

[ylzhang2447]

Makes sense to me. Validating it on the Phase 0 jobs (flags all 7 invalid, passes all 6 genuine) is convincing, and it's defensive about missing or malformed files. One small question: it detects whether tests ran by matching a pytest-style summary, so is pytest the verifier for all tasks? If yes, looks good.

[weiddeng]

No. All 89 Terminal-Bench 2 tasks and 97 of 100 TBLite tasks use pytest. Three TBLite tasks don't: okhttp-trailers-crash runs a Gradle test, and protein-sequence and smiles-data-lab run a Python checker. A genuine failure on one of them would have been flagged and dropped from the score.

Fixed in 64f519a: the audit reads each task's tests/test.sh (from the task folder in the trial's config.json) and requires the pytest summary only if that script runs pytest. Other verifiers keep their reward, with a warning, unless their output is empty or shows a known setup failure. An unreadable script is flagged as before. The Phase 0 results don't change (the same 7 trials are flagged), and all 38 Terminus-2 baseline trials that scored below 1 printed a pytest summary. Could you take another look and approve?

[ylzhang2447]

Thanks for checking the non-pytest tasks. Reading tests/test.sh and only requiring the pytest summary when the script actually runs pytest addresses it well, and keeping the other verifiers' reward with a warning is the right call. LGTM.

[ylzhang2447]

The test.sh check (requiring the pytest summary only when the task's script runs pytest, and keeping other verifiers' reward with a warning) addresses the non-pytest tasks. LGTM.

The base branch was changed.

[ylzhang2447]

Re-reviewed after the new commits. The pytest-script check and the new host_command_env == "container" check both look right and each has a test. LGTM. (CI red looks like it came from the main merge, not this PR's changes.)

[weiddeng]

Added 84429f9, the dependency bump from #134 (webpki-roots 1.0.10, rustls-webpki 0.103.16, vite 8.1.6), so the advisory gates pass. No other change.

[ylzhang2447]

Re-reviewed after the retarget to main. The audit logic, timeout mapping, pytest-summary fix, and host_command_env check all read correctly and have tests; CI is green. LGTM.