A public, timestamped record of how a contract verifier does on real pull requests it has never seen.
Updated 2026-11-02 10:18 UTC. 
| changes | judge not verified | broke the build | caught | missed | false alarms | needs review | |
|---|---|---|---|---|---|---|---|
| Real upgrades, 9 projects' history | 179 | 0 | 24 | 21 | 0 | 1 | 25 |
| Open pull requests, decided live | 288 | 25 | 0 | 0 | 0 | 0 | 68 |
| Stress test, Go: projects never used to tune it | 236 | 2 | 7 | 7 | 0 | 0 | 52 |
| Stress test, Go: hold-out projects, run once | 256 | 0 | 22 | 12 | 0 | 2 | 63 |
| Stress test, Java: projects never used to tune it | 451 | 63 | 10 | 9 | 0 | 0 | 10 |
| Stress test, Java: hold-out projects, run once | 492 | 55 | 0 | 0 | 0 | 0 | 166 |
| Stress test, Java: projects it was tuned on | 84 | 0 | 2 | 2 | 0 | 0 | 21 |
Caught: it broke the build and the verifier said Don't merge first. Missed: the verifier said Merge and the compiler says it breaks, the failure that matters most. Judge not verified: not counted, because a planted break did not fail that project's build, so the compiler's answer proves nothing there.
| contract → project | change | date | where it breaks |
|---|---|---|---|
| cri-api → cri-o | v0.36.3 → v0.37.0 | 2026-09-07 | server/container_create.go:1657 |
| cri-api → containerd | v0.36.4 → v0.37.0 | 2026-08-27 | internal/cri/server/container_create.go:331 |
| cri-api → containerd | v0.36.1 → v0.36.3 | 2026-07-23 | internal/cri/server/container_checkpoint_linux.go:483 |
| lnd → lndclient | v0.20.1-beta → v0.21.0-beta | 2026-06-05 | github.com/lightningnetwork/lnd@v0.20.1-beta/invoices/sql_migration.go:514 |
| cri-api → cri-o | v0.35.1 → v0.36.0-beta.0 | 2026-04-13 | cmd/crio/main.go:408 |
| milvus → milvus-client | v2.6.6-0.20251119054300-fcb3986f4af1 → v2.6.6-0.20251124145901-0b96e4c8af45 | 2025-12-10 | milvusclient/read_options.go:105 |
| lnd → lndclient | v0.19.3-beta → v0.20.0-beta | 2025-11-13 | lightning_client.go:1351 |
| cri-api → cri-o | v0.34.1 → v0.35.0-alpha.1 | 2025-10-16 | vendor/k8s.io/cri-client/pkg/remote_image.go:102 |
| tipb → tidb | v0.0.0-20250513092957-b555ca3fc078 → v0.0.0-20250529123214-bb8180a479ec | 2025-06-04 | pkg/distsql/request_builder.go:100 |
| cri-api → containerd | v0.32.3 → v0.33.0 | 2025-05-27 | internal/cri/instrument/instrumented_service.go:51 |
| lnd → lndclient | v0.18.5-beta → v0.19.0-beta | 2025-05-26 | invoices_client.go:392 |
| cri-api → cri-o | v0.33.0-beta.0.0.20250313010358-ab383b81657e → v0.33.0-beta.0.0.20250324233632-87ee4e17aba6 | 2025-04-08 | cmd/crio/main.go:407 |
| tipb → tidb | v0.0.0-20250321085733-a91a8fafd4ed → v0.0.0-20250331100511-d2c561dad347 | 2025-04-01 | br/pkg/storage/batch.go:55 |
| tipb → tidb | v0.0.0-20241212101007-246f91188357 → v0.0.0-20250321085733-a91a8fafd4ed | 2025-03-23 | pkg/planner/core/plan_to_pb.go:279 |
| dapr → go-sdk | v1.15.0-rc.9 → v1.15.0-rc.17 | 2025-02-28 | client/conversation.go:107 |
| tipb → tidb | v0.0.0-20240823074000-a40c2347786e → v0.0.0-20240919023442-cf70966bef25 | 2024-09-20 | pkg/expression/infer_pushdown.go:240 |
| dapr → go-sdk | v1.14.0-rc.2 → v1.14.0-rc.5 | 2024-07-23 | client/subscribe.go:184 |
| tipb → tidb | v0.0.0-20240305085524-87f5b80908ab → v0.0.0-20240318032315-55a7867ddd50 | 2024-03-18 | pkg/util/execdetails/execdetails.go:755 |
| dapr → go-sdk | v1.12.1-0.20231013174004-b6540a1c464d → v1.12.1-0.20231030205344-441017b888c5 | 2023-11-01 | client/client.go:372 |
| dapr → go-sdk | v1.12.0-rc.4 → v1.12.1-0.20231013174004-b6540a1c464d | 2023-10-13 | client/actor.go:222 |
| tipb → tidb | v0.0.0-20230523034258-1bbc3bbbd369 → v0.0.0-20230602100112-acb7942db1ca | 2023-06-07 | expression/builtin_grouping.go:101 |
Every hour a GitHub Actions job finds the open pull requests that change
a .proto file in 17 open-source projects (Istio, etcd, raft, Temporal,
containerd, cri-api, Envoy's data-plane-api, Linkerd, SPIFFE, Consul, lnd,
Dapr, authzed, Jaeger, Loki, Cortex, Thanos; the pairs are in
corpus.json). For each one it:
- decides the whole pull request (its merge base to its head) with the pinned verifier in bin/: whether the project that uses the contract (Istio for istio/api, etcd for raft, and so on) breaks: Merge, Don't merge or Needs review;
- builds that consumer against the pull request's code with the Go compiler, which says compiles or breaks; the verifier never sees this answer;
- commits both to predictions/ and recounts SCOREBOARD.md.
The record is committed while the pull request is still open, so the prediction cannot be fitted to what happened afterwards. The commit history and the Actions run linked in each record show when it was made.
Open pull requests rarely break anything, so a second workflow, history (run on demand), mines every real upgrade of a contract module in seven projects' git histories: Kubernetes CRI in containerd and cri-o, OpenTelemetry's OTLP in the OpenTelemetry Go SDK, Dapr's runtime API in its Go SDK, tipb in TiDB, lnd's RPC in lndclient, containerd's API in nerdctl (history/suites/). The Go compiler builds each project's code at each upgrade against the old and the new module, so every upgrade that really broke the build is known; the verifier, which was never tuned on these projects, decides each one. Results: HISTORY.md.
- Missed: the verifier said Merge; the compiler says the consumer breaks. The failure that matters most: a customer would have trusted it.
- False alarm: the verifier said Don't merge; the compiler says the consumer builds.
- Needs review: the verifier could not prove either; each row says why.
- Not judged: the compiler could not build the consumer before the change either, or the contract's Go code is not in a module the consumer builds against. Those predictions are graded later by what happened to the pull request (reverts and fix-up commits).
Each record holds the pull request, the commits decided, the verifier's
version and sha256, and the compiler's result. bin/VERSION names the
verifier's source commit; gunzip -c bin/verifier-linux-amd64.gz | sha256sum
must match it.