Skip to content

test: split API coverage across balanced CI shards - #1908

Merged
mldangelo-oai merged 10 commits into
mainfrom
mdangelo/codex/ci-api-test-shards
Oct 24, 2026
Merged

mldangelo-oai merged 10 commits into
mainfrom
mdangelo/codex/ci-api-test-shards

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A Windows API shard took 817 seconds, with almost all its work in api.test.ts, limiting what additional file-based shards could improve. Separately, a compound persistence test exceeded the existing 30-second macOS deadline. Windows retains its existing 120-second timeout.

This PR splits the API tests along their existing semantic boundaries and gives the persistence/cache scenarios independent test cases. It is stacked on #1902; this parent-relative change touches only API tests and their duration metadata.

Changes

  • Keep orchestration in api.test.ts; move finding validation to api-validation.test.ts and durable workflow resume to api-workflow.test.ts. Each file owns its temporary directories and cleanup. Existing mock restoration and subprocess isolation stay intact.
  • Initialize the per-OS weights from measured parent CI testcase sums. The original Windows groups measured 376.7, 368.2, and 70.3 seconds; the two large groups now land on separate expanded shards.
  • Give recovery/failure/retry/target checks, model/access-program cache keys, and safety-identifier cache keys independent persistence cases. Preserve cross-client persistence, warmed-client model changes, and first→second→unset factory environments. Codex creation/responses remain mocked; source snapshots and review persistence use real workbench subprocesses, while get-scan remains synthetic. Independent baselines increase validation calls from 15 to 20 and assertions from 22 to 27.
  • Preserve every original behavior and add two cases/five assertions overall: Linux/macOS inventory changes from 344 to 346 API cases; Windows changes from 341 to 343. Runtime versions, shard counts, coverage policy, retries, and timeouts are unchanged.

Testing

Current head 58bf29d270fb382e23fa0e7b24ac0f41345f0a0a is stacked on routing 087c0c9607919c9d7dd9f332780f587b31c24e27, based on main 8fd19efadb160153a15e0ad47d8edf32391ce463.

  • Fresh plugin and SDK builds, SDK/MCP types, and formatting passed. The regenerated plugin includes the new three-part MCP bundle and receipt module from main.
  • 38 shard/timing tests, 10 selector/gate tests, 18 package-skeleton tests, the three persistence cases, and one completed-scan bundle-loading case passed: 70 focused cases, including 492 Bun assertions. The persistence cases used the unchanged 30-second timeout and completed in 6.62, 13.77 and 13.28 seconds.
  • Three fresh independent code reviews and a separate verification pass completed without findings. Source comparison preserves the 154 unchanged registrations and accounts for every transition in the decomposed case. Each test file is assigned exactly once per OS.
  • Hosted source CI passed all 63 jobs on attempt 1. Tested merge 38f4039a has the head's source tree. All 26 SDK report/log/upload bindings were verified; all 236 shared files ran once per OS. API inventory remains 346 cases/3,542 assertions on Linux and macOS, and 343/3,499 on Windows. Every whole-suite inventory/assertion change is source-accounted, with no lost positive Linux coverage on unchanged inputs.

Risk and rollout

Test organization and scheduling change; this parent-relative patch does not change package behavior. The persistence cases repeat independent setup while retaining real persistence checks. Fresh code review, scoped checks and hosted qualification of this head are complete. Land after #1902. No runtime version, timeout, retry, coverage policy, or public API changes are included.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@mldangelo-oai mldangelo-oai added the skip-release-notes Omit internal changes from generated release notes label Oct 23, 2026

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused split. This owner-level merge-safety review found no blocking issues: the moved coverage and isolation are preserved, and the persistence cases retain the important cache and client transitions.

Types, formatting, and 52 shard/report tests passed. Hosted reports from the exact reviewed tree pass all three API files on Linux, macOS, and Windows; the API inventory changes only by the intended persistence split. Local API execution was limited by environment ownership checks. All current GitHub checks are successful or skipped.

@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/ci-select-required-checks branch from dc98c63 to f49e592 Compare October 23, 2026 21:47
@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/ci-api-test-shards branch from d0088ec to 4f5b34f Compare October 23, 2026 21:53

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused split. This owner-level merge-safety review found no blocking issues: the 154 unaffected test registrations are preserved, and the three persistence cases retain cross-client recovery, warmed-client model changes, access-program cache separation, and safety-identifier transitions.

Fresh types, formatting, and 52 shard/report tests passed. The exact reviewed tree also passed all 90 Linux validation cases, including all three persistence scenarios (27 assertions). Local API execution was limited by fixture-ownership checks. The remaining hosted matrix is still running; this approval does not imply those checks have completed.

@mldangelo-oai
mldangelo-oai marked this pull request as ready for review October 23, 2026 22:18
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T23:13:19.821214Z 58bf29d Draft marked ready
🔒 Security Review ✅ Completed 2026-10-23T23:13:26.165131Z 58bf29d Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@mldangelo-oai
mldangelo-oai marked this pull request as draft October 23, 2026 22:25
@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/ci-select-required-checks branch from f49e592 to 087c0c9 Compare October 23, 2026 22:39
@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/ci-api-test-shards branch from 4f5b34f to 58bf29d Compare October 23, 2026 22:41

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused split. This owner-level merge-safety review found no blocking issues: the 154 unaffected registrations are preserved, and the persistence cases retain cross-client recovery, warmed-client model changes, access-program separation, and safety-identifier transitions.

Fresh SDK/plugin builds, types, formatting, and 79 focused tests passed. Local persistence and completed-bundle checks stopped at the review environment’s existing fixture-ownership guard; those checks are not claimed as passing. The current hosted matrix is still running. The exact reviewed tree also passed all 12 Windows workflow cases in hosted CI.

@mldangelo-oai
mldangelo-oai marked this pull request as ready for review October 23, 2026 23:11
Base automatically changed from mdangelo/codex/ci-select-required-checks to main October 24, 2026 01:58
@mldangelo-oai
mldangelo-oai merged commit 92993e5 into main Oct 24, 2026
125 checks passed
@mldangelo-oai
mldangelo-oai deleted the mdangelo/codex/ci-api-test-shards branch October 24, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-release-notes Omit internal changes from generated release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants