Skip to content

ci(invoice): separate scheduled evaluation from PR checks - #1895

Merged
mldangelo-oai merged 4 commits into
mainfrom
mdangelo/codex/ci-invoice-evaluation
Oct 24, 2026
Merged

mldangelo-oai merged 4 commits into
mainfrom
mdangelo/codex/ci-invoice-evaluation

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Invoice Desk currently launches inference orchestration for routine PR updates even though it evaluates the sample with a pinned published CLI. Run that evaluation manually or on a weekly main baseline, and keep deterministic PR checks focused on changes to the sample and its helpers.

Changes

  • Remove the PR-target dispatcher and completed-workflow fallback, including their API lookups, dispatch permission, and obsolete security-lint exemptions.
  • Preserve manual exact-head PR scans and add a weekly protected-main baseline on Mondays at 03:17 UTC. Apply cancellation only after target validation, and group by both target and exact SHA. Duplicate evaluations of the same revision coalesce; delayed older requests cannot cancel a newer revision.
  • Run dependency-free behavior checks only for sample/helper/workflow changes, skip title/body edits, and cancel superseded behavior jobs.
  • Update the sample and GitHub Actions guides to distinguish published-CLI evaluation from candidate package/runtime validation, and document manual baseline and exact-head PR evaluation steps.

Testing

Review target: 0177c7523049398cdcb5772f0ed108e4a2562c35, based on main b3448fc0ad5faeec6bdb1e96b1f99828cce6ac7b. Three fresh native reviews and an independent verifier passed on this exact head with no findings.

  • Node 22.13.0: node --test .github/scripts/invoice-desk-target.test.mjs .github/scripts/invoice-desk-source.test.mjs — 16 passed on the final repair. The preceding workflow/documentation repair also passed all 36 tests when including examples/invoice-desk/qa/app.test.mjs.
  • Bun 1.3.14, from sdk/typescript: bun test --timeout 30000 tests-ts/invoice-desk-workflow.test.ts — 10 passed, including the actual workflow's validated-target and exact-revision cancellation regression. A comparison against the previous workflow confirms different revisions no longer share a group.
  • Bun 1.3.14, from sdk/typescript: bun test --timeout 30000 tests-ts/workflow-quality.test.ts — 18 passed before the final exact-revision group adjustment.
  • actionlint 1.7.12 on both Invoice workflows — passed.
  • zizmor 1.30.1, offline with the repository configuration, on both Invoice workflows — no findings at medium severity or above.
  • Prettier and whitespace checks passed for the three-file final repair; all seven local Markdown links in the affected sample guide resolve. The preceding guide repair checked all 21 local targets across the three edited guides.
  • The earlier implementation also passed the 36 Node tests on Node 24.0.0. Hosted source CI passed on the final 0177c7523049398cdcb5772f0ed108e4a2562c35 head: 49 successful jobs and one intentional skip on attempt 1, including the cancellation and guide repairs.
  • Combined validation of the original nine changes passed on b4741b89f26666a5375899f2c2be6a068bef3465, based on main b3448fc0ad5faeec6bdb1e96b1f99828cce6ac7b: node-ci passed all 64 jobs and Action CI passed all three jobs, both on attempt 1.
  • The literal SDK package test commands passed on the earlier combined snapshot 4bbbb61a8948445661db7a30f24299aae9deb2d8, based on 5c8a8d662fb5f207fd5dc1ff3bc9197e6e8d629b. Fixed seed 12345 and recorded random seed 1267795626 each produced 6,996 passes, 83 skips, zero failures, and 101,231 assertions. These full-suite results and the later combined hosted graph are separate source snapshots.
  • No live inference was run; credentialed evaluation still runs only from protected main under the existing environment policy.

Risk and rollout

Automatic inference on each PR update is intentionally removed. Maintainers can explicitly evaluate an open PR by supplying its exact current head SHA, and weekly baselines evaluate main. This is not a required merge check; no branch-protection settings or public CLI interfaces change.

Protected-main execution, environment approval, exact-source checks before and after approval, committed-blob extraction, removal of the answer key before inference, and read-only scan cache access are preserved. The changed triggers and schedule take effect after this workflow reaches main. Existing environment approval/wait policies still apply to scheduled runs. Different revisions intentionally retain separate approval requests; stale pending requests can be cancelled manually and skip inference after approval if their PR revision is no longer eligible. Hosted approval and duplicate cancellation remain operational validation items after deployment.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/ci-invoice-evaluation branch from cd8f8c9 to fffac24 Compare October 23, 2026 18:33

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for separating published-CLI evaluation from routine PR checks. No blocking findings in this owner-level merge-safety and simplification review. Manual exact-head scans and weekly main baselines retain the protected workflow, environment approval, source rechecks, and credential boundaries. One nonblocking documentation suggestion is inline.

At this rebased head, all 36 sample/helper tests and all nine mocked workflow tests passed, along with actionlint/ShellCheck, zizmor, formatting, and diff checks. The six-file change and its orchestration/test inputs are unchanged from the previous review. No credentialed inference, live scheduling, or environment-approval transition was run; those behaviors activate from protected main.

Comment thread examples/invoice-desk/README.md Resolved

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating the baseline instructions. This owner-level merge-safety review found no issues: the README now directs maintainers to the supported manual workflow on main with the PR number and exact head SHA, resolving the earlier suggestion.

Only the README changed. Formatting, whitespace and all seven local links passed; the workflow name and input names match the implementation. Prior runtime/workflow checks remain applicable to identical source and dependency inputs and were not repeated. No live inference or dispatch was run.

@mldangelo-oai
mldangelo-oai marked this pull request as ready for review October 23, 2026 19:47
@mldangelo-oai mldangelo-oai added the skip-release-notes Omit internal changes from generated release notes label Oct 23, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T21:07:48.234680Z 0177c75 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-23T21:08:51.535650Z 0177c75 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e2613660a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/invoice-desk-scan.yml Resolved
Comment thread .github/workflows/invoice-desk-scan.yml Resolved
@mldangelo-oai
mldangelo-oai marked this pull request as draft October 23, 2026 20:11
@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/ci-invoice-evaluation branch from e261366 to 0177c75 Compare October 23, 2026 20:26

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tightening cancellation and updating the guides. I found no material issues at 0177c752: rejected requests cannot reach the scan concurrency group, and different source revisions remain isolated while protected-main and post-approval eligibility checks stay intact.

Fresh validation passed: 36 sample/helper cases on Node 22.13.0, 28 mocked workflow/quality cases on Bun 1.3.14, SDK/MCP type and generation checks, actionlint/ShellCheck, configured zizmor, formatting, and 21 local documentation links.

No live inference, workflow dispatch, scheduled run, or hosted approval/cancellation transition was exercised. Broad runtime/native suites were not run, and hosted CI was still in progress.

@mldangelo-oai
mldangelo-oai marked this pull request as ready for review October 23, 2026 21:06
@mldangelo-oai mldangelo-oai reopened this Oct 23, 2026
@mldangelo-oai mldangelo-oai reopened this Oct 24, 2026
@mldangelo-oai
mldangelo-oai merged commit 6f9490d into main Oct 24, 2026
190 checks passed
@mldangelo-oai
mldangelo-oai deleted the mdangelo/codex/ci-invoice-evaluation branch October 24, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-release-notes Omit internal changes from generated release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants