Skip to content

Make continuation checkpoints fail closed - #7948

Open
steve8708 wants to merge 25 commits into
mainfrom
steve8708/durable-inbox-checkpoint-20261021
Open

steve8708 wants to merge 25 commits into
mainfrom
steve8708/durable-inbox-checkpoint-20261021

Conversation

@steve8708

Copy link
Copy Markdown
Contributor

Summary

  • Wait for the durable run-boundary checkpoint before emitting auto_continue.
  • Mark checkpoint failures as terminal run errors so they cannot be mistaken for resumable boundaries.
  • Treat a rejected thread projection write as a completion failure, allowing run-manager to retain its projection retry marker.

Verification

  • vitest run src/agent/run-manager.spec.ts src/server/agent-chat-plugin.thread-history.spec.ts (260 tests)
  • pnpm --filter @agent-native/core typecheck
  • pnpm guards (93 checks)

This slice does not change the ordinary busy-message 409 paths. Comparable before/after operational rates are unavailable.

@github-actions

Copy link
Copy Markdown
Contributor

Here's a visual recap of what changed:

Visual recap

Open the full interactive recap

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

@steve8708

Copy link
Copy Markdown
Contributor Author

Fixed in 0ffc9d8: completion hooks run after the projection attempt, and projection failure still propagates so the run retains its retry marker. Added a regression test; core typecheck and 17 focused tests pass.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

@steve8708

Copy link
Copy Markdown
Contributor Author

Disposition of the two review-body findings: both are fixed at this head. A failed checkpoint status lookup suppresses completion callbacks and continuation publication, while finalization still persists the run as errored with error:run_terminal_lookup_failed; the keeps checkpoint status unreadable separate from checkpoint persistence failure spec asserts that write. A failed durable Stop leaves the managed-persistence flag false, so finalization retries the atomic abort write; surfaces durable cleanup failure after stopping the in-memory run verifies the retry, and does not write an aborted status when the atomic abort transaction fails verifies failure stays visible.

This was sent from a bot

builder-io-integration[bot]

This comment was marked as outdated.

@steve8708

Copy link
Copy Markdown
Contributor Author

Review recap for f4aac7d:

  • Both current inline findings are fixed: explicit abort state wins checkpoint status-read failures, and the Netlify scheduler baseline records the first unique deploy-marker response before readiness checks.
  • The body finding about a staged driver terminal hiding the checkpoint lookup failure was valid and is fixed. A focused regression confirms run_terminal_lookup_failed remains visible.
  • Abort event-flush reporting was already handled by flushRunEventsBeforeAbort through reportAbortFlushFailure and an existing focused spec, so no code change was needed.
  • The hot run-event insert now reuses its validated parsed event. A focused test confirms one parse of the input event.
  • Verification passed: run-manager 277/277, run-store 131/131, scheduler helper 37/37, Core typecheck, formatting, and diff checks. The changes are backend and QA-path only, with no browser impact.

This was sent from a bot

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

builder-io-integration[bot]

This comment was marked as outdated.

@steve8708

Copy link
Copy Markdown
Contributor Author

The remaining review-body findings from 5603548581 are addressed in b841979:

  • The processor-unwind regression latches the persisted checkpoint terminal so a late driver done cannot replace auto_continue.
  • Continuation abort cleanup now verifies every durable abort result and throws on false or rejected writes so the existing diagnostics surface failures. Once the predecessor is already terminal, cleanup targets only the successor.

The focused run-manager and continuation suites passed 333/333; core typecheck and all 93 guards passed.

This was sent from a bot

@builder-io-integration builder-io-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builder reviewed your changes and has a few items to flag 🟡

Review Details

Code Review Summary

This incremental update makes a persisted checkpoint terminal authoritative over late driver terminal events: it records the staged checkpoint event and ignores later terminal sends. It also strengthens continuation handoff cleanup by checking abort results and avoiding dispatch after the turn is aborted. The changes preserve the PR’s transactional run-status/event approach. Risk: high, because the PR changes durable run and turn lifecycle state.

The previously open checkpoint-overwrite finding is fixed in the current head and its thread has been resolved. The added regression test covers a late done during driver unwind. Targeted lifecycle, store, and continuation tests passed (369 tests across 3 files). One low-severity issue remains: the new abort helper assumes reason user; if a turn was aborted with a different valid reason, an already-aborted run can return false from markRunAborted, producing a misleading continuation-dispatch failure diagnostic. No new medium- or high-severity issues were confirmed.

✅ Good coverage of the checkpoint/driver race and abort-before-dispatch paths.

🧪 Browser testing: Skipped — no user-facing UI changes.

@steve8708

Copy link
Copy Markdown
Contributor Author

Follow-up for the low-severity review note: continuation abort now handles a run that is already durably aborted for another valid reason. If markRunAborted(..., "user") returns false, it checks isRunAborted before treating the abort as complete. Missing confirmation still fails closed and stops continuation dispatch. Added a regression for the already-aborted case.

Verified locally: continuation suite 55/55, @agent-native/core typecheck, formatting, git diff --check, and all 93 guards passed.

This was sent from a bot

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant