Repository navigation
feat: migrate Phase 6A messaging to NATS JetStream - #31
Merged
Merged
Conversation
Review current queue, event and deployment behavior and propose a fenced delivery design, implementation plan and local/dev/prod deployment runbook. Document drained cutover, rollback, recovery and compatibility qualification. Refresh roadmap and operations links, and include the owner's AGENTS.md engineering workflow instructions.
…n slots Prepare deliveries through exact acquisition and settle only committed run outcomes. Keep independent thread-owned resources reserved through cancellation and close reusable Worker checkpointer and engine contexts explicitly.
nhtua
changed the title
nhtua
marked this pull request as draft
October 26, 2026 22:51
Add validated local and aggregate capacities, immutable durable topology checks, and broker-confirmed pause/resume primitives with explicit partial outcomes. Preserve operator pauses and pending ACK progress during repeated bootstrap. Qualify pinned broker pause semantics and runtime bind/fetch/ACK permissions while denying shared topology mutation and pause operations to that role. Keep production entry points and database gates unchanged.
Use qualified broker create-only semantics when a missing durable is provisioned, then reread and compare the winner on an existing-resource conflict. Preserve concurrent operator pauses, configuration and ACK state. Reject unqualified server versions before creation and cover paused/drifted races against the pinned broker, plus context routing and error propagation.
Require an active transport and open execution gate under the shared state lock. Preserve distinct hold outcomes so closed-gate deliveries keep progress without acquiring a run fence or burning delivery attempts. Add SQLite and PostgreSQL regressions for inactive/closed gates, stale state observations, reopening, and fixture isolation. Record the bounded Phase 3B2 execution slices in the implementation plan.
Keep an unresolved command disposition intact when a duplicate delivery is held by a closed gate, so the original owner can settle its committed result. A new held delivery still records a deferred gate outcome without authorizing ACK. Cover claimed and deferred owner evidence, both gate reasons, successful settlement, and unadmitted holds in SQLite and PostgreSQL.
Record the reviewed B2b completion and carry the settlement-proof check into the bounded run-pull/ACK slice. Production entry remains on the legacy worker.
Reserve one execution slot before each bounded pull from the reviewed run-v1 consumer. Keep broker progress independent from worker execution and ACK only after PostgreSQL settlement or a confirmed durable handoff. Preserve admission proof for exact settlement and serialize concurrent checkpointer setup across worker processes. Keep the production polling entry unchanged for this slice.
Require the current running owner, fence, and lease before a broker-owned run continues. Flush an installation-scoped control subscription before checking ownership, wake the supervisor on notices, and interrupt it when its execution slot is cancelled. Cover the ownership and cleanup boundaries with unit and pinned PostgreSQL/JetStream tests, and record the reviewed 3B2c and 3B2d slices in the implementation plan.
Persist MaxDeliver advisories and live-owner observations before settling an exhausted run source. Requeue only safe unstarted work, verify the exact JetStream source before deletion, and keep uncertain observations recoverable. Require the advisory observer before worker intake and document that pinned broker qualification plus full-capacity reconciliation remain production gates.
Grant the relay qualification role read-only access to the exhaustion durable, and assert that pausing execution consumers leaves the advisory observer active. Update PostgreSQL migration rollback expectations to the new Alembic head. Keep the broker acceptance gate open until the pinned matrix rerun passes.
Keep PostgreSQL authoritative for event replay and use scoped ephemeral consumers for per-process live notices. Recover active browser streams from snapshots and reconnect after broker loss; limit API NATS permissions to the consumer creation and inbox subjects it needs.
Alert on advisory stream capacity and add a bounded, gated operator reconciliation path for run sources whose MaxDeliver advisory was lost. Record capacity-scan provenance without inventing delivery counts, and publish SSE event notices as authorized event references only.
Use the disposable stream in bounded scan tests, publish ordered event notices in successive relay batches, and update PostgreSQL migration expectations to the new Alembic head.
Match the pinned next_by_subj inclusive sequence behavior and preserve bounded scan resumption without duplicate source candidates.
Route the existing worker module command through the bounded durable consumer while retaining CLI identity and graceful drain behavior. Keep legacy polling confined to the explicit synthetic test harness until full-stack broker qualification.
Ensure a delayed JetStream reference for an already delivered event neither duplicates replay nor blocks a later event. Record the Task 3 qualification and Task 4 regression status.
Pass the stale-event regression in both fanout tests and the full PR CI matrix. Correct existing Playwright conversation-row locators and stale expectations so the 66-test browser suite runs cleanly. Record Task 4 completion evidence in the migration plan.
Inventory Phase 5 and Phase 6A run, job, event, and outbox totals without modifying data or exposing payloads. Block known duplicate-job conflicts, bound reported IDs, and update the migration status and runbook.
Add resumable queue cutover and rollback controls, protected NATS setup, and explicit local, Compose development, and production deployment paths. Document the migration runbook and retain native and Compose reload workflows.
Capture the current Alembic revision before guarded downgrade attempts so PostgreSQL tests remain valid when new revisions are added.
Ignore generations already advanced before a persisted rollback cursor and validate only queued runs still awaiting a bump. Cover rollback batch bounds, stale blockers, epoch retries, and clarify rollback epoch handling in the runbook.
nhtua
marked this pull request as ready for review
October 27, 2026 14:44
Add stable startup cutover previews, explicit dev confirmation, resume and verification flows, and a production manual-start guard. Document the operator sequence and dev stack teardown while carrying the approved Phase 6A deployment, NATS NUI, artifact setup, and client responsiveness changes for PR 31.
Require the operator to pass the Cutover ID printed by the read-only dev preview. Reject stale or mismatched IDs before stopping app services, and update the startup instructions to show the two-command flow.
Have the dev startup wrapper create missing protected NATS credentials and initialize or verify the mounted artifact volume before application services start. Preserve existing credentials and storage identity, and document the first-run flow.
Enable Vite polling for the container reload stack, where host bind mounts may not forward filesystem events. Keep native Vite watching unchanged and document how to select the reload stack.
Use source-mounted Vite and Uvicorn services for every dev Compose start, with polling watchers for shared filesystems. Keep built Nginx in production and update runbook endpoints and restart guidance.
Add Terra and Sol provider profiles to both environment examples, align managed Connector operator and keyring settings across native and Compose setups, and correct the dev stack's Vite port override.
Respect explicit Alembic database configurations and align the Compose port assertions with the development stack.
Keep each test module together and run the complete unit suite across three jobs per Python version to stay within the CI timeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 6A migrates queue delivery from PostgreSQL polling to NATS JetStream while preserving Phase 1–5 behavior. PostgreSQL remains authoritative for admitted work, ownership fences, dispatch state, events, and results.
Implemented
Verification
481eee1,b10eb4e, and920b6bd: root tests, API Ruff, full offline API unit suite, Vue typecheck, frontend unit tests, and staged Rayloc scan.git diff --checkpassed.b10eb4eare green.Remaining qualification
Review references
Phase 6B sandbox execution remains separate. Deployment targets for this work are native processes and single-host Docker Compose; Kubernetes and ECS are out of scope.