fix: close the downtime Onebox causes on change and deploy

[vishr]

What this changes

Onebox caused downtime it did not need to when a project changed and deployed. This closes those gaps. The design is in docs/plans/2026-10-17-zero-downtime-gaps.md and the implementation plan in docs/superpowers/plans/2026-10-17-zero-downtime-gaps.md.

Managed proxy

• The applied-state marker is now a pair, <config digest> <compose digest>. A pure decision picks one action:
  • unchanged;
  • update changed containers only, no restart;
  • recreate Traefik (its container definition changed);
  • restart Traefik (configuration changed).
• An ob upgrade changes only the discovery image, so it no longer restarts Traefik. Repairing a missing discovery controller or its output no longer restarts Traefik either.
• A failed or interrupted update writes a pending marker, so the retry is another update, not a configuration apply with a restart. A failed replace keeps its own pending marker and stays labelled as a replace.
• The update path pulls the discovery image before stopping discovery. If a later step fails, it starts the old discovery container again. It proves the new controller published by stopping discovery, recording the output's inode, and waiting for a new one. It uploads only compose.yaml, never config/ (which can hold .env credentials).
• A configuration apply renames the live config/ and dynamic/ aside instead of deleting them. Traefik's bind mounts follow the renamed directories, so it keeps its routes until it restarts. The retired directories are removed only after a successful apply.
• Every path pulls the discovery image before stopping or recreating discovery. If a later step fails, it runs docker start onebox-discovery, then compose up -d discovery when that container is gone.
• At the start of a run, staged directories from failed uploads are removed; those holding retired configuration stay until an apply succeeds.
• A running Traefik whose image differs from the local image for its tag counts as a definition change.
• ob status says which half of the proxy drifted.

Deploy plan

• The plan prints a proxy: line with the action above, or start Traefik when Traefik is not running.
• The line is binding. Executing a sealed plan recomputes it after a read-only observation and refuses, as drift, before any host mutation when it differs.

Managed services

• Every managed service gets a 2-minute shutdown window, as stop_grace_period: 2m0s in the rendered document.
• ob service apply passes --timeout 120.
• Destroy runs docker stop -t 120 before docker rm -f.
• PostgreSQL's shutdown checkpoint and Redis/Valkey's shutdown save are no longer killed after 10s.

Rolling

With the managed proxy, a roll retires a replica only after every replacement is healthy and published under each of the workload's proxy services in onebox.yml. That includes newcomers adopted on resume. It stops a replica only after its address is a server of no service in the document. A container that is not on the proxy network counts as withdrawn, so adding a route, removing one, or changing its protocol retires old replicas normally. A failed, empty or unparsable read counts as neither published nor withdrawn. A newcomer that is never published is removed and the old replicas keep serving; a replica that is never withdrawn is not stopped. Every newcomer is re-checked before each retirement and before success.

Recovery

ob abort and rollback of an interrupted deploy now restore the predecessor before removing the interrupted release's serving containers:

1. Remove only the non-serving containers.
2. Rename a running container that holds the transient rollout name to a free slot.
3. Remove the drain file from predecessor replicas the interrupted roll had drained. A lost lock or connection here aborts.
4. Roll the predecessor, which drains and retires the rest.
5. Sweep, verify, then the existing postcondition.

A workload counts as already restored only if none of its containers belongs to the interrupted release.

Docs

• add-a-database.mdx: the shutdown window.
• safety-envelope.mdx: the routing gates, the proxy actions, the 3s converge-buffer assumption, and the one-time upgrade restarts.
• status/capabilities.mdx: the routing gates.

No issue was filed first.

Closes #

Why this is correct

• Every change starts from a test that failed without it. Main ones:
  • Proxy: TestDecideProxy, TestEnsureProxyComposeOnlyChangeSkipsSwapAndRestart, TestEnsureProxyFailedConvergeRetriesAsConverge, TestEnsureProxyFailedReplaceIsPlannedAsReplace, TestEnsureProxyConvergeStopsDiscoveryBeforeSampling, TestEnsureProxyPullsDiscoveryBeforeStopping, TestEnsureProxyRestartsDiscoveryAfterFailedUp, TestEnsureProxyConfigSwapRetiresLiveDirsInsteadOfDeleting, TestTraefikDefinitionChanged.
  • Rolling: TestRollRetiresOldOnlyAfterNewcomerPublished, TestRollRemovesNewcomerNeverPublished, TestRollResumeBlocksAdoptedHealthyUnpublishedNewcomer, TestRetireOldReplicaOffProxyNetworkIsWithdrawn, TestRetireWaitsWhileOldReplicaServesARemovedRoute, TestRetireUnreadableRoutesAreNotWithdrawal.
  • Recovery: TestRecoverySingleReplicaKeepsInterruptedNewcomerServing, TestRecoverySurgesWhenOnlyInterruptedNewcomerServes, TestRestoreReleaseRolesRollsInterruptedOwnedMatchingRevision.
  • Services: TestRenderedServicesCarryStopGrace.
• Two independent reviews ran over the whole branch. Every finding of the first is fixed. The second reverted 16 individual fixes one at a time in a scratch worktree, and a test caught every revert.
• A later /code-review found 10 more items:
  • Fixed 7: discovery restore on the apply path, the binding plan line, re-pulled Traefik images, failed-upload cleanup, single admission and the container name constant. New tests: TestEnsureProxyApplyRestoresDiscoveryWhenRecreateFails, TestEnsureProxyRefusesPlannedActionDrift, TestObserveProxyFlagsRepulledTraefikImage, TestEnsureProxyRemovesFailedUploadsButKeepsRetired, TestRollAdmitsEachNewcomerOnce, TestExecuteBindsSealedProxyAction.
  • A final review then found that admitting each newcomer only once let a roll retire every old replica after an admitted newcomer failed under load. That optimization is withdrawn. Converge and replace now always wait for Traefik health before the final marker. New regressions: TestRollStopsWhenAdmittedNewcomerFailsLater, TestRollKeepsLastHealthyOldWhenNewcomersFail, TestEnsureProxyRetryWaitsForHealthBeforeCommit, TestEnsureProxyProceedsWhenPlannedActionMatches, TestExecuteProceedsWhenSealedProxyActionMatches.
  • Kept 3 by design, recorded in spec §6: a never-effective drain fails the roll rather than stopping a routed replica; the failed-apply limitation below; the recovery slot search.
• No existing assertion was weakened. Where a test encoded the old behavior, it now asserts the new contract. Three tests the review found vacuous were tightened, and the shared fakes now model ownership and routing honestly.
• internal/app/testdata/contract-verdicts.json: only service-document digests moved (the new stop_grace_period).
• The proxy update path relies on Compose not recreating proxy when its dependency discovery is recreated. Verified locally on Docker Compose 5.6.0, both with up -d discovery and a full up -d.
• Ran, each separately (the host was short on memory): go mod tidy -diff, the gofmt check, go vet ./..., go test ./... (2314 passed), golangci-lint run ./... (0 issues), docs-generate-check, env-namespace, and the site build. Together these are what just check runs, plus lint.
• Not run: the Docker e2e suites (just e2e, just server-e2e) and amd64. Also not run: the check that compose stop discovery leaves Traefik running. The local Docker VM had stopped, so that check is listed for the target hosts below.

Effect on the safety envelope

Yes. This changes what Onebox does to a running production system.

• One-time on upgrade:
  • The first deploy or ob proxy apply restarts Traefik once (the legacy one-digest marker takes the configuration path).
  • The first ob service apply recreates each managed service once, to install the shutdown window.
  • The first deploy does so for PostgreSQL services with declared extensions.
• Afterwards:
  • Traefik restarts only on a configuration change and is recreated only when its own definition changes.
  • A roll can now stop with an error instead of proceeding, when discovery does not publish a newcomer or withdraw a drained replica within 30s.
  • Recovery runs in the new order above.
• Known limitation: after a failed proxy apply, a roll that skips the proxy step (ob secrets push, recovery) can read a routing document Traefik is not serving. Run ob proxy apply first; ob status shows the proxy as drifted in that state.
• Before release, on each target host:
  • Record docker compose version.
  • Confirm that up -d of a recreated dependency keeps its depends_on dependent's container.
  • Confirm that compose stop of a dependency leaves the dependent running.
  • Confirm that docker inspect -f '{{.Image}}' of a container equals docker image inspect -f '{{.Id}}' of its tag, and that up -d recreates a container after its tag is re-pulled to a new image.
• /status/capabilities updated.

Checklist

• just check passes locally (its steps run one at a time, as listed above).
• Tests cover the new behaviour, including the failure paths.
• Generated documentation is current (just check verifies this).
• I have accepted the CLA, or will when the bot asks on my first pull request.

https://claude.ai/code/session_01Dm55dV6FgUFE7vnWQEHCB9