Repository navigation
Keep a failed Studio restart from reading as an operator stop - #95
Open
mattmacrocket wants to merge 1 commit into
Open
mattmacrocket wants to merge 1 commit into
mattmacrocket wants to merge 1 commit into
Conversation
A production install showed the Studio Build Mac "unavailable" at every sensor start with no failure in the log. Its rows had been saved as stopped weeks earlier, and the startup path treats any non-active row as an operator stop, so nothing retried and nothing explained why. Three changes close that gap: - Startup only honors a persisted "stopped" row as intent. A persisted "degraded" host is one that should be running and failed, so it is retried through the same fail-closed activation path. - Restart is disable then enable, and disable persists "stopped" so a crash between the halves fails closed. When the enable half fails the rows are returned to "degraded" with failure_count and last_failure_at recorded, so the next startup or reconcile retries them instead of reading the interrupted restart as a deliberate stop. Enable of an already stopped host keeps its existing contract and stays stopped. - The startup log distinguishes the two states: an operator stop is an INFO line that points at the Decoys tab; a failure is a WARNING that carries the diagnostics reason. The enable, disable, and restart routes now log the paired client, fingerprint, decoy, and outcome, so an operator action on any decoy leaves a trail in the sensor log. Tests cover the degraded-row retry, the failed-restart persistence and its later recovery, the unchanged enable contract, both startup log shapes, and the route audit line for success and failure.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Diagnosing the production Studio Build Mac "unavailable" at every sensor start took two database reads, because the sensor could not say whether the host was stopped on purpose or had failed, and a failed restart leaves the same persisted state as a deliberate stop.
What changes
degradedhosts. Only astoppedrow is operator intent. Adegradedrow goes through the same fail-closed activation path as a normal start.degraded, notstopped. Restart is disable then enable, and disable persistsstoppedso a crash between the halves fails closed. When the enable half fails, the five rows return todegradedwithfailure_countandlast_failure_atrecorded, and the orchestrator's diagnostics carry the reason. The next startup or reconcile retries. Enable of an already stopped host keeps its contract and stays stopped on failure.Deception integrity
Control surface only (
api/,__main__.py, the deep orchestrator's persistence). Nothing attacker-visible changes: no decoy response content, banners, timing, or port set are touched.Verification
ruff check src testsclean.Operator note
The production rows were written
stoppedon 2026-09-03, which matches the early-2.1 test build issue already described in the user guide. This PR does not bulk-enable anything; the host is enabled from the Decoys tab.