fix(setup): address review of isolated-setup-verify checks

Check 10 counted a key as hardware only for an sk- public key or an
OpenPGP-format card. ssh-format signing through gpg-agent uses the
card's aut slot and shows a plain ssh-ed25519 or ssh-rsa key, so a card
gpg --card-status finds now counts whatever gpg.format is.

Check 7a was inconclusive on every correctly secured host, because with
denyRead covering ~/ every path under it looks missing. It now proves
the denial with a sentinel: a file known to exist on the host, such as
the shell rc, that the sandbox cannot see. Inconclusive is left for the
case with no sentinel.

The step-1 output spec gains the inconclusive and unchecked verdicts
the skill now emits, with cases for both and a step-2 case for the
unchecked routing. The rest is review follow-up: managed settings in
check 1's merge, the opt-in question for flagged variable names, the
running claude-code version beside an unchecked check 5, the
credential-shaped wording of the no-print rule, the update prompt's
denial probes, and probe output attributed to stat in the fixtures.

Generated-by: Claude Code (Claude Opus 5.5)

Author: shahar1

docs/setup/secure-agent-setup.md

@@ -3516,18 +3516,21 @@ Verify my secure-agent-setup install is complete. Check each item
 below and report ✓ done / ✗ missing / ⚠ partial, with the evidence
 (file path, line, command output). Do not attempt to fix anything
 — surface the gaps and stop. Never print a credential file's
-contents or an environment variable's value. Before check 1,
-locate my framework checkout (an `.apache-magpie/` snapshot in this
-repo, else the directory my shell rc sources `agent-iso.sh` from,
-else ask me); without one, report checks 5, 10a and 13c as
-"unchecked — no framework checkout". A path that `stat -c '%F %t,%T'`
+contents or a credential-shaped variable's value. Before check 1,
+locate my framework checkout (this repo if it is the framework,
+else an `.apache-magpie/` snapshot in this repo, else the directory
+my shell rc sources `agent-iso.sh` from, else ask me; a candidate
+counts only if it has `tools/agent-isolation/pinned-versions.toml`);
+without one, report checks 5, 10a and 13c as "unchecked — no
+framework checkout", still giving my running `claude --version`. A path that `stat -c '%F %t,%T'`
 shows as `character special file 1,3` is a sandbox placeholder, not
 a file: take its host state from git or from a read I run, and
 never delete it:
 
 1. The merged settings (project `.claude/settings.json`,
    project-local `.claude/settings.local.json`, user
-   `~/.claude/settings.json`; name the file each key came from) have
+   `~/.claude/settings.json`, and the managed settings where
+   readable; name the file each key came from) have
    `sandbox.enabled: true`, the `permissions.deny` block, the
    `permissions.ask` block, the
    `sandbox.network.allowedDomains` block, the
@@ -3562,12 +3565,17 @@ never delete it:
 7. Denials, each as its own Bash call. (a) `test -r` each of
    `~/.aws/credentials`, `~/.ssh/id_*` private keys,
    `~/.config/gh/hosts.yml`, `~/.netrc`, `~/.docker/config.json`:
-   any readable one is ✗ unless it is a sandbox placeholder; if
-   none is visible, report "inconclusive" and ask me to `ls` them
-   from my own terminal. (b) `env | cut -d= -f1`: names only; any
-   credential-shaped name (TOKEN, SECRET, PASSWORD, API_KEY, `*_KEY`,
-   `AWS_*`, `GH_*`, `GITHUB_*`, `AZURE_*`) outside the `agent-iso.sh`
-   passthrough list is ⚠, and so is a `CLAUDE_CODE_ENTRYPOINT` of an
+   any readable one is ✗ unless it is a sandbox placeholder. With
+   `denyRead` covering `~/`, prove the denial with a sentinel: a file
+   under `~/` that exists on my host (such as my shell rc) and that
+   no `allowRead` entry covers; a sandboxed `test -e` that cannot see
+   it is ✓. With no sentinel, report "inconclusive" and ask me to
+   `ls` the candidates from my own terminal; 7a turns ✓ once my `ls`
+   shows a file the sandbox cannot see. (b) `env | cut -d= -f1`:
+   names only; any credential-shaped name (TOKEN, SECRET, PASSWORD,
+   API_KEY, `*_KEY`, `AWS_*`, `GH_*`, `GITHUB_*`, `AZURE_*`) outside
+   the `agent-iso.sh` passthrough list is ⚠ (ask me whether I opted
+   it in with `AGENT_ISO_ALLOW`), and so is a `CLAUDE_CODE_ENTRYPOINT` of an
    IDE extension (such as `claude-vscode`), which never went through
    `claude-iso`. (c) `curl https://example.com` is denied.
 8. **Project-root coverage in the sandbox allowlists.** For this
@@ -3916,8 +3924,9 @@ anything — I will decide what to apply:
    `sandbox.excludedCommands` entry, and any missing vetted-ops
    `Edit` deny rule. See
    [Read-only vetted ops never prompt](#read-only-vetted-ops-never-prompt).
-6. Re-run `cat ~/.aws/credentials`, `echo $AWS_ACCESS_KEY_ID`,
-   `curl https://example.com` and confirm each is still denied.
+6. Re-run check 7's probes (7a `test -r` with the sentinel, 7b
+   variable names only, 7c `curl https://example.com`) and confirm
+   each is still denied.
    Note any newly-allowed call as a regression to investigate.
 ```
 

plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md

@@ -18,7 +18,7 @@ when_to_use: >-
 capability: capability:platform
 surface_hash: sha256:ce33cb9437d44992
 license: Apache-2.0
-measured_tokens: 8586
+measured_tokens: 8836
 ---
 
 <!-- Placeholder convention (see AGENTS.md#placeholder-convention-used-in-skill-files):
@@ -84,7 +84,7 @@ Local modifications go in the override file; framework changes go via PR to `apa
   The doc-allowed "richer custom statusLine" that embeds the framework's sandbox-prefix logic in a larger script is ⚠ (the by-name helper is absent, but the functionality is there).
   Use ⚠ for any *intentional* variation from the doc default; ✗ only for genuine gaps.
 - **Surface evidence.** Each check's report line names the file path, version string, command output, or `sandbox.enabled` value; never just "✓" or "✗" alone.
-- **Never print a credential.** No probe may print a credential file's contents or an environment variable's value: probe files with `test -r`, never `cat` or `head`, and list variables by name only.
+- **Never print a credential.** No probe may print a credential file's contents or a credential-shaped variable's value: probe files with `test -r`, never `cat` or `head`, and list variables by name only.
 - **Sandbox placeholders are not files.** Inside the Claude Code sandbox a denied path that does not exist on the host can appear as a `/dev/null` bind mount: `stat -c '%F %t,%T' <path>` prints `character special file 1,3`.
   `git status` lists such placeholders as untracked (`.bash_profile`, `.claude/settings.json`, `.mcp.json`), and reading one returns nothing, so an empty settings file seen this way is not evidence for checks 1 or 6.
   Take the host state from git (`git ls-files <path>`, `git log -- <path>`) or from a read the user runs outside the sandbox, and never delete the placeholder.
@@ -100,9 +100,10 @@ Look, in order, for: the current repository when it is the framework itself; an
 A candidate counts only when it has `tools/agent-isolation/pinned-versions.toml`; a copy under `~/.claude/agent-isolation/` is not a checkout.
 With none found, ask the user for the path once.
 With no checkout at all, report check 5's version comparisons, 10a's script comparison and 13c as **unchecked — no framework checkout**, not ✗ or ⚠, and say which path to supply.
+Check 5 still reports the running `claude --version` beside that verdict.
 
 1. Effective settings shape: `sandbox.enabled: true`, `permissions.deny`, `permissions.ask`, `sandbox.network.allowedDomains`, the `sandbox.filesystem` allowlist (`allowRead`/`allowWrite`), and `sandbox.filesystem.denyRead` containing `~/` (or the resolved absolute home).
-   Evaluate the merge of the project `.claude/settings.json`, the project-local `.claude/settings.local.json` and the user-scope `~/.claude/settings.json`, and name the file each key came from.
+   Evaluate the merge of the project `.claude/settings.json`, the project-local `.claude/settings.local.json`, the user-scope `~/.claude/settings.json` and, where readable, the managed settings (`/etc/claude-code/managed-settings.json` and `managed-settings.d/` on Linux, `/Library/Application Support/ClaudeCode/` on macOS, `C:\Program Files\ClaudeCode\` on Windows), and name the file each key came from.
    A missing `denyRead` home entry is ✗: Linux bubblewrap reads are deny-only, so without it every credential file under `~/` stays readable to sandboxed commands (`tools/sandbox-lint` enforces the same rule).
    A missing committed project `.claude/settings.json` is not ✗ on its own when the merged settings pass; say which file carries them.
 2. User-scope `~/.claude/settings.json` wiring: `PreToolUse` `Bash` matcher → `sandbox-bypass-warn.sh`, `PostToolUse` `Bash` matcher → `sandbox-error-hint.sh`, `statusLine` → `sandbox-status-line.sh` (or a custom statusline script that embeds the framework's prefix logic, the doc-allowed variant; report ⚠).
@@ -138,10 +139,14 @@ With no checkout at all, report check 5's version comparisons, 10a's script comp
    - **7a — credential files.** For each of `~/.aws/credentials`, the `~/.ssh/id_*` private keys (not `*.pub`), `~/.config/gh/hosts.yml`, `~/.netrc` and `~/.docker/config.json`, run `test -r <path>` and `stat` it (`stat -c '%F %t,%T'` on Linux, `stat -f %HT` on macOS).
      The first one the sandbox reports readable is ✗, unless `stat` prints `character special file 1,3` (a sandbox placeholder, which counts as denied).
      `Operation not permitted` from `stat` (Seatbelt) is a denial of a file that exists: ✓.
-     Under bubblewrap a denied file and a missing one look the same, so when no candidate is visible report 7a as **inconclusive**, not ✓, and ask the user to `ls` the candidates from their own terminal; ✓ once one of them exists on the host.
+     Under bubblewrap a denied file and a missing one look the same, so a candidate that is not visible proves nothing on its own.
+     When check 1 found `denyRead` covering `~/`, prove the denial with a sentinel: a file under `~/` known to exist on the host (such as the shell rc check 4 read) that no `allowRead` entry covers.
+     A sandboxed `test -e <sentinel>` that fails proves the denial: ✓.
+     Only with no sentinel available is 7a **inconclusive**: ask the user to `ls` the candidates from their own terminal, and report ✓ once one exists there that the sandbox cannot see.
    - **7b — clean environment.** List the variable names, never the values: `env | cut -d= -f1 | sort`.
      Flag every credential-shaped name (containing `TOKEN`, `SECRET`, `PASSWORD` or `API_KEY`, ending in `_KEY`, or starting with `AWS_`, `GH_`, `GITHUB_` or `AZURE_`) that is not on the `agent-iso.sh` passthrough list; any such name is ⚠, listed by name.
      A single empty variable proves nothing: the reference `sandbox.credentials` already unsets `AWS_ACCESS_KEY_ID` for sandboxed commands, wrapper or not.
+     An `AGENT_ISO_ALLOW` / `CLAUDE_ISO_ALLOW` opt-in is invisible from inside the session, so ask the user whether a flagged name was opted in before calling it a gap.
      Report `CLAUDE_CODE_ENTRYPOINT`.
      An IDE extension's value (such as `claude-vscode`) means the session never went through `claude-iso`, so no wrapper stripped its environment: ⚠, with the credential-shaped names found or "none found" ([why](../../../../docs/setup/secure-agent-setup.md#the-clean-env-wrapper)).
    - **7c — network.** `curl https://example.com` should deny at the permission-prompt layer (`Permission to use Bash with command curl … has been denied`).
@@ -175,7 +180,9 @@ With no checkout at all, report check 5's version comparisons, 10a's script comp
 
 10. **Hardware-key touch overlay and the signing key.** Only when commits are signed (`git config --get commit.gpgsign` is `true`) or a remote is reached over ssh through gpg-agent (`SSH_AUTH_SOCK` names its socket); otherwise **n/a**.
     10c applies to any ssh-format signing key.
-    10a, 10b, 10d and 10e apply only to a hardware key and are **n/a** otherwise: with `gpg.format=ssh` the public key line starts with `sk-` (`sk-ssh-ed25519@openssh.com`, `sk-ecdsa-sha2-nistp256@openssh.com`); with OpenPGP, `gpg --card-status` finds a card (the user runs it when the sandbox cannot see the device).
+    10a, 10b, 10d and 10e apply only to a hardware key and are **n/a** otherwise.
+    A hardware key is an `sk-` public key (`sk-ssh-ed25519@openssh.com`, `sk-ecdsa-sha2-nistp256@openssh.com`), or a card `gpg --card-status` finds, whatever `gpg.format` is: ssh-format signing through gpg-agent uses the card's `aut` slot and shows a plain `ssh-ed25519` or `ssh-rsa` key.
+    The user runs `gpg --card-status` when the sandbox cannot see the device.
     Five sub-checks:
     [`conditional-checks.md`](conditional-checks.md#check-10--hardware-key-touch-overlay-and-the-signing-key).
 
@@ -291,8 +298,8 @@ If anything is ✗ or ⚠, suggest the matching follow-up skill without invoking
 
 - ✗ on checks 1 / 2 / 3 / 4 → `setup-isolated-setup-install` (missing install pieces).
   For a missing `denyRead` home entry, name the settings file to add it to.
-- **Unchecked — no framework checkout** on check 5, 10a or 13c → not a failure; ask the user for their `magpie` clone's path, or to clone one ([install item 2](../isolated-setup-install/SKILL.md)), and re-run.
-- Inconclusive 7a → the user runs `ls` on the candidate credential paths from their own terminal; one that exists there while the sandbox cannot see it is ✓.
+- **Unchecked — no framework checkout** on check 5, 10a or 13c → not a failure; re-run `setup-isolated-setup-verify` with the path of the user's `magpie` clone, cloning one first if they have none ([install item 2](../isolated-setup-install/SKILL.md)).
+- Inconclusive 7a (no sentinel) → the user runs `ls` on the candidate credential paths from their own terminal; one that exists there while the sandbox cannot see it is ✓.
 - ⚠ on check 7b (an IDE-launched session, or credential-shaped names in the environment) → no skill fixes it: start the session through `claude-iso` from a terminal, or keep secrets out of the shell that starts the IDE and rely on `sandbox.credentials` ([The clean-env wrapper](../../../../docs/setup/secure-agent-setup.md#the-clean-env-wrapper)).
 - **✗ on check 5 (claude-code below the `min_version` floor, running under Claude Code)** → **hard fail; stop.**
   Tell the operator to upgrade (`npm install -g --no-save @anthropic-ai/claude-code@latest`) and re-run; the setup cannot be certified on a below-floor runtime.

plugins/magpie-setup/skills/isolated-setup-verify/conditional-checks.md

@@ -160,12 +160,15 @@ reading this file; read the section for a check whose condition holds.
     `true`) or a remote is reached over ssh through gpg-agent
     (`SSH_AUTH_SOCK` names its socket); otherwise report **n/a**.
     10c applies to any ssh-format signing key. 10a, 10b, 10d and 10e
-    apply only to a hardware key and are **n/a** otherwise: with
-    `gpg.format=ssh` the public key line starts with `sk-`
-    (`sk-ssh-ed25519@openssh.com`, `sk-ecdsa-sha2-nistp256@openssh.com`);
-    with OpenPGP, `gpg --card-status` finds a card (the user runs it
-    when the sandbox cannot see the device). A plain `ssh-ed25519` key
-    needs no touch, so the overlay has nothing to show.
+    apply only to a hardware key and are **n/a** otherwise. A
+    hardware key is an `sk-` public key (`sk-ssh-ed25519@openssh.com`,
+    `sk-ecdsa-sha2-nistp256@openssh.com`), or a card
+    `gpg --card-status` finds, whatever `gpg.format` is: ssh-format
+    signing through gpg-agent uses the card's `aut` slot and shows a
+    plain `ssh-ed25519` or `ssh-rsa` key. The user runs
+    `gpg --card-status` when the sandbox cannot see the device. A
+    plain software key needs no touch, so the overlay has nothing to
+    show.
     Five sub-checks, plus a note for a hardware key: the key's own touch policies
     (`ykman openpgp info`, run by the user — the sandbox does not
     see the device) are what make the overlay matter; report them

tools/skill-evals/README.md

@@ -35,7 +35,7 @@ Suites are currently implemented for:
 - **pr-management-stats** — 6 cases across 1 step (build-loop)
 - **pr-management-triage** — 33 cases across 6 steps (backport-check, classify-loop, deliver-note, guard-reroute, terminal-links, interaction-progress); the pre-filters, decision table and pagination are pytest-tested in `tools/pr-management`
 - **list-skills** — 8 cases across 2 steps (step-1-command, step-2-present)
-- **setup-isolated-setup-verify** — 31 cases across 4 steps (runtime-routing, codex-native, step-1-classify, step-2-recommend)
+- **setup-isolated-setup-verify** — 34 cases across 4 steps (runtime-routing, codex-native, step-1-classify, step-2-recommend)
 - **setup-isolated-setup-update** — 15 cases across 4 steps (runtime-routing, step-snapshot-drift, step-tool-freshness, step-after-report)
 - **setup-isolated-setup-doctor** — 26 cases across 3 steps (runtime-routing, interpret-probes, after-report)
 - **contributor-activity-sweep** — 13 cases across 3 steps (step-0-resolve-inputs, step-1-classify-reviews, step-2-render)

tools/skill-evals/evals/setup-isolated-setup-verify/README.md

@@ -5,14 +5,14 @@
 
 Behavioral evals for the `setup-isolated-setup-verify` skill.
 
-## Suites (31 cases total)
+## Suites (34 cases total)
 
 | Suite | Step | Cases | What it covers |
 |---|---|---|---|
 | codex-native | Codex security model | 3 | untrusted or absent guard despite project trust, unsafe vetted write allowance, and the credential read-isolation gap |
 | runtime-routing | Runtime routing | 2 | Codex and Gemini route to their native adapters and never require Claude files |
-| step-1-classify | The 12 checks | 20 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10), dev-tool paths missing from settings.local.json (check 8 ⚠), whole-user hook dir unreadable from the sandbox (check 8 ✗), working directories missing under the read block with a glob entry (check 15 ⚠), roborev installed with `review` and `show` auto-allowed no config-edit deny and an incomplete `ask` list (check 17b, 17d and 17e ✗), roborev's hook replacing the shared whole-user dispatcher (check 17f ✗), `~/.roborev/` made writable with and without step B.2 (check 17c ✗), a roborev MCP server registered (check 17g ⚠), the vetted-ops checkout listed as a working directory (check 16f ✗), `denyRead` missing from every merged scope (check 1 ✗), settings only in `settings.local.json` beside a sandbox placeholder (check 1 ✓ naming the file), an IDE-launched session with credential-shaped variable names (check 7 ⚠), a plain `ssh-ed25519` signing key with `commit.gpgsign` on (check 10 ✓, the overlay sub-checks n/a) |
-| step-2-recommend | After the report | 6 | all-pass, install needed, update needed, project-root missing, multiple gaps, roborev findings (settings edit by the operator, `.roborev.toml` left to them) |
+| step-1-classify | The 12 checks | 22 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10), dev-tool paths missing from settings.local.json (check 8 ⚠), whole-user hook dir unreadable from the sandbox (check 8 ✗), working directories missing under the read block with a glob entry (check 15 ⚠), roborev installed with `review` and `show` auto-allowed no config-edit deny and an incomplete `ask` list (check 17b, 17d and 17e ✗), roborev's hook replacing the shared whole-user dispatcher (check 17f ✗), `~/.roborev/` made writable with and without step B.2 (check 17c ✗), a roborev MCP server registered (check 17g ⚠), the vetted-ops checkout listed as a working directory (check 16f ✗), `denyRead` missing from every merged scope (check 1 ✗), settings only in `settings.local.json` beside a sandbox placeholder (check 1 ✓ naming the file), an IDE-launched session with credential-shaped variable names (check 7 ⚠), a plain `ssh-ed25519` signing key with `commit.gpgsign` on (check 10 ✓, the overlay sub-checks n/a), no sentinel and no `ls` for 7a (check 7 inconclusive), no framework checkout (checks 5, 10 and 13 unchecked, installed versions still reported) |
+| step-2-recommend | After the report | 7 | all-pass, install needed, update needed, project-root missing, multiple gaps, roborev findings (settings edit by the operator, `.roborev.toml` left to them), no framework checkout (pass, with a re-run once a clone exists) |
 
 ## Run
 

tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/case-1-all-pass/report.md

@@ -120,7 +120,10 @@ Effective sandbox.enabled: true
 ## Check 7 — Denial commands
 
 test -r <path>, for each of ~/.aws/credentials, ~/.ssh/id_ed25519, ~/.config/gh/hosts.yml, ~/.netrc, ~/.docker/config.json:
-  none readable — No such file or directory for every path
+  rc=1 for every path
+
+stat -c '%F %t,%T' <path>, for each of the same paths:
+  stat: cannot statx '<path>': No such file or directory
 
 ls ~/.aws/credentials ~/.ssh/id_ed25519 (run by the user from their own terminal):
   /home/alice/.aws/credentials