[shahar1]

Summary

A setup-isolated-setup-verify run on a Linux host (Claude Code started from the VS Code extension, adopter settings only in .claude/settings.local.json) reported several checks as passing that proved nothing. This PR closes those false passes:

• Check 1 evaluates the merged settings (managed, user, project, project-local) instead of only the committed project file, names the file each key came from, and requires sandbox.filesystem.denyRead to cover ~/. Without that entry, bubblewrap reads are deny-only and credential files stay readable from the sandbox.
• Check 7a probes common credential paths with test -r / stat (never cat). When denyRead covers ~/, a sentinel file known to exist on the host proves the denial. It reports inconclusive only when no sentinel is available. Before, a missing ~/.aws/credentials produced the same output as a working denial.
• Check 7b lists environment variable names only and flags credential-shaped ones. It reports CLAUDE_CODE_ENTRYPOINT and gives ⚠ for IDE-launched sessions, which never pass through claude-iso. The old probe (echo $AWS_ACCESS_KEY_ID) passed regardless, because sandbox.credentials already unsets that variable. The clean-env wrapper doc gains a paragraph on IDE launches.
• Check 5 / 10a / 13c first locate the framework checkout (the current repo, an .apache-magpie/ snapshot, or the directory the shell rc sources agent-iso.sh from), and report unchecked rather than ✗ or ⚠ when there is none. A marketplace-only install has no tools/ directory, and the running claude-code version is still reported.
• Check 10 runs the touch-overlay sub-checks only for a hardware key (an sk- public key, or a card that gpg --card-status finds). 10c still applies to every ssh-format key.
• The verify and doctor skills explain sandbox placeholders: write-protected paths that do not exist show up as character devices 1,3, so git status lists them as untracked.
• The verification prompts in docs/setup/secure-agent-setup.md (including "Keeping the setup updated") are kept in sync, and never print a credential.

Type of change

• Skill change (.claude/skills/<name>/) — eval fixtures updated below
• Documentation (docs/, README.md, CONTRIBUTING.md)

Test plan

• prek run --all-files --skip workspace-pytest passes
• prek run workspace-pytest --from-ref upstream/main --to-ref HEAD passes
• Eval suite for the affected skill, skill-eval --cli "claude -p" tools/skill-evals/evals/setup-isolated-setup-verify/: 29/34 pass. All 7 new cases pass. Of the 5 failures, 4 also fail on unmodified upstream/main (step-1 case-7; step-2 case-3, case-5, case-6; the baseline run was 20/27). The fifth, step-1 case-14, passed on two re-runs and is not touched by this PR.
• New eval cases: step-1 case-17-deny-read-missing, case-18-local-settings-only, case-19-ide-entrypoint-env, case-20-plain-ssh-signing-key, case-21-7a-inconclusive, case-22-no-framework-checkout; step-2 case-7-unchecked-no-checkout. The step-1 status enum gains inconclusive and unchecked.
• Reviewed by an independent agent. Its requested changes are in the second commit.

RFC-AI-0004 compliance

• Sandbox — the verify checks now detect a missing home denyRead and stop printing credential contents
• Vendor neutrality — placeholders only (check-placeholders passes)

Notes for reviewers (optional)

• Companion PR fix(agent-isolation): guard Linux allowAllUnixSockets with denyRead #1809 covers the Linux allowAllUnixSockets exposure (verify check 12) and the sandbox-lint rules. Both PRs add step-1 eval cases and touch the same README case counts, so whichever merges second needs a trivial rebase.
• Not included: a clean-env wrapper mode for the VS Code extension's claudeCode.claudeProcessWrapper. agent-iso.sh's env -i allowlist would drop the CLAUDE_CODE_* variables the extension needs, so that needs its own design and will be raised as a separate issue.

Generative AI: drafted with Claude Code (Claude Opus 5.5) and checked by an independent Claude Opus review agent; opened as a draft for the author's line-by-line review before it is marked ready.

🤖 Generated with Claude Code

[potiuk]

Thanks, this is a clear improvement over the old cat/echo probes. A few of the new rules still let a false pass or false fail through; details inline. The first is the one that matters most.