Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/setup/sandbox-troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -742,6 +742,23 @@ The message names the rule that refused the request, and it points back at this
The full create-time refusal table lives in [`tools/container-gateway/README.md` → What the policy refuses](../../tools/container-gateway/README.md#what-the-policy-refuses).
Adjust the request rather than widening the sandbox: a `403` from the gateway is the policy working as intended, not a sandbox misconfiguration.

A tool that drives the CLI for you can hide that message, by capturing the CLI's output or by calling an HTTP-upgrade endpoint, for which the CLI prints only the status (`unable to upgrade to h2c, received 403`).
The gateway logs every refusal as a `deny <method> <path>: <reason>` line in `<run-dir>/container-gateway.log`, so read the reason there.

Some workflows cannot pass the gateway, however the request is adjusted:

- **BuildKit.** `docker buildx build`, and `docker build` wherever buildx is its default, talk to the builder over `POST /grpc` and `/session`, a gRPC stream the gateway cannot inspect, so it refuses both (`deny POST /grpc: unknown-endpoint`).
A buildx `docker-container` builder container is label-checked like any other resource and is refused when it does not carry the project label (`label-check on buildx_buildkit_default`).
- **The daemon socket inside a container.** A tool that bind-mounts `DOCKER_HOST`'s socket into its own container is refused, since the run directory is never a bind source.
- **An untyped `docker inspect <name>`.** The CLI asks for a container of that name first, and a name that is not a project container gets a label-check `403` (`label-check on <image reference>` in the log), where a tool checking whether an image exists may expect a `404`; `docker image inspect` asks for the image directly.

Apache Airflow's `breeze`, for example, runs into all three.
Excluding it from the sandbox is no way past them, for the reason given under Root cause above.

An agent that meets such a refusal stops and reports it to the user, with the `deny` lines from the log.
It does not route around it: no other socket, no sandbox bypass, and no switch to a workflow the project's own instructions rule out, such as running tests on the host when they say to use the project's container tooling.
The user decides, for example to run the tool from their own terminal.

### Notes

- The gateway's `docker`-backend discovery on macOS reads the current `docker context`.
Expand Down
1 change: 1 addition & 0 deletions docs/setup/secure-agent-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -1877,6 +1877,7 @@ automatically. Five classes of failure are recognised today:
|---|---|
| `Could not open a connection to your authentication agent` / `agent refused operation` / `ssh-add: error fetching identities` / `Permission denied (publickey)` | [SSH agent / Yubikey unreachable](sandbox-troubleshooting.md#ssh-agent--yubikey-appears-unreachable-from-inside-the-sandbox) |
| `Cannot connect to the Docker daemon` / `open /var/run/docker.sock: operation not permitted` / `Cannot connect to Podman` / podman `connect: permission denied` | [Docker / Podman socket denied](sandbox-troubleshooting.md#docker--podman-command-fails-with-a-socket-error) |
| `container-gateway: …; see docs/setup/sandbox-troubleshooting.md…` / `unable to upgrade to h2c, received 403` | [Container gateway policy refusal](sandbox-troubleshooting.md#docker--podman-command-fails-with-a-socket-error) |
| `127.0.0.1 … Permission denied` / `Operation not permitted … bind` / `Errno 49 … assign requested address` / `Connection refused … 127.0.0.1` | [Localhost port-bind blocked](sandbox-troubleshooting.md#test-cannot-bind-to-a-localhost-port) |
| `/tmp/…: Read-only file system` / `mktemp: failed to create` | [Temp files fail under `/tmp`](sandbox-troubleshooting.md#temp-files-fail-with-read-only-file-system-under-tmp) |
| `x509: OSStatus -26276` / `HTTP 401: Requires authentication (https://api.github.com…` | [`gh` ran inside the sandbox](sandbox-troubleshooting.md#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,4 @@
for installs that do not carry the framework source (see `isolated.py`).
"""

FRAMEWORK_FINGERPRINT = "sha256:520552c9c1bc08b8"
FRAMEWORK_FINGERPRINT = "sha256:e1576c7a6ffe4d0e"
4 changes: 4 additions & 0 deletions tools/agent-isolation/sandbox-error-hint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,10 @@ elif match "bwrap: Can't create file (at )?.*: Read-only file system"; then
hint="An allowRead entry that is a parent of an allowWrite path makes bubblewrap mount that path read-only. See ${doc_path}#every-bash-call-fails-with-bwrap-cant-create-file--read-only-file-system"
elif match 'Cannot connect to the Docker daemon|open /var/run/docker\.sock: operation not permitted|Cannot connect to Podman|connect: permission denied.*podman\.sock'; then
hint="Docker / Podman runtime socket denied by the sandbox. See ${doc_path}#docker--podman-command-fails-with-a-socket-error"
# A gateway policy refusal: its 403 body, or the bare status an HTTP-upgrade
# endpoint (BuildKit's /grpc) reports instead of the body.
elif match 'container-gateway: .*; see docs/setup/sandbox-troubleshooting\.md|unable to upgrade to h2c, received 403'; then
hint="The container gateway refused this request by policy (the reason is a 'deny' line in container-gateway.log in its run directory). Do not route around it with another socket, a sandbox bypass, or a workflow the project's instructions rule out: report the refusal to the user. See ${doc_path}#docker--podman-command-fails-with-a-socket-error"
elif match '127\.0\.0\.1.*[Pp]ermission denied|[Oo]peration not permitted.*bind|Errno 49.*assign requested address|HTTP.*Connection refused.*127\.0\.0\.1'; then
hint="Localhost port-bind or loopback HTTP may be sandbox-blocked. See ${doc_path}#test-cannot-bind-to-a-localhost-port"
elif match "/tmp/[^ ]*'?: Read-only file system|Read-only file system: '/tmp/|mktemp: failed to create"; then
Expand Down
23 changes: 23 additions & 0 deletions tools/agent-isolation/tests/test_sandbox_error_hint.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@
import subprocess
from pathlib import Path

import pytest

SCRIPT = Path(__file__).parent.parent / "sandbox-error-hint.sh"
DOC = "docs/setup/sandbox-troubleshooting.md"

Expand Down Expand Up @@ -135,6 +137,27 @@ def test_docker_signature(self) -> None:
assert result.returncode == 1
assert f"{DOC}#docker--podman-command-fails-with-a-socket-error" in result.stderr

@pytest.mark.parametrize(
"stderr",
[
(
"Error response from daemon: container-gateway: label-check: buildx_buildkit_default does not "
"belong to this project; see docs/setup/sandbox-troubleshooting.md#docker--podman-command-fails-with-a-socket-error"
),
"ERROR: failed to dial gRPC: unable to upgrade to h2c, received 403",
],
)
def test_gateway_policy_refusal_signature(self, stderr: str) -> None:
result = _run(_bash(stderr=stderr))
assert result.returncode == 1
assert f"{DOC}#docker--podman-command-fails-with-a-socket-error" in result.stderr
assert "report the refusal to the user" in result.stderr

def test_gateway_backend_error_is_not_a_policy_refusal(self) -> None:
# A 502 is a backend outage, not a policy decision, and carries no catalog pointer.
result = _run(_bash(stderr="Error response from daemon: container-gateway: backend timed out"))
assert result.returncode == 0

def test_tmp_signature(self) -> None:
result = _run(
_bash(stderr="mktemp: failed to create directory via template '/tmp/tmp.XXXXXXXXXX': Read-only file system")
Expand Down
Loading