Warden — govern the deliverables of long-horizon agent tasks

The only "control" most people have over an agent's deliverable is the agent's own word: "Done!" That is exactly the signal you cannot trust. Warden replaces trust the agent with trust a contract you wrote.

Warden is a tiny, dependency-light tool (one pip/uv install) that lets you:

1. Specify what an acceptable deliverable looks like as a declarative contract (machine-checkable acceptance criteria).
2. Verify any deliverable against the contract and get a structured verdict (per-check pass/fail with evidence).
3. Gate acceptance — warden verify exits non-zero unless every required check passes, so it drops straight into CI or a pre-merge hook.
4. Govern the loop — warden run drives an agent, and on rejection feeds the failing-check evidence back to the agent and retries, up to a budget, so the agent converges on a contract-satisfying deliverable instead of stopping at "looks done."

The core failure mode of long-horizon agents is false completion — declaring a task done when the deliverable does not satisfy intent. Warden is built to eliminate it.

Install (under a minute)

# Run without installing:
uvx --from git+https://github.com/zhenthebuilder/warden warden --help

# Or install the CLI globally:
uv tool install git+https://github.com/zhenthebuilder/warden     # uv
pipx install git+https://github.com/zhenthebuilder/warden        # pipx

# Or in a project:
pip install git+https://github.com/zhenthebuilder/warden

One-liner installer:

curl -fsSL https://raw.githubusercontent.com/zhenthebuilder/warden/main/install.sh | sh

60-second tour

warden init                       # writes a starter contract.yaml
warden verify -c contract.yaml    # gate the current directory; exit 1 if rejected
warden types                      # list built-in check types

A contract is just YAML:

name: my-service
checks:
  - name: tests-pass
    type: command
    run: "python -m pytest -q"

  - name: api-documented
    type: regex_present
    path: README.md
    pattern: "## API"

  - name: no-secrets
    type: regex_absent
    path: src/config.py
    pattern: "AKIA[0-9A-Z]{16}"

  - name: meets-spec            # fuzzy criteria via an LLM judge
    type: llm_judge
    prompt: "Does README.md document BOTH install and usage? Answer YES/NO."
    file: README.md
    required: false

Govern an agent loop

Let Warden drive any agent (anything that reads a prompt and edits files) until the contract passes:

# the agent must be able to write files in the deliverable dir (-d).
warden run -c contract.yaml -d . --max-iters 3 \
  --agent 'claude -p --permission-mode bypassPermissions --output-format json --model claude-haiku-4-5-20251001' \
  --agent-json \
  --task "Implement the /healthz endpoint and add a test for it."
# swap in ANY agent with --agent '<cmd>' (a prompt is piped on stdin, or use {prompt}).

On each rejection Warden appends the failing checks' evidence and the agent's previous deliverable to the prompt and re-invokes the agent, so a stateless backend revises rather than regenerates. Nothing is reported as "accepted" unless the contract verifies.

Built-in check types

type	what it asserts
file_exists / file_absent	a path is present / absent
command	a shell command exits with the expected code (run tests, linters, builds)
regex_present / regex_absent	a pattern is / isn't found in a file
python_defines	importing a file yields a callable symbol
python_assert	a list of Python assertions all pass
llm_judge	a model answers YES to a rubric (fuzzy criteria)

Register your own with warden.checks.register("my_type").

As a library

from warden import load_contract, evaluate
verdict = evaluate(load_contract("contract.yaml"), workdir=".")
print(verdict.accepted, verdict.score)
print(verdict.to_json())

Why this works (the evidence)

We A/B-tested the governed loop on HumanEval (OpenAI, 164 problems). The contract used only information a deployer has — the function must parse and define the entry point, import/execute, and pass the assertions in its own docstring examples plus self-tests the agent proposes. The official HumanEval grader (never shown to the agent) measures real correctness.

See bench/ for the runner, raw logs, and scoring code, and the paper for the full write-up.

Headline result — on HumanEval-Bundle (a long-horizon proxy: a library of 8 functions, "done" only if all 8 work; 60 deliverables): the baseline ships 30% of deliverables silently broken; the Warden gate cuts that by a third (18 → 12 false completions) and blocks 7 — all 7 genuinely broken, zero good deliverables withheld. Correctness is unchanged: the value is the gate catching flaws the agent declared done, not better coding. On standard single-function HumanEval a strong model is near-saturated (pass@1 ≈ 0.98), so governance is near-cost-free insurance with little to repair — both settings are reported honestly.

License

MIT.