Skip to content

Repository navigation

operations.md

Latest commit

 

History

History
245 lines (187 loc) · 13.2 KB

operations.md

File metadata and controls

245 lines (187 loc) · 13.2 KB
Raw

Crew Operations Guide

Audience & purpose: end users and operators who need to run crewd by hand, troubleshoot a stuck daemon, or upgrade/uninstall — a companion to user-guide.md, the user manual (day-to-day tool usage doesn't need this document; troubleshooting does). This guide covers running and troubleshooting the crewd daemon in practice: lifecycle procedures, crash recovery, upgrades, and the real (as opposed to imagined) install/uninstall path. For the full flag-by-flag command reference, see cli-reference.md — this guide doesn't repeat flags, only the procedures around them.

Examples below invoke crewd bare — nothing puts it on your PATH; alias it or substitute the full path. Where installed and source-built binaries live is specified once in cli-reference.md.

Daemon Lifecycle

Starting the runtime

crewd serve --repo <path> --state-dir "$HOME/.omp/crew" [--idle-seconds <n>]

In normal use you don't run this yourself — the OMP extension spawns it on first use per repository (see development.md). Run it by hand for debugging or CI. See cli-reference.md for every flag, and the state-directory note before you pick a --state-dir — the CLI's own default when it's omitted resolves the same precedence the extension uses (StateRoot::resolve), but both read it from the current process environment independently, so they only land in the same place if that environment agrees. Pass --state-dir explicitly (as above) whenever you're not sure the two shells' environments match.

Single-instance enforcement

serve takes an advisory flock(2) on <runtime-dir>/runtime.lock — not an O_EXCL create. The lock file itself is never deleted; a "stale" lock is simply one whose flock the kernel has already released because its owning process died. If a runtime is already running, the new process exits 73 (EX_TEMPFAIL) and prints the live holder's identity as JSON on stdout.

Graceful shutdown

crewd stop --repo <path> --state-dir "$HOME/.omp/crew"

The runtime journals a stop record, closes the database actor, removes the socket, then releases the lock — in that order, so the socket's disappearance is proof the journal already shut down cleanly. stop polls for the socket to disappear for up to 10 seconds after sending SIGTERM; if no runtime is found holding the lock, it exits 1 immediately rather than signalling anything.

Watching events live

# Every run in the project, live-tailed after an initial catch-up replay
crewd monitor --repo <path> --state-dir "$HOME/.omp/crew"

# Filtered to one run
crewd monitor --repo <path> --state-dir "$HOME/.omp/crew" --run-id <run-id>

There's no separate "replay" vs. "live" mode — monitor always replays what's already journaled and then keeps tailing new events until you interrupt it. If the daemon restarts mid-session, monitor reconnects and resumes from where it left off rather than exiting. For day-to-day use inside OMP, prefer /crew (see user-guide.md) — this CLI form is for scripting or debugging outside an OMP session.

Diagnosing a runtime

crewd doctor --repo <path> --state-dir "$HOME/.omp/crew" [--json]

Runs the full check catalog documented in cli-reference.md against the same paths serve would use. Exits 0 healthy, 1 unhealthy. Inside OMP, prefer crew_doctor / /crew doctor — same check catalog, no need to resolve paths by hand.

Crash Recovery

serve runs recovery once, synchronously, right after opening the database and before the socket accepts any connection. Every run the journal still calls queued, starting, or working at that moment is transitioned to failed, however recent its last event: serve holds the single-instance flock and starts with an empty adapter registry, so a non-terminal run in the journal provably has no live process behind it, and there's no evidence the work completed. Runs sitting in paused, waitingUser, or waitingPeer are left alone by default, since those states mean the run is intentionally waiting on a human or a peer worker, not stuck. Each run is recovered independently; one failure doesn't abort the sweep for the others.

Recovery is boot-time only, deliberately: a run supervised by the running daemon can be silent for minutes without being dead — no adapter emits a heartbeat — so age alone must never fail a live run. crewd doctor's stale_runs check is the live-daemon counterpart: it names runs whose last journaled event is older than five minutes, read-only, transitioning nothing.

Recovering from an unexpected crash

  1. Check the log (only present without --foreground): cat <runtime-dir>/runtime.log
  2. Check for orphaned processes: ps aux | grep crewd
  3. Restart: crewd serve --repo <path> --state-dir "$HOME/.omp/crew" — there's nothing to clean up by hand first. The lock file doesn't need removing (a crashed process's flock is already released by the kernel), and the next serve runs recovery automatically on startup.

Install, Upgrade, and Uninstall

Crew ships as an OMP marketplace plugin (extension + skills, cloned from this repository) plus a crewd binary downloaded on demand as a GitHub Release asset. There is no Homebrew formula, apt/deb/rpm package, or any other system package — don't reach for a package manager here.

Installing / uninstalling

/marketplace add nikolasd/crew     # registers this repo as a marketplace source
/marketplace install crew@crew   # installs the extension + skills

Exit and start a new omp session — /reload-plugins does not reload extension modules, so /crew-install (and every crew_* tool) only exists once a fresh session has loaded the newly-installed module. Then, in that session:

/crew-install              # downloads and verifies the crewd binary

Uninstalling works in any session:

/marketplace uninstall crew@crew   # removes the extension + skills

This repository is public. /marketplace add clones it via HTTPS — no authentication required. /crew-install downloads the crewd binary via the GitHub REST API; a GITHUB_TOKEN or GH_TOKEN environment variable (or a gh auth login session) is optional but recommended — without one you may hit GitHub's unauthenticated rate limit (60 requests/hour); with one, it's 5,000/hour. See the README's Installation section for details. The crewd binary itself resolves in two tiers (see development.md): OMP_CREW_BINARY (a local-development override) if set, otherwise the SHA-256-verified binary /crew-install cached under the Crew state root — there's no separate binary install step beyond that command.

After uninstalling, confirm no crewd process is still running: ps aux | grep crewd, and kill <pid> anything that's left (this shouldn't happen in normal operation — the extension doesn't detach a crewd process that outlives every session using it on purpose, but a hard OMP crash can leave one behind since serve is spawned detached).

Removing state (event journal, leases, artifacts) is a separate, optional step, since it's not part of the package: rm -rf <state-root> for the resolved state root (see the state-directory precedence in the CLI reference) — or just the one repository's subdirectory under <state-root>/repos/<repository-id>/ if you want to keep other repositories' history.

Upgrading

The extension and the crewd binary are no longer one atomic install — upgrading each is a separate step. /marketplace upgrade crew@crew refreshes the extension + skills from this repository; if that bumps the extension's version, re-run /crew-install to download the matching binary (a version-mismatched cached binary is rejected rather than silently reused). If you're testing an unreleased build instead, use the OMP_CREW_BINARY override described in development.md instead of installing anything.

  1. Stop the running daemon for any repository you're about to touch: crewd stop --repo ... (or just let the next ensureRuntime() call reconnect after the update — a stale old binary still running won't be replaced automatically, so stop it first if you want the new version active immediately).
  2. Update: /marketplace upgrade crew@crew, then /crew-install if the version changed. Confirm with crewd version (should report the new version) and crewd doctor --json (should report healthy: true, or only expected, pre-existing failures).

Troubleshooting

Runtime won't start:

  • Check whether one's already running: crewd status --repo <path> --state-dir "$HOME/.omp/crew" (exit 73 from serve means another instance holds the lock — that's not a bug, connect to it instead of restarting).
  • Check the log: cat <runtime-dir>/runtime.log.
  • Run crewd doctor --json — it doesn't need a live connection and will usually name the exact failing check (permissions, disk space, schema mismatch, unsupported platform, etc.).

crew_health fails but you're not sure why:

  • Run crew_doctor (or /crew doctor) — it diagnoses without needing a live connection, which is exactly the case crew_health can't help with.

Doctor reports a state_dir_writable or socket_permissions failure:

  • Check ownership and mode: ls -ld <runtime-dir> should be 0700, owned by you; ls -l <runtime-dir>/runtime.sock (if present) should be 0600.

Adapter conformance failures:

  • crewd adapters --json reports each adapter's effective capabilities from a fixture run — start there before assuming a live-vendor issue.
  • CREW_DISABLE_VENDOR_CLI=1 cargo test --test conformance runs the same checks offline; a live run needs the variable set to 0 explicitly, never merely unset — .cargo/config.toml's own [env] block sets it to 1 for every process cargo launches, so under cargo test the variable is never absent, and unsetting it in your shell only lets that block re-supply 1: CREW_DISABLE_VENDOR_CLI=0 CREW_LIVE_CWD=<trusted-dir> cargo test --test conformance (see cli-reference.md) — there is no per-adapter live-gate variable, one switch gates all four.
  • Confirm the vendor CLI itself is installed and authenticated — a conformance failure here is usually the vendor CLI, not Crew.

Open implementation gaps (as opposed to operational issues) are tracked outside this repository. Engineering lessons from closed findings — every fix, with the test that proved it — live in engineering-lessons.md.

Dashboard

The dashboard is a live, read-only web interface showing your runs, their status, cost, and transcript. To reach it:

  1. Enable and configure in your crew config (~/.omp/crew.json or .omp/crew.json at the repo root):
{
  "dashboard": { "enabled": true, "port": 4747 }
}
  1. Get the URL and token: When the daemon starts, it prints:
dashboard_started addr=127.0.0.1:4747 url=http://127.0.0.1:4747/?token=8f3c…

The token is per-daemon-run and required for every request. The URL may also appear in /crew health output (depends on your OMP/Crew versions).

  1. Open in your browser using the full URL with token.

Note: Port conflicts: if two daemons try the same port, the second logs dashboard_bind_failed and runs without a dashboard.

Cost and Task Columns

The cost column shows what each run consumed:

  • Dollars ($): Reported only for vendors with billable pricing. Blank otherwise.
  • Tokens: Model tokens consumed, where reported. Blank if unavailable.
  • Dash (—): Shown when neither cost nor tokens are available.
  • Totals: Sum dollars and tokens separately; blank if that column is all blanks.

The task column shows the first line of the first journaled prompt (160 characters, truncated if longer).

Prompt Journaling Privacy

Every prompt you submit, every decision you make, and every model response are recorded in a durable journal (SQLite database in your Crew state directory). This is by design — it lets you replay runs, audit what happened, and recover from crashes.

Before any text reaches the journal, Crew's Redactor passes over it and masks secrets (API keys, tokens, passwords, custom patterns you've configured). The redacted version is what's stored. See ADR-0028 for the redaction design.

Redacted journal content appears in:

  • Session transcripts (what you see as you work)
  • Audit exports (/crew export) — the full journal
  • Event streams (/events and events/replay) — live and replay
  • Dashboard — task prompts (first line, truncated) and run transcripts

To start fresh and erase all runs/journals/state for a repo: delete the Crew state directory (default: ~/.omp/crew/ or the path set in CREW_STATE_DIR). Crew will recreate it empty on the next run.