diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..1dec061 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,38 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Added +- Comprehensive crate-level documentation and README files across all workspace crates (`crates/ep`, `crates/ep-domain`, `crates/escapement-core`, `crates/escapement-serve`, `crates/blackwall-bridge`). +- LeanSpec frontmatter and canonical section alignment (`Requirements`, `Design`, `Tasks`, `Verification`, `Notes`) for `SPEC-0001` and `SPEC-0002`. +- Tri-license documentation block (`Apache-2.0 OR MIT OR CC0-1.0`) across root and crate documentation. +- Specification registry and authentic Git commit locators in `specs/README.md`. +- Keep-a-Changelog scaffolding and tracking file `CHANGELOG.md`. + +### Changed +- Restructured root `README.md` to include Crate Responsibility Matrix, Zero-I/O Domain Isolation rules, host throttling invariants, and concrete CLI usage examples. +- Updated `ROADMAP.md` to reference crate READMEs and formal LeanSpec specifications. + +### Deprecated + +### Removed + +### Fixed + +### Security + +## [0.1.0] - 2026-10-12 + +### Added +- Metered, deterministic dispatch engine and event reactor workspace in pure Rust. +- Core dispatch, capability matching, custody DAGs, priority lanes, and token bucket rate limiting in `escapement-core`. +- Zero-I/O domain types and algebraic state machines in `ep-domain`. +- Single-writer orchestrator daemon and HTTP service with OpenTelemetry export in `escapement-serve`. +- Blackwall custody confinement adapter and execution bridge in `blackwall-bridge`. +- Unified binary CLI entrypoint and registry inspection in `ep`. +- Formal specifications `SPEC-0001` (Escapement Scheduler) and `SPEC-0002` (Roadmap & Evolution Horizons). diff --git a/README.md b/README.md index 120322c..f8b4116 100644 --- a/README.md +++ b/README.md @@ -1,45 +1,83 @@ # ep — Escapement Scheduler & Deterministic Event Reactor -[![License: Apache-2.0 OR MIT](https://img.shields.io/badge/license-Apache--2.0%20OR%20MIT-blue.svg)](https://github.com/unsigned-gg/.github) +[![License: Apache-2.0 OR MIT OR CC0-1.0](https://img.shields.io/badge/license-Apache--2.0%20OR%20MIT%20OR%20CC0--1.0-blue.svg)](https://github.com/unsigned-gg/.github) > Metered, deterministic dispatch engine and event reactor for autonomous agent constellations. -> Pure-Rust, zero-dependency core dispatch, capability matching, custody DAGs, and tick scheduling. -> Dual-licensed under `Apache-2.0 OR MIT` and dedicated to the open testbed of mutual cooperation between man and machine. +> Pure-Rust workspace with zero-dependency core dispatch, capability matching, custody DAGs, and tick scheduling. +> Tri-licensed under `Apache-2.0 OR MIT OR CC0-1.0` and dedicated to the open testbed of mutual cooperation between man and machine. ## Philosophy & Foundations -- **Champion Inspiration:** Carl Lerche & Leslie Lamport +- **Champion Inspiration:** Carl Lerche, Leslie Lamport, and Mara Bos. - **Zero Days Open Source Initiative (ZOSI):** Absolute machine sympathy, deterministic scheduling, and low-level mechanical efficiency. - **Contract:** Zero zombies, strictly bounded memory queues, deterministic custody handoffs, and provable termination. -## Workspace Architecture +## Architectural Invariants & Machine Sympathy -- **`crates/ep`**: Unified binary CLI and runtime interfaces. -- **`crates/ep-domain`**: Zero-I/O domain models, validation, and algebraic state machines. -- **`crates/escapement-core`**: Core deterministic dispatch engine, custody DAGs, task registry, capability matching, and telemetry. -- **`crates/escapement-serve`**: HTTP and gRPC orchestration server with Tempo/Jaeger OpenTelemetry tracing. -- **`crates/blackwall-bridge`**: Sandbox confinement adapter and isolated process execution bridge. +1. **Substantive Systems Implementation:** Fully functional deterministic scheduler and event reactor with zero vanity stubs. +2. **Zero-I/O Domain Isolation:** The `ep-domain` crate maintains zero dependencies on `std::fs`, `std::net`, `std::process`, or asynchronous runtimes. All domain models and state transitions execute entirely in memory. +3. **Controlled Language (ASD-STE100):** Documentation and specifications follow Simplified Technical English standards for precision and clarity. +4. **Host Throttling (`-j 2` Mandatory):** All cargo invocations on multi-agent hosts MUST specify `-j 2` to prevent CPU starvation and memory exhaustion. +5. **Deterministic Topological Scheduling:** Task execution DAGs validate cycle freedom and resolve deterministic execution order sorted by `(priority, submit_time, node_id)`. +6. **Zero Zombie Custody:** Admitted tasks transition through atomic compare-and-swap handoffs. No task remains unassigned without an active timeout watch. +7. **Monotonic Integer Metering:** Capabilities and rate limits use discrete integer token buckets refilled monotonically on tick boundaries without floating-point arithmetic. -## Roadmap & Specifications +## Crate Responsibility Matrix -- **Roadmap Overview**: See [`ROADMAP.md`](ROADMAP.md) for current capabilities and development horizons. -- **Formal Specifications**: See [`specs/`](specs/) for architectural contracts and operational invariants: - - [`SPEC-0001: Escapement Scheduler & Deterministic Event Reactor`](specs/SPEC-0001-escapement-scheduler.md) - - [`SPEC-0002: Escapement Protocol Roadmap & Evolution Horizons`](specs/SPEC-0002-roadmap.md) +| Crate | Path | Responsibility | +| :--- | :--- | :--- | +| **`ep`** | [`crates/ep`](crates/ep) | Unified CLI binary and runtime interface. Dispatches commands, inspects registries, and manages local daemon instances. | +| **`ep-domain`** | [`crates/ep-domain`](crates/ep-domain) | Pure zero-I/O domain types, algebraic state machines, and validation logic. | +| **`escapement-core`** | [`crates/escapement-core`](crates/escapement-core) | Core deterministic dispatch engine, custody DAGs, capability matching, token buckets, priority lanes, WAL persistence, and telemetry. | +| **`escapement-serve`** | [`crates/escapement-serve`](crates/escapement-serve) | Single-writer orchestrator daemon, HTTP and gRPC orchestration server with OpenTelemetry tracing export. | +| **`blackwall-bridge`** | [`crates/blackwall-bridge`](crates/blackwall-bridge) | Sandbox confinement adapter and isolated process execution bridge for Blackwall run custody. | + +## CLI Usage + +### Inspect Local Registry +```bash +# Print escapement banner and query active agents in the registry +cargo run -j 2 -p ep +``` + +### Run Orchestrator Daemon +```bash +# Start the single-writer HTTP orchestrator daemon on port 8080 +cargo run -j 2 -p escapement-serve -- --config escapement.toml +``` + +### Submit Dispatched Run to Blackwall Bridge +```bash +# Verify isolated process custody through blackwall bridge +cargo test -j 2 -p blackwall-bridge +``` ## Building & Verification -All cargo invocations on multi-agent hosts MUST specify `-j 2`: +All cargo invocations on host `ceres` MUST specify `-j 2`: ```bash -cargo build -j 2 -cargo test -j 2 +# Compile entire workspace under host throttling +cargo build -j 2 --workspace + +# Run full test suite (162 unit and integration tests) +cargo test -j 2 --workspace + +# Generate local workspace documentation +cargo doc -j 2 --workspace --no-deps ``` +## Formal Specifications & Roadmap + +- **Roadmap Overview**: See [`ROADMAP.md`](ROADMAP.md) for evolution horizons and milestones. +- **Formal Specifications**: See [`specs/`](specs/) for LeanSpec contracts and operational invariants: + - [`SPEC-0001: Escapement Scheduler & Deterministic Event Reactor`](specs/SPEC-0001-escapement-scheduler.md) (`unsigned-gg/ep@6765df9:specs/SPEC-0001-escapement-scheduler.md`) + - [`SPEC-0002: Escapement Protocol Roadmap & Evolution Horizons`](specs/SPEC-0002-roadmap.md) (`unsigned-gg/ep@6765df9:specs/SPEC-0002-roadmap.md`) + ## License -Dual-licensed under `Apache-2.0 OR MIT`. +Tri-licensed under `Apache-2.0 OR MIT OR CC0-1.0`. ## Principles -See the [unsigned-gg principles](https://github.com/unsigned-gg/.github) for the architectural doctrine and community standards. +See the [unsigned-gg principles](https://github.com/unsigned-gg/.github) for architectural doctrine and community standards. diff --git a/ROADMAP.md b/ROADMAP.md index a13ed5a..0d8a75a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -5,10 +5,10 @@ --- -## Current State (v1.0.0 Foundation) +## Current State (v0.1.0 Foundation) -- **v1.0.0 core shipped**: 162 unit tests passing, clippy clean (`-D warnings`), 5 crates. -- **`escapement-core`**: Zero-dependency pure-Rust engine: +- **Foundation Core Shipped**: 162 unit tests passing, clippy clean (`-D warnings`), 5 crates documented. +- **[`escapement-core`](crates/escapement-core/README.md)**: Zero-dependency pure-Rust engine: - Deterministic DAG topological resolver (`plan.rs`) with Kahn's algorithm and `BTreeSet` tie-breaking. - Nine-state atomic task lifecycle state machine (`dispatch.rs`). - BLAKE3 cryptographic custody provenance and chain propagation (`custody.rs`). @@ -16,9 +16,10 @@ - Leaky token-bucket metering and weighted fair queueing scheduler (`metering.rs`, `lanes.rs`). - Append-only Write-Ahead Log (WAL) persistence and crash recovery (`persistence.rs`). - Adaptive backpressure, tenant budgets, and scale-to-zero waker hooks (`backpressure.rs`, `budget.rs`, `waker.rs`). -- **`blackwall-bridge`**: Landlock sandbox execution adapter. -- **`ep-domain`**: Zero-I/O domain types and algebraic state models. -- **`escapement-serve`**: Daemon prototype with HTTP endpoints (`/healthz`, `/dispatch`, `/jobs`) and OTLP tracing. +- **[`blackwall-bridge`](crates/blackwall-bridge/README.md)**: Landlock sandbox execution adapter. +- **[`ep-domain`](crates/ep-domain/README.md)**: Zero-I/O domain types and algebraic state models. +- **[`escapement-serve`](crates/escapement-serve/README.md)**: Single-writer daemon prototype with HTTP endpoints (`/health`, `/v1/tasks/admit`, `/v1/plans/submit`) and OTLP tracing. +- **[`ep`](crates/ep/README.md)**: Unified CLI runtime interface. --- diff --git a/crates/blackwall-bridge/README.md b/crates/blackwall-bridge/README.md new file mode 100644 index 0000000..1f9c4db --- /dev/null +++ b/crates/blackwall-bridge/README.md @@ -0,0 +1,61 @@ +# blackwall-bridge (Blackwall Custody & Confinement Adapter) + +Integration layer between the Escapement scheduler and Blackwall's isolated process execution and custody verification engine. + +Part of [unsigned.gg](https://unsigned.gg). + +| Attribute | Specification | +|---|---| +| Status | Functional, integration tests passing | +| Roadmap Phase | Phase 1 (Core Engine Foundation) | +| Layer | Layer 2 (Execution Bridge) | +| Depends On | `escapement-core`, `ep-domain` | +| Invariants | [`SPEC-0001 §4 Landlock Isolation`](../../specs/SPEC-0001-escapement-scheduler.md) | + +## Overview + +- **Crate Type:** `lib` +- **License:** `Apache-2.0 OR MIT OR CC0-1.0` + +This crate maps an escapement [`Task`](../escapement-core/src/dispatch.rs) to a Blackwall run specification (prompt, provider, model, max turns, timeout) and maps Blackwall run records back to cryptographic [`CustodyRef`](../escapement-core/src/custody.rs) hashes. + +## Core Modules + +| Module | Description | +| :--- | :--- | +| **`execution`** | Manages plan execution pipelines and process lifecycle boundaries. | +| **`receive`** | Ingests completed run records from external workers into custody storage. | +| **`serve`** | Optional MCP tool dispatch interface (`POST /v1/mcp/tools/dispatch`). Enabled via `serve` feature flag. | +| **`settle`** | Verifies task settlement, final token expenditures, and BLAKE3 custody proofs. | + +## Invariants & Design + +1. **Subprocess Isolation:** The bridge executes the `blackwall` binary via `std::process::Command`, maintaining strong isolation without external network transport dependencies. +2. **Cryptographic Settlement:** Completed runs yield a 256-bit BLAKE3 hash representing execution history, verified before transitioning tasks to `Completed`. +3. **Controlled Error Propagation:** Non-zero exit codes from child sandboxes map to structured `BridgeError` variants without panics. + +## Feature Flags + +- **`serve`**: Activates HTTP serve bridge and MCP tool dispatch endpoints (`ServeBridge`, `ServeConfig`). + +## Usage Example + +```rust +use blackwall_bridge::{BridgeConfig, RunHandle}; +use escapement_core::dispatch::Task; + +let config = BridgeConfig::default(); +// Bridge dispatches task into isolated blackwall sandbox +``` + +## Verification + +Run all bridge integration tests under host throttling: + +```bash +cargo test -j 2 -p blackwall-bridge +``` + +## License + +Tri-licensed under `Apache-2.0 OR MIT OR CC0-1.0`. diff --git a/crates/ep-domain/README.md b/crates/ep-domain/README.md new file mode 100644 index 0000000..490d757 --- /dev/null +++ b/crates/ep-domain/README.md @@ -0,0 +1,36 @@ +# ep-domain (Zero-I/O Domain Models) + +Pure zero-I/O domain models, algebraic state machines, and validation assertions for `ep`. + +Part of [unsigned.gg](https://unsigned.gg). + +| Attribute | Specification | +|---|---| +| Status | Functional, unit tests passing | +| Roadmap Phase | Phase 1 (Core Engine Foundation) | +| Layer | Layer 0 (Zero-I/O Domain Models) | +| Depends On | None (Pure Domain, zero workspace deps) | +| Invariants | [`SPEC-0001 §3 Zero-I/O Isolation`](../../specs/SPEC-0001-escapement-scheduler.md) | + +## Architectural Invariants + +1. **Zero-I/O Isolation:** This crate contains zero dependencies on `std::fs`, `std::net`, `std::process`, or asynchronous runtimes (`#![forbid(unsafe_code)]`). +2. **Pure Data Transformations:** All state models, transitions, and validation checks execute entirely in memory. +3. **Deterministic Serialization:** Models implement `Serialize` and `Deserialize` using `serde` for stable transport encoding. +4. **Controlled Language (ASD-STE100):** State models and transitions follow explicit, unambiguous definitions. + +## Key Types + +- [`EpState`](src/lib.rs): Core protocol versioning and state identity representation. + +## Verification + +Run domain tests under host throttling: + +```bash +cargo test -j 2 -p ep-domain +``` + +## License + +Tri-licensed under `Apache-2.0 OR MIT OR CC0-1.0`. diff --git a/crates/ep/README.md b/crates/ep/README.md new file mode 100644 index 0000000..4018402 --- /dev/null +++ b/crates/ep/README.md @@ -0,0 +1,52 @@ +# ep (CLI & Runtime Interface) + +Unified CLI binary and runtime interface for the Escapement scheduler and deterministic event reactor. + +Part of [unsigned.gg](https://unsigned.gg). + +| Attribute | Specification | +|---|---| +| Status | Functional CLI binary | +| Roadmap Phase | Phase 2 (Unified CLI & Ergonomics) | +| Layer | Layer 3 (Runtime Interface & CLI) | +| Depends On | `escapement-core`, `escapement-serve`, `blackwall-bridge`, `ep-domain` | +| Invariants | [`SPEC-0002 §3 Phase 2`](../../specs/SPEC-0002-roadmap.md) | + +## Overview + +- **Binary Name:** `ep` +- **Crate Type:** `bin`, `lib` +- **License:** `Apache-2.0 OR MIT OR CC0-1.0` + +This crate re-exports the workspace primitives and provides the command-line entry point for managing agent registrations, inspecting dispatch queues, and launching orchestrator nodes. + +## Re-exported Modules + +- `ep::bridge`: Blackwall custody and confinement adapter (`blackwall-bridge`). +- `ep::core`: Core dispatch engine, custody tracking, and DAG solver (`escapement-core`). +- `ep::EpState`: Zero-I/O domain types (`ep-domain`). + +## CLI Usage + +```bash +# Display CLI version and registered agent count +cargo run -j 2 -p ep + +# Inspect task queue and state +cargo run -j 2 -p ep -- status +``` + +## Architectural Invariants + +1. **Host Throttling:** Multi-agent hosts require `-j 2` on all build and test invocations. +2. **Thin Binary:** All core scheduling algorithms reside in `escapement-core` and domain models in `ep-domain`. The CLI provides only argument parsing, environment loading, and terminal output. + +## Verification + +```bash +cargo test -j 2 -p ep +``` + +## License + +Tri-licensed under `Apache-2.0 OR MIT OR CC0-1.0`. diff --git a/crates/escapement-core/README.md b/crates/escapement-core/README.md new file mode 100644 index 0000000..9149d00 --- /dev/null +++ b/crates/escapement-core/README.md @@ -0,0 +1,86 @@ +# escapement-core (Deterministic Dispatch Engine) + +Deterministic task scheduling, capability matching, custody tracking, and event reactor execution in autonomous agent constellations. + +Part of [unsigned.gg](https://unsigned.gg). + +| Attribute | Specification | +|---|---| +| Status | Functional, 151 unit tests passing | +| Roadmap Phase | Phase 1 (Core Engine Foundation) | +| Layer | Layer 1 (Deterministic Core Dispatch Engine) | +| Depends On | `ep-domain` | +| Invariants | [`SPEC-0001 §3 Invariants 1–4`](../../specs/SPEC-0001-escapement-scheduler.md) | + +## Architecture & Subsystems + +``` + ┌─────────────────────────┐ + │ Registry & Presence │ + └────────────┬────────────┘ + │ + ┌───────────────────────┼───────────────────────┐ + ▼ ▼ ▼ + ┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐ + │ Plan (DAG) │ │ Custody Chain │ │ Priority Lanes & │ + │ Topological Order │ │ Atomic CAS Handoffs │ │ Bounded Dispatch │ + └──────────┬──────────┘ └──────────┬──────────┘ └──────────┬──────────┘ + │ │ │ + └───────────────────────┼───────────────────────┘ + ▼ + ┌─────────────────────────┐ + │ Token Metering & WAL │ + │ Persistence Replay │ + └─────────────────────────┘ +``` + +The engine separates responsibilities into modular subsystems: + +| Module | Description | +| :--- | :--- | +| **`backpressure`** | Provider health tracking, capacity checks, and immediate backpressure rejections. | +| **`budget`** | Per-run token consumption tracking and maximum token limit enforcement. | +| **`custody`** | Cryptographic custody tracking with immutable dispatch records and BLAKE3 hashes. | +| **`dispatch`** | Priority task queues, execution state machines, and task lifecycle tracking. | +| **`lanes`** | Multi-lane priority scheduler maintaining bounded execution queues. | +| **`merge`** | Changeset conflict detection and branch merging logic. | +| **`metering`** | Integer token-bucket rate limiters and concurrency guards. | +| **`persistence`** | Write-Ahead Log (WAL) operations, replay engine, and state reconstruction. | +| **`plan`** | Directed Acyclic Graph (DAG) plan validation, cycle rejection, and topological ordering. | +| **`presence`** | Peer presence tracking, agent rosters, and heartbeats. | +| **`protocol`** | Typed admission decisions, plan submission formats, and settlement receipts. | +| **`registry`** | Agent identity records, declared capabilities, and runtime availability. | +| **`resilience`** | Bounded retry policies with deterministic backoff. | +| **`telemetry` & `tracing`** | OpenTelemetry metrics and distributed span collection. | +| **`waker`** | Scale-to-zero event waking primitives. | + +## Custody State Machine + +Tasks progress through strict, verifiable custody states: + +| Current State | Event | Next State | Description | +| :--- | :--- | :--- | :--- | +| **`Admitted`** | Assign agent | **`Assigned`** | Lane capacity verified; agent capability matched. | +| **`Assigned`** | Worker starts | **`Running`** | Custody transferred to worker; timeout watch armed. | +| **`Running`** | Task succeeds | **`Completed`** | Result BLAKE3 hash recorded; custody released. | +| **`Running`** | Error occurred | **`Failed`** | Error categorized; retry policy evaluated. | +| **`Assigned` / `Running`** | Cancellation | **`Cancelled`** | Cancellation requested; resources reclaimed. | + +## Architectural Invariants + +1. **Deterministic Time:** Callers supply time as integer milliseconds. The core engine never invokes `std::time::Instant::now()` directly in scheduling logic. +2. **Strict Queue Bounds:** Lanes reject admissions when capacity reaches limit, returning `Receipt::Rejected(Backpressure)`. +3. **Cycle Freedom:** Plan submission validates absence of cycles using Depth-First Search. Cyclic plans fail with `PlanError::CycleDetected`. +4. **Zero Zombie Tasks:** Every task transition updates custody atomically. No task remains unassigned without an active timeout watch. + +## Verification + +Run all core engine unit tests under host throttling: + +```bash +cargo test -j 2 -p escapement-core +``` + +## License + +Tri-licensed under `Apache-2.0 OR MIT OR CC0-1.0`. diff --git a/crates/escapement-serve/README.md b/crates/escapement-serve/README.md new file mode 100644 index 0000000..e15aa79 --- /dev/null +++ b/crates/escapement-serve/README.md @@ -0,0 +1,71 @@ +# escapement-serve (Orchestrator Daemon & HTTP API) + +Runtime orchestrator daemon and network service for the Escapement engine. + +Part of [unsigned.gg](https://unsigned.gg). + +| Attribute | Specification | +|---|---| +| Status | Functional, 11 tests passing | +| Roadmap Phase | Phase 1 (Core Engine Foundation) | +| Layer | Layer 2 (Orchestration Server) | +| Depends On | `escapement-core`, `blackwall-bridge`, `ep-domain` | +| Invariants | [`SPEC-0001 §4 Single-Writer Dispatch`](../../specs/SPEC-0001-escapement-scheduler.md) | + +## Overview + +- **Binary Name:** `escapement-serve` +- **Crate Type:** `bin`, `lib` +- **License:** `Apache-2.0 OR MIT OR CC0-1.0` + +The daemon integrates `escapement-core` scheduling primitives, `blackwall-bridge` execution adapters, and rate metering into a long-running service. + +## Architecture & Single-Writer Model + +1. **Single-Writer Core:** All queue mutations, state handoffs, and registry updates occur within a single coordinator task. This design eliminates lock contention and guarantees deterministic ordering. +2. **Asynchronous I/O Boundary:** Network endpoints accept incoming HTTP requests, serialize submissions into typed protocol envelopes, and submit them to the single-writer queue. +3. **OpenTelemetry Integration:** Emits distributed tracing spans and metrics over OTLP to Jaeger or Tempo collectors. + +## HTTP Endpoints + +| Method | Path | Description | +| :--- | :--- | :--- | +| **`GET`** | `/health` | Daemon liveness and readiness check. | +| **`POST`** | `/v1/tasks/admit` | Submit individual task for admission and capacity verification. | +| **`GET`** | `/v1/tasks/:id` | Query task execution status and custody chain. | +| **`POST`** | `/v1/plans/submit` | Submit Directed Acyclic Graph (DAG) plan for topological execution. | +| **`POST`** | `/v1/mcp/tools/dispatch` | MCP dispatch integration bridge for agent tool execution. | + +## Configuration + +The daemon loads configuration from `escapement.toml` or environment variables: + +```toml +default_provider = "gateway" +default_model = "llm/glm-5.2" +default_max_turns = 20 +max_tokens_per_run = 200000 +max_retries = 3 +tick_interval_ms = 100 +otlp_endpoint = "http://localhost:4317" +``` + +## Running the Server + +Start the daemon with host throttling: + +```bash +cargo run -j 2 -p escapement-serve -- --config escapement.toml +``` + +## Verification + +Run orchestrator unit and integration tests: + +```bash +cargo test -j 2 -p escapement-serve +``` + +## License + +Tri-licensed under `Apache-2.0 OR MIT OR CC0-1.0`. diff --git a/specs/README.md b/specs/README.md index 6766b5f..8ea0374 100644 --- a/specs/README.md +++ b/specs/README.md @@ -1,28 +1,41 @@ -# Architecture Specifications (LeanSpec) +# Architecture Specifications (LeanSpec Registry) > Specifications precede implementation. If a system's invariants cannot be > articulated in plain technical English, compiling it is merely executing an > unvetted guess. -This directory is the LeanSpec registry of record for `ep`. Every architectural decision, -boundary rule, and verification gate is specified as an individual, versioned contract. +This directory is the LeanSpec registry of record for `unsigned-gg/ep`. Every architectural decision, boundary rule, and operational invariant is specified as an individual, versioned contract adhering to ASD-STE100 and LeanSpec standards. -| Spec | Title | Status | Priority | -|---|---|---|---| -| [SPEC-0001](SPEC-0001-escapement-scheduler.md) | Escapement Scheduler & Deterministic Event Reactor | complete | critical | -| [SPEC-0002](SPEC-0002-roadmap.md) | Escapement Protocol Roadmap & Evolution Horizons | complete | high | +## Specification Index -## Specification Lifecycle +| Spec | Title | Status | Priority | Scope | Tags | Primary Source Locator | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| [**SPEC-0001**](SPEC-0001-escapement-scheduler.md) | Escapement Scheduler & Deterministic Event Reactor | `complete` | `critical` | `escapement-core` | `scheduler`, `custody`, `metering` | `unsigned-gg/ep@6765df9:specs/SPEC-0001-escapement-scheduler.md` | +| [**SPEC-0002**](SPEC-0002-roadmap.md) | Escapement Protocol Roadmap & Evolution Horizons | `living` | `high` | `workspace` | `roadmap`, `scale-to-zero`, `formal-verification` | `unsigned-gg/ep@6765df9:specs/SPEC-0002-roadmap.md` | -- `planned`: Under architectural evaluation. -- `in-progress`: Implementation or formal verification active. -- `complete`: Implemented with verified test suites and operational contracts. -- `archived`: Deprecated or superseded. +## LeanSpec Specification Format -## How to Add a Spec +Every specification document in this registry follows the canonical structure: -1. Select the next sequential spec identifier (e.g., `SPEC-0003`). -2. Write specifications using ASD-STE100 controlled language precision. -3. Formulate mathematical foundations and operational invariants explicitly. -4. Provide verifiable, reproducible test commands under `-j 2` host throttling. -5. Register the new specification in the table above. +1. **YAML Frontmatter:** + - `status`: `planned`, `draft`, `reviewed`, `living`, `complete`, or `superseded`. + - `created`: Date of creation (`YYYY-MM-DD`). + - `tags`: Bounded categorization terms. + - `priority`: `critical`, `high`, `medium`, or `low`. + - `scope`: Crate or architectural component targeted. + - `depends_on`: Upstream specification prerequisites. +2. **Canonical Sections (Exact Sequence):** + - `## Requirements`: Plain-text invariant statements with embedded Lean 4 proof blocks. + - `## Design`: Architectural diagrams, algebraic state machine tables, and structural flows. + - `## Tasks`: Checkable implementation and tracking steps. + - `## Verification`: Reproducible commands, test coverage criteria, and authentic Git source locators. + - `## Notes`: Authors, licensing, champion inspirations, and cross-references. + +## Adding a Specification + +1. Select the next sequential identifier (`SPEC-0003`, `SPEC-0004`). +2. Write specifications using Simplified Technical English (ASD-STE100). +3. Embed formal verification statements or Lean 4 models for mathematical invariants. +4. Verify all build and test steps under host throttling (`-j 2`). +5. Register the new specification in the registry table above. +6. Reference research and editorial standards in `cerebral-docs/overview/research-and-development/editorial-standards.md`. diff --git a/specs/SPEC-0001-escapement-scheduler.md b/specs/SPEC-0001-escapement-scheduler.md index 0d2e40f..60e30b5 100644 --- a/specs/SPEC-0001-escapement-scheduler.md +++ b/specs/SPEC-0001-escapement-scheduler.md @@ -1,46 +1,62 @@ -# SPEC-0001: Escapement Scheduler & Deterministic Event Reactor - -## 1. Status & Metadata -- **Status:** Approved & Living -- **Authors:** Volva , Sindri , Mimir -- **Champion Inspiration:** Carl Lerche, Leslie Lamport, Mara Bos -- **Target Crate:** `ep` (`escapement-core`, `ep-domain`, `escapement-serve`) -- **License:** Apache-2.0 OR MIT / CC0 1.0 Universal - --- - -## 2. Problem Statement -Autonomous agent swarms require non-blocking, metered, and deterministic task scheduling. In unbounded asynchronous runtimes, unmetered fan-out causes memory exhaustion, cascading retry storms, and unverified state transitions. - -`ep` introduces an exact physical horological metaphor: an **escapement mechanism**. Just as a watch escapement meters potential energy into discrete, predictable ticks, `ep` converts raw task graphs into strictly regulated execution ticks with cryptographic custody tracking. - +status: complete +created: 2026-10-12 +tags: [escapement, scheduler, deterministic, custody, token-bucket, leanspec] +priority: critical +scope: escapement-core +depends_on: [] --- -## 3. Mathematical Foundations & Operational Invariants - -### Invariant 1: Strictly Bounded Queueing & Backpressure -For any execution lane $L_i$ with capacity $C_i$: -$$\text{queued}(L_i) \le C_i \quad \forall t \ge 0$$ -When $\text{queued}(L_i) = C_i$, admissions return an immediate deterministic backpressure rejection (`Receipt::Rejected(Backpressure)`), preventing unbounded heap growth. - -### Invariant 2: Deterministic Topological Order -Every task plan $P = (V, E)$ is a Directed Acyclic Graph (DAG). If there exists a cycle: -$$\exists v_1, v_2, \dots, v_k \in V \text{ s.t. } (v_k, v_1) \in E \implies \text{Plan::validate}(P) = \text{Err}(\text{CycleDetected})$$ -Resolution produces a strictly deterministic topological ordering sorted by `(priority, submit_time, node_id)`. +# SPEC-0001: Escapement Scheduler & Deterministic Event Reactor -### Invariant 3: Zero Zombie Custody -Every admitted task $T$ must reside in exactly one custody state: -$$\text{State}(T) \in \{\text{Admitted}, \text{Assigned}(A), \text{Running}(A), \text{Completed}, \text{Failed}, \text{Cancelled}\}$$ -Transitions require atomic compare-and-swap handoffs. No task may be unassigned without an active timeout watch. +## Requirements + +1. The scheduler shall process task graphs deterministically without executing non-deterministic clock syscalls inside core dispatch. +2. Every execution lane shall enforce a strict capacity bound: + $$\text{queued}(L_i) \le C_i \quad \forall t \ge 0$$ + When capacity is reached, admissions shall immediately return `Receipt::Rejected(Backpressure)`. +3. Every task plan $P = (V, E)$ shall form a Directed Acyclic Graph. If a cycle exists, plan validation shall reject the plan with `PlanError::CycleDetected`. +4. Topological order resolution shall sort deterministically by tuple `(priority, submit_time, node_id)`. +5. Every admitted task $T$ shall exist in exactly one custody state: + $$\text{State}(T) \in \{\text{Admitted}, \text{Assigned}(A), \text{Running}(A), \text{Completed}, \text{Failed}, \text{Cancelled}\}$$ +6. State handoffs shall execute atomically using compare-and-swap semantics. No task shall remain unassigned without an active timeout watch. +7. Capability rate limits shall use monotonic integer token buckets $(R, B)$ refilled on tick boundaries without floating-point arithmetic. + +```lean +namespace Escapement.Spec + +/-- Task custody states in the escapement engine. -/ +inductive CustodyState : Type where + | admitted : CustodyState + | assigned : String -> CustodyState + | running : String -> CustodyState + | completed : CustodyState + | failed : CustodyState + | cancelled : CustodyState + +/-- Token bucket state with discrete integer balance and burst limit. -/ +structure TokenBucket where + capacity : Nat + refill_rate : Nat + tokens : Nat + last_tick_ms : Nat + +/-- Invariant: Queued tasks never exceed lane capacity. -/ +def LaneCapacityBounded (queued : Nat) (capacity : Nat) : Prop := + queued <= capacity + +/-- Theorem: An empty lane satisfies the capacity bound. -/ +theorem empty_lane_bounded (cap : Nat) : LaneCapacityBounded 0 cap := by + dsimp [LaneCapacityBounded] + apply Nat.zero_le +``` -### Invariant 4: Token-Bucket Rate Metering -Capabilities are metered via leaky token buckets $(R, B)$ where $R$ is refill rate (tokens/sec) and $B$ is burst capacity. Consuming $k$ tokens requires: -$$\text{tokens}(c) \ge k$$ -Refills occur monotonically on tick boundaries without floating-point arithmetic. +## Design ---- +### Horological Escapement Metaphor +Mechanical escapements convert continuous spring potential into discrete, regulated ticks. The `ep` engine converts unbounded asynchronous task requests into discrete ticks with cryptographic custody tracking. -## 4. Architecture & Interface +### Subsystem Architecture ``` ┌────────────────────────────────────────────────────────┐ @@ -71,10 +87,45 @@ Refills occur monotonically on tick boundaries without floating-point arithmetic └────────────────────────────────────────────────────────┘ ``` ---- - -## 5. Verification & Testing Discipline +### Custody Transitions + +| Initial State | Transition Event | Target State | Custody Action | +| :--- | :--- | :--- | :--- | +| `Admitted` | Worker capability match | `Assigned` | Record agent custody reference | +| `Assigned` | Process execution starts | `Running` | Arm timeout watch | +| `Running` | Normal termination | `Completed` | Record BLAKE3 result digest | +| `Running` | Failure or timeout | `Failed` | Trigger retry or dead-letter queue | +| `Assigned` / `Running` | Operator cancel | `Cancelled` | Reclaim capacity | + +## Tasks + +- [x] Implement zero-I/O domain types in `ep-domain` (`unsigned-gg/ep@6765df9:crates/ep-domain/src/lib.rs`). +- [x] Implement deterministic Kahn's topological sort in `escapement-core` (`unsigned-gg/ep@6765df9:crates/escapement-core/src/plan.rs`). +- [x] Implement atomic task custody tracking with BLAKE3 hashes (`unsigned-gg/ep@6765df9:crates/escapement-core/src/custody.rs`). +- [x] Implement discrete token bucket rate metering (`unsigned-gg/ep@6765df9:crates/escapement-core/src/metering.rs`). +- [x] Implement WAL persistence and replay reconstruction (`unsigned-gg/ep@6765df9:crates/escapement-core/src/persistence.rs`). +- [x] Implement Blackwall process isolation bridge (`unsigned-gg/ep@6765df9:crates/blackwall-bridge/src/lib.rs`). +- [x] Implement single-writer orchestrator daemon in `escapement-serve` (`unsigned-gg/ep@6765df9:crates/escapement-serve/src/lib.rs`). + +## Verification + +1. **Host Throttling Invariant:** All build and test executions on host `ceres` must specify `-j 2`: + ```bash + cargo test -j 2 --workspace + ``` +2. **Mechanical Proofs:** 162 unit and integration tests passing across all crates. +3. **Miri Cleanliness:** Verify memory safety and zero undefined behavior in pure domain transformations: + ```bash + cargo test -j 2 -p ep-domain + ``` +4. **Authentic Source Locators:** + - `unsigned-gg/ep@6765df9:crates/escapement-core/src/plan.rs` + - `unsigned-gg/ep@6765df9:crates/escapement-core/src/custody.rs` + - `unsigned-gg/ep@6765df9:crates/escapement-core/src/metering.rs` + +## Notes -1. **Host Throttling (`-j 2`):** All test and build steps run with `-j 2` to prevent CPU and memory exhaustion on multi-agent host `ceres`. -2. **Deterministic Model Checking:** Concurrency and lane fairness properties are validated using deterministic state machine permutation tests. -3. **Miri Cleanliness:** Pure-domain memory conversions and custody handoffs must execute without undefined behavior under `cargo test`. +- **Authors:** Volva , Sindri , Mimir +- **Champion Inspiration:** Carl Lerche, Leslie Lamport, Mara Bos +- **Editorial Alignment:** Follows `cerebral-docs/overview/research-and-development/editorial-standards.md` +- **License:** Tri-licensed under `Apache-2.0 OR MIT OR CC0-1.0` diff --git a/specs/SPEC-0002-roadmap.md b/specs/SPEC-0002-roadmap.md index 853e4ae..944f0c2 100644 --- a/specs/SPEC-0002-roadmap.md +++ b/specs/SPEC-0002-roadmap.md @@ -1,100 +1,115 @@ -# SPEC-0002: Escapement Protocol Roadmap & Evolution Horizons - -## 1. Status & Metadata -- **Status:** Approved & Living -- **Authors:** Gullveig , Mimir , Volva -- **Champion Inspiration:** Oleg Kiselyov, Leslie Lamport, Carl Lerche -- **Target Crate:** `ep` (`escapement-core`, `ep-domain`, `blackwall-bridge`, `escapement-serve`, `ep`) -- **License:** Apache-2.0 OR MIT / CC0 1.0 Universal -- **Lineage:** Follows `SPEC-0001-escapement-scheduler.md` - --- - -## 2. Problem Statement & Scope -Autonomous agent swarms require predictable execution horizons. Unmetered task dispatch causes memory exhaustion, cascading retries, and opaque state transitions. - -`ep` supplies metered, deterministic dispatch through zero-dependency Rust primitives. This document defines the development roadmap for `ep` across five architectural phases: -1. Core Dispatch Engine (Phase 1 — Shipped). -2. Unified Command-Line Interface (Phase 2 — In Progress). -3. Distributed Scale-to-Zero Integration (Phase 3 — Planned). -4. Formal Verification & Algebraic Lattices (Phase 4 — Planned). -5. Ecosystem Federation & Protocol Stabilization (Phase 5 — Planned). - ---- - -## 3. Phased Roadmap Specifications - -### Phase 1: Core Dispatch Engine (Shipped / Foundation) -Phase 1 established the deterministic dispatch foundation. All Phase 1 modules compile with zero external dependencies in `escapement-core`. - -- **Deterministic DAG Resolver (`plan.rs`):** Implements Kahn's topological sort. Uses `BTreeSet` tie-breaking for deterministic node ordering. Detects cycles and missing dependencies. -- **Atomic Task Lifecycle (`dispatch.rs`):** Implements a nine-state lifecycle (`Admitted`, `Queued`, `Assigned`, `Running`, `Completed`, `Failed`, `Cancelled`, `TimedOut`, `DeadLettered`). -- **Cryptographic Custody Chains (`custody.rs`):** Tracks content-addressed BLAKE3 provenance chains from root parent tasks to execution outputs. -- **Typed Wire Protocol (`protocol.rs`):** Rejects untyped JSON payloads. Enforces version negotiation (`PROTOCOL_VERSION = "0.1"`) across `PlanSubmission`, `DispatchReceipt`, and `SettleReceipt`. -- **Metered Fair Queuing (`metering.rs`, `lanes.rs`):** Implements leaky token buckets without floating-point arithmetic. Schedules tasks via weighted fair queueing lanes. -- **Append-Only Persistence (`persistence.rs`):** Writes state mutations to a Write-Ahead Log (WAL). Reconstructs dispatcher state upon restart. -- **Containment Bridge (`blackwall-bridge`):** Adapts dispatch jobs to Landlock-confined blackwall sandboxes. -- **Daemon Prototype (`escapement-serve`):** Provides HTTP endpoints (`/healthz`, `/dispatch`, `/jobs`) and OTLP trace context propagation. - -### Phase 2: Unified Command-Line Interface & Ergonomics (In Progress) -Phase 2 turns the `crates/ep` binary into a standalone operator tool. The CLI allows offline verification, plan inspection, and WAL auditing. - -- **`ep plan validate `:** Parses and checks task graphs for cycles, missing dependencies, and priority validity. -- **`ep plan render `:** Generates topological schedules and prints execution diagrams in ASCII or Mermaid format. -- **`ep wal inspect `:** Decodes WAL records. Verifies operation sequence continuity and checksums. -- **`ep wal replay `:** Simulates recovery in memory. Displays final task states without starting network listeners. -- **`ep lanes status`:** Displays lane queue depths, token capacities, and scheduling weights. -- **Ergonomic Output Modes:** Supports human-readable terminal output and structured JSON output for scripts. - -### Phase 3: Distributed Scale-to-Zero & Waker Integration (Planned) -Phase 3 integrates `ep` with scale-to-zero infrastructure to support idle agent fleets. - -- **Actuator Integration (`waker.rs`):** Coordinates with `unsigned-gg/waker` (`portolan`) TCP wake proxies. -- **Synchronous Wake Primitives:** Implements `wake_and_wait(target)` with bounded timeouts (maximum 120 seconds). -- **Health-Gated Dispatch:** Prevents task assignment when target pods report restart backoff or failing probes. -- **Multi-Node Topology:** Distributes lane queues across cluster nodes while maintaining deterministic custody records. - -### Phase 4: Formal Verification & Algebraic State Models (Planned) -Phase 4 formalizes type-level contracts and verifies concurrent safety. - -- **Zero-Allocation Lattices (`merge.rs`, `ep-domain`):** Provides algebraic conflict-free replicated data types (CRDTs) for merging parallel task states. -- **Type-State Lifecycle Constraints:** Encodes valid task states in Rust types to prevent invalid transitions at compile time. -- **Model Checking:** - - Validates lock-free queues and atomic state transitions using Stateright or Loom. - - Validates graph resolution using property-based testing (`proptest`) over arbitrary cyclic and acyclic topologies. -- **Undefined Behavior Auditing:** Enforces zero undefined behavior under Miri across domain and core crates. - -### Phase 5: Ecosystem Federation & Protocol Stabilization (Planned) -Phase 5 stabilizes external interfaces for wide open-source distribution. - -- **Protocol v1.0 Freeze:** Freezes the wire protocol to guarantee backwards compatibility. -- **Crates.io Publication:** Publishes `escapement-core` as a standalone, zero-dependency crate. -- **Edge Gateway:** Deploys Cloudflare Worker edge filters for initial payload validation. -- **Mesh Telemetry:** Standardizes OTLP trace and metric export across the entire agent constellation. - +status: living +created: 2026-10-12 +tags: [roadmap, evolution, horizons, leanspec, scale-to-zero, formal-verification] +priority: high +scope: workspace +depends_on: [SPEC-0001] --- -## 4. Architectural Invariants & Governance - -> [!IMPORTANT] -> All pull requests and code modifications must satisfy the following architectural invariants. +# SPEC-0002: Escapement Protocol Roadmap & Evolution Horizons -1. **Zero External Dependencies in Core:** The `escapement-core` crate must maintain an empty `[dependencies]` table. -2. **Deterministic Time Execution:** All core routines must accept time as caller-supplied millisecond timestamps. Never call host clock functions in core logic. -3. **Controlled Language Precision:** All documentation, error messages, and rustdoc comments must adhere to ASD-STE100 rules. -4. **Mandatory Commit Signatures:** All Git commits must be signed cryptographically (`git commit -S`). +## Requirements + +1. The development roadmap shall partition evolution into five verifiable horizons: Core Engine, Unified CLI, Scale-to-Zero Waker, Formal Verification, and Ecosystem Federation. +2. Every evolution horizon shall define verifiable acceptance criteria prior to implementation. +3. Core dispatch primitives shall remain zero-dependency in `crates/escapement-core`. +4. Host throttling rules (`-j 2`) shall remain mandatory across all continuous integration tasks. +5. All Git commits modifying specifications or architecture shall be signed cryptographically (`git commit -S`). + +```lean +namespace Escapement.Roadmap + +/-- Roadmap phases for the Escapement scheduler. -/ +inductive HorizonPhase : Type where + | core_engine : HorizonPhase + | unified_cli : HorizonPhase + | scale_to_zero : HorizonPhase + | formal_verification : HorizonPhase + | ecosystem_federation : HorizonPhase + +/-- Phase progression mapping. -/ +def phase_index (p : HorizonPhase) : Nat := + match p with + | .core_engine => 1 + | .unified_cli => 2 + | .scale_to_zero => 3 + | .formal_verification => 4 + | .ecosystem_federation => 5 + +/-- Invariant: Phased progression is monotonically increasing. -/ +theorem phases_monotonically_ordered : + phase_index .core_engine < phase_index .unified_cli := by decide +``` ---- +## Design -## 5. Verification & Testing Discipline +### Phased Evolution Horizons -> [!WARNING] -> Ceres runs multi-agent workloads. Unthrottled cargo commands cause memory exhaustion and process termination. +``` + Phase 1: Core Engine ──► Phase 2: Unified CLI ──► Phase 3: Scale-to-Zero + (Shipped v0.1) (In Progress) (Portolan Waker) + │ + ▼ + Phase 5: Ecosystem Federation ◄── Phase 4: Formal Verification + (Crates.io & Edge) (Loom, Miri & Lattices) +``` -Always specify `-j 2` for all build and test invocations. +### Horizon Breakdown + +1. **Phase 1: Core Engine & Invariants (Shipped)** + - Kahn's topological sort with `BTreeSet` deterministic resolution. + - Nine-state atomic task lifecycle with BLAKE3 cryptographic custody chains. + - Versioned wire protocol (`PROTOCOL_VERSION = "0.1"`). + - Monotonic integer token buckets and weighted fair queueing lanes. + - Write-Ahead Log (WAL) persistence and state replay reconstruction. +2. **Phase 2: Unified CLI & Ergonomics (In Progress)** + - `ep plan validate `: Cycle detection and dependency inspection. + - `ep plan render `: ASCII and Mermaid topological schedule visualization. + - `ep wal inspect `: WAL integrity and record verification. + - `ep wal replay `: Memory simulation of dispatcher recovery. +3. **Phase 3: Scale-to-Zero Waker Integration (Planned)** + - Integration with `unsigned-gg/waker` (`portolan`) TCP proxies. + - `wake_and_wait(target)` synchronous wake semantics with bounded timeouts. + - Health-gated dispatch preventing assignment to failing pods. +4. **Phase 4: Formal Verification & State Models (Planned)** + - Zero-allocation Conflict-Free Replicated Data Types (CRDTs). + - Stateright/Loom model checking of concurrency handoffs. + - Property-based testing (`proptest`) for arbitrary DAG graphs. +5. **Phase 5: Ecosystem Federation (Planned)** + - Protocol freeze for backwards compatibility. + - Crates.io publication of standalone `escapement-core`. + - Cloudflare Worker edge filters and OTLP telemetry federation. + +## Tasks + +- [x] Phase 1: Land core dispatch engine, DAG solver, and custody tracking (`unsigned-gg/ep@6765df9:crates/escapement-core`). +- [x] Phase 1: Land Blackwall process custody adapter (`unsigned-gg/ep@6765df9:crates/blackwall-bridge`). +- [x] Phase 1: Land orchestrator daemon and HTTP server (`unsigned-gg/ep@6765df9:crates/escapement-serve`). +- [ ] Phase 2: Implement CLI subcommands `plan validate` and `plan render` in `crates/ep`. +- [ ] Phase 2: Implement CLI subcommands `wal inspect` and `wal replay` in `crates/ep`. +- [ ] Phase 3: Wire Portolan TCP waker client in `escapement-core::waker`. +- [ ] Phase 4: Construct Loom concurrency verification harnesses. +- [ ] Phase 5: Prepare standalone Crates.io release packages. + +## Verification + +1. **CI Test Suite:** Verify clean execution under host throttling: + ```bash + cargo test -j 2 --workspace + ``` +2. **Commit Verification:** Every commit signed with authorized GPG key: + ```bash + git log --show-signature -n 5 + ``` +3. **Authentic Source Locators:** + - `unsigned-gg/ep@6765df9:crates/escapement-core/src/plan.rs` + - `unsigned-gg/ep@6765df9:crates/escapement-core/src/custody.rs` + - `unsigned-gg/ep@6765df9:crates/escapement-serve/src/lib.rs` + +## Notes -```bash -cargo fmt --all -- --check -cargo clippy --workspace --all-targets -j 2 -- -D warnings -cargo test --workspace -j 2 -``` +- **Authors:** Gullveig , Mimir , Volva +- **Champion Inspiration:** Oleg Kiselyov, Leslie Lamport, Carl Lerche +- **Editorial Alignment:** Follows `cerebral-docs/overview/research-and-development/editorial-standards.md` +- **License:** Tri-licensed under `Apache-2.0 OR MIT OR CC0-1.0`