Repository navigation
Modalities, interfaces, Jinja and turns, MCP schemas from the server, models chosen by evals - #3
Merged
Conversation
…mas from the server, models chosen by evals, a pinned model's settings held to what it takes docs/96-MODALITIES-INTERFACES-TEMPLATES-TURNS.md designs and records each part: - §A modalities: `documents` and `video` shapes; six model kinds held per line by `model-kind:` (modelkinds.rs); `agent.voice:` (voice.rs) with a named transcriber and speaker keeping a caller's voice on the machine behind a hosted model (egress.rs, `plays-role:`); `knowledge.embedded-by:` / `reranked-by:`, which make meaning search writable air-gapped. - §B abstraction: `interface:`, `implements:`, `chosen-by:` on tools, skills and knowledge, an implementation limited to `calls:` / `passes:` and asking more of a person, never less (interfaces.rs, fourteen rules), then written out longhand; `tool.category:`. - §B4 models chosen by evals: `needs.accept-when:` (accept.rs, spec/accept-when.yaml held by both ports), named `benchmarks:` in the catalogue (eleven run by DeepEval, six by their own harness), `metric.name:`, `needs.prefer:`, `model.benchmarks:` and `speed:` (`needs.scores:` could never be satisfied before: the catalogue refused `benchmarks:`). Python: selection.py, benchmarks.py, `CaseOutcome.scores`, `Verdict.measured`, the resolver holding lines before and after a run. - §B5 a pinned model's settings: `model-can.ignores`, `ignores-while-thinking`, `thinks-unless-told` read from Pydantic AI 2.54 and recorded for GPT-5.5, GPT-5.4, GPT-5.4 mini, Claude Opus 5 and Sonnet 5; settings.rs refuses a line the model would throw away; `ModelEntry.ignored_settings`. - §C `template-language: jinja` in the dialect Jinja2 and MiniJinja render alike (jinja.rs, `holes.fill(language=)`); which texts are filled is `template: yes` in the schema. - §D `turns:` on agents, variants and eval cases (turns.rs). - §E `pact-sync` records a server's tools as `resource.published:`; `tool.schemas-from:`, `resource.if-it-changes:`, `tool.unlisted-actions:` (snapshots.rs, `mcp_bridge.when_it_changed`). - Also: `loader/a-shape-named-like-a-built-in`; the catalogue cache keyed on the file's mtime; recording a benchmark copies the shipped row first. Fixture tests/trees/a-desk-that-listens-reads-and-chooses with 46 broken-once cases; site-docs gains five guide pages; schema golden blessed (additions only); README 4505 tests (1489 Rust + 3016 adapter). Gate: ./scripts/test-all.sh green (Rust, every shipped tree, TypeScript, 3016 adapter tests); cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all -- --check clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
The design and its critique are in
docs/96-MODALITIES-INTERFACES-TEMPLATES-TURNS.md; user guides are new undersite-docs/guide/.documentsandvideoinput shapes; six model kinds (language, decision, embedding, transcription, speech, reranking), each checked where a model is named;agent.voice:(pipeline, interruption, transcriber, speaker, voice, languages, audio retention). A local transcriber and speaker keep a caller's voice on the machine behind a hosted model.knowledge.embedded-by:makes meaning search possible air-gapped.tools/email.yamlwithinterface: yes; Gmail and Outlookimplements: email, picked per person or per run (chosen-by:). An implementation may only say how it reaches its server and may never ask less of a person than the interface. The same works for skills and document sets;category:labels tools.needs.accept-when:—mmlu.score > 50,refund-tone.score >= 0.8(any metric of the suite, a DeepEval G-Eval included),pass-rate >= 90%,cost < 0.05 USD— with named benchmarks declared in the catalogue andprefer: cheapest | fastest | best-on-the-evals. Fixesneeds.scores:, which could never be satisfied (the catalogue refusedbenchmarks:).pact checkrefuses asettings:line the pinned model would throw away (temperature on GPT-5.5 withoutthinking: none, top-k on OpenAI models, any sampling on Claude Opus 5), read from the Pydantic AI code this repository pins.template-language: jinjain a dialect both ports render the same way, sandboxed;turns:from system, developer, user, agent and tool, on agents, variants and eval cases.pact-syncrecords a server's tools into the tree for review;schemas-from:,if-it-changes:(changes an approval rests on always stop) andunlisted-actions:.Compatibility
The schema only grows: 70 fields added, none removed or changed. A new rule refuses a workspace shape named like a built-in shape. No tree among the 245 in this repository and Bud Agent Flow trips any new rule.
Verification
./scripts/test-all.sh: Rust, every shipped tree under--deny-warnings, TypeScript type-check, all seven adapter targets. 3016 adapter tests pass (pytest -n 4). Clippy and fmt are clean.mkdocs build --strictpasses.pact-sync --urlagainst a live server, and running a live DeepEval benchmark.--from-fileand recording a figure are tested.Not yet done at run time
Listed in §H of the doc and on
site-docs/status/gaps.md: turns reaching a model, renaming calls for interface implementations, enforcingeach-call-asks-a-person, a voice transport, and the TypeScript port.Bud Agent Flow's companion PR depends on this one.
🤖 Generated with Claude Code