This document defines the repository architecture and the process for extending doteBench. User workflows are documented in Using doteBench.
The benchmark connects data construction, generation, and evaluation:
manifest XML and evaluation annotations
|
+-- materialization -- complete data root
+-- candidate compiler -- native request -- generated audio
+-- evaluator -- per-case records -- aggregate report
dotebench.instructions owns XML validation, Unicode character
spans, operation regions, and deterministic source and target rendering.
dotebench.compilers owns one-take and sequential planning. Candidate
packages own native request construction and their service backends. Shared
speech and scoring backends live in services/backends; services/utils owns
configuration, dependency resolution, process lifecycle, and transport utilities.
Evaluation modules own XML scope selection, scoring policy, and aggregation.
Services may depend on shared root data types and utilities; they must not import
candidates, evaluation, or materialization, directly or transitively.
Shared backends own reusable computation and service entrypoints. Evaluators may
call backend computation functions directly. Importing a backend does not start
its service.
See Evaluation protocol for metric implementations, model
preparation, and request contracts.
Materialization ends at a complete data root. Candidate generation and evaluation read its manifests and content-addressed WAVs without depending on provider recipes or materialization services.
src/dotebench/
├── instructions/ XML parsing and transcript rendering
├── compilers/ shared one-take and sequential planning
├── candidates/ model-specific compilers, adapters, and backends
├── services/
│ ├── utils/ configuration, process management, and shared utilities
│ └── backends/ one Python module per shared speech or scoring backend
├── materialization/ recipes, synthesis backends, and verification
├── evaluation/ formal scoring and aggregation
├── dataset.py manifest loading and validation
├── execution.py generation and evaluation runners
└── storage.py persisted run records
data/ benchmark manifests
release/ locks, provenance, notices, and reference identities
configs/ candidate and service configurations
envs/ isolated evaluator and materialization environments
third_party/ pinned official upstream submodules
tests/ offline public contract and regression tests
Each built-in candidate directory contains a README for its public setup, model-input mapping, compiler behavior, and capability boundary.
A candidate integration consists of:
- A registry entry declaring its public ID and compiler support.
one_take.pyand, for editing candidates,sequential.pycompiler entrypoints.- A model-specific
compile.pythat maps benchmark XML to native requests. - An adapter and backend boundary that classify candidate and infrastructure failures.
- A flat YAML configuration under
configs/. - For an imported native runtime, an official upstream gitlink, a locked
environment,
runtime.json, and the smallest reviewed inference patch. - Request-level tests that inspect the final native call, plus service lifecycle and failure-contract tests.
Candidate code may render transcripts with the public instruction parser or implement an equivalent conversion. It cannot read manifest annotations or frozen evaluation alignments. Auxiliary alignment belongs to the candidate and must record its own model and cache identity.
Each compiler entrypoint has an adjacent JSON manifest containing a positive
integer version. Generation records the candidate, compiler name, and version
in run.json, and checks that this identity is consistent through execution.
python -m dotebench.compilation check
python -m dotebench.compilation check \
--candidate ming-uniaudio \
--compiler sequentialIncrement the version of every affected compiler when changing its model input, XML rendering, or shared compilation behavior. A change to logic shared by one-take and sequential compilation requires updating both versions. Review the version changes alongside the candidate request tests. Frozen runs retain the version recorded at creation.
Native runtime identity is separate. It covers the official commit, patch,
service adapter, common launcher, and environment lock. Startup verifies the
gitlink, upstream URL, clean source tree, patch hashes, and prepared runtime.
The service reports the captured identity through /health, and
generation checks it against the installed integration resources.
Every declaration has an instance key, endpoint, health path, startup timeout, GPU assignment, environment, and argument-array command. Nested dependencies start before their consumers. Repeated instance keys must resolve to identical settings. Cycles and conflicts fail during composition.
The process manager records startup metadata and separate logs, terminates process groups on failure or interruption, and passes only resolved direct endpoints to the consumer. Model loading and request semantics remain in the backend. Each backend owns its request schema, routes, concurrency policy, model loading, and resource cleanup. Shared HTTP utilities accept explicit handlers and lifecycle callbacks. Evaluators and candidate adapters validate the service identities required by their own workflows.
Asset utilities prepare and verify files against caller-supplied model locks. Callers own their model registry and expected identities; backends own checkpoint and loader requirements.
Candidate services live beside their compilers and adapters. Shared backend loaders can also support candidate-owned auxiliary services.
- Implement reusable measurement and its service entrypoint in
services/backends/, using shared transport and lifecycle utilities. - Declare its environment, command, and direct dependencies in
configs/. - Connect the ability through
evaluation/abilities.py; put benchmark scope selection, scoring, and aggregation inevaluation/. - Register any evaluation model and its expected lock in
evaluation/models.pyandrelease/evaluation-models.json. - Cover computation, request validation, service lifecycle, and evaluator behavior with offline tests, and document preparation and scoring in the Evaluation protocol.
Review the public contract version whenever the scoring behavior changes.
The dataset version is declared by release/shards.json; the evaluator
protocol has its own version in release/evaluation-protocol.json. Python distribution
metadata has an independent package version. Compiler versions identify reviewed compilation behavior. Runtime identities
describe their source and dependency content.
Changes to manifest semantics, XML grammar, evaluation formulas, failure penalties, or aggregation require a new public contract version. Changes to compilation behavior require incrementing the affected compiler versions. Dependency releases and upstream model revisions keep their upstream versions.
Use the root locked environment:
export UV_PROJECT_ENVIRONMENT=/path/to/envs/dotebench-development
export UV_LINK_MODE=hardlink
uv sync --frozen --extra dev --extra metrics
uv run --frozen pytest
uv run --frozen python -m dotebench.compilation check
PYTHONPATH=src uv run --frozen python scripts/build_text_provenance.py --checkThe default suite is CPU-only and offline. It uses small fixtures to test stable benchmark contracts: XML semantics, manifest validation, scoring, service orchestration, materialization identities, and public packaging. Run the compiler and provenance validators alongside pytest.
Each model integration has one self-contained suite under
tests/candidates/<name>/. It covers that candidate's compiler,
adapter, transport, and native service boundary. Run it with the locked
candidate environment documented in that candidate's README. Shared candidate
test helpers and cross-candidate boundary checks live under
tests/candidates/common/. Candidate suites are included in the
source distribution and are outside the wheel package.
Full-manifest scans, model-backed smoke tests, historical comparisons, and
release-bundle audits are release validation work. Store their programs, logs,
and machine-readable results in the project artifact workspace rather than the
repository. Changes to public files run
tests/publication/test_repository.py.
Release builders in scripts/ produce versioned inputs:
build_audio_bundle.pycreates the redistributable audio archive and checksum from the provenance ledger;build_materialization_metadata.pyrefreshes reference materialization metadata from a verified data root;build_text_provenance.pyrebuilds and checks the public text provenance ledger.
Release investigation, comparison, and one-off audit programs belong in the project artifact workspace.