This guide covers installation, data materialization, candidate generation, evaluation, and run records. Detailed contracts are documented in data, materialization, and evaluation. Candidate-specific setup lives beside each implementation and is linked from the root README.
doteBench supports Python 3.10 through 3.12 and uses uv for locked environments.
git clone --recurse-submodules https://github.com/studio-dots-ai/doteBench.git
cd doteBench
export UV_PROJECT_ENVIRONMENT=/path/to/envs/dotebench-development
export UV_LINK_MODE=hardlink
uv sync --frozen --extra dev --extra metrics
uv run --frozen dotebench --help
uv run --frozen python -m dotebench.compilation checkKeep environments, model snapshots, generated audio, service logs, and runs outside the checkout.
Set BUNDLE_ARCHIVE to the audio bundle archive path. Verify it with the
companion SHA-256 file, then extract it:
BUNDLE_DIR=$(dirname "$BUNDLE_ARCHIVE")
BUNDLE_FILE=$(basename "$BUNDLE_ARCHIVE")
(cd "$BUNDLE_DIR" && sha256sum --check "$BUNDLE_FILE.sha256")
tar -xzf "$BUNDLE_ARCHIVE"A complete data root also requires Common Voice 20.0 and the prepared materialization models. Follow Data materialization, then run:
export DOTEBENCH_SOURCE_ROOT="$PWD"
export DOTEBENCH_MODEL_ROOT=/path/to/models
export DOTEBENCH_ENV_ROOT=/path/to/envs
uv run --frozen dotebench materialize-data \
--bundle-root /path/to/audio-bundle \
--common-voice-root /path/to/common-voice-20.0 \
--output-root /path/to/materialized-dataVerify construction identities, then validate benchmark inputs:
uv run --frozen dotebench verify-materialization \
--data-root /path/to/materialized-data \
--reference release/materializations/reference.json
uv run --frozen dotebench validate-data \
--data-root /path/to/materialized-data \
--audio-root /path/to/materialized-dataThe first command checks how the data root was constructed. The second checks that every selected case can be loaded and used.
The identity candidate copies source audio for a smoke test:
uv run --frozen dotebench generate \
--candidate identity \
--compiler one_take \
--category text --shard easy \
--data-root /path/to/materialized-data \
--audio-root /path/to/materialized-data \
--run /path/to/runs/identity-text-easyFor a learned system, select a supported candidate and configuration:
uv run --frozen dotebench generate \
--candidate dots-tts-edit \
--compiler one_take \
--config configs/dots_tts_edit.yaml \
--data-root /path/to/materialized-data \
--audio-root /path/to/materialized-data \
--run /path/to/runs/dots-tts-editone_take compiles the complete XML into one model call.
sequential applies one XML operation per call, with each output
becoming the next input. Follow the selected integration README for its native
request mapping, configuration, and capability boundary.
Prepare the metric models registered in
release/evaluation-models.json:
uv run --frozen dotebench prepare-evaluation-models \
--model-root /path/to/evaluation-modelsEvaluate the case selection recorded by generation and render its report:
uv run --frozen --extra metrics dotebench evaluate \
--data-root /path/to/materialized-data \
--audio-root /path/to/materialized-data \
--run /path/to/runs/dots-tts-edit \
--config configs/evaluation_bundle.yaml \
--override runtime.python="$DOTEBENCH_ENV_ROOT/dotebench-metrics-cpu/bin/python" \
--override services.qwen3_asr.python_bin="$DOTEBENCH_ENV_ROOT/dotebench-qwen-asr/bin/python" \
--override services.qwen3_aligner.python_bin="$DOTEBENCH_ENV_ROOT/dotebench-qwen-asr/bin/python" \
--override services.utmos.python_bin="$DOTEBENCH_ENV_ROOT/dotebench-quality-speaker/bin/python" \
--override services.speaker_similarity.python_bin="$DOTEBENCH_ENV_ROOT/dotebench-quality-speaker/bin/python" \
--override services.emotion.python_bin="$DOTEBENCH_ENV_ROOT/dotebench-qwen-omni/bin/python"
uv run --frozen dotebench report --run /path/to/runs/dots-tts-editSee Evaluation protocol for model preparation, metrics, and failure handling.
A run is one benchmark execution for a fixed shard selection, candidate, compiler, and resolved configuration. Generation and evaluation share one directory:
run.json
audio/<case-id>.wav
generation.jsonl
evaluation.jsonl
summary.json
run.json records one dataset version, selected manifest checksums,
case IDs, compiler identity, candidate configuration, runtime identity, service
graph, and stage state. A run
can cover one shard or all seven shards. Generation failures remain in the
evaluation denominator. Infrastructure failures leave the run incomplete and
prevent a formal report.