Build an evaluation pipeline you can explain, reproduce, compare, and share.
English | 中文 | Documentation | Open Bench
SURE-EVALUATION is a general system evaluation framework. It integrates evaluation tools for tasks such as speech recognition, speech generation, speech enhancement, translation, classification, and language understanding, while turning the complete evaluation process into versioned nodes.
You can choose a task-specific pipeline and its tools, such as different normalizers, transcription models, conversions, or scoring implementations. Every evaluation records the full path from model output to final score and writes it as structured data. A result is therefore tied to the exact inputs, pipeline identity, node sequence, configuration, and report that produced it.
The framework uses a lightweight root package and independently managed optional node environments. This design supports a broad task surface without forcing every user to install every model and tool.
- Broad coverage: one framework covers text, audio, generation, recognition, translation, classification, and structured detection tasks.
- Convenient: ask the CLI for the available pipelines and required inputs, then prepare only the environment selected by the exact pipeline.
- Trustworthy and reviewable: each run records
pipeline_id, ordered computation nodes, input contracts, and structured output artifacts. - Comparable: results can be compared against the same explicit evaluation definition instead of an ambiguous metric label.
- Customizable: the same metric can have several registered routes, so you can select a different normalizer, model, backend, or score implementation without renaming the metric.
SURE-EVALUATION uses four ordered evaluation stages. Routes include only the stages they need:
audio or model output
-> frontend (optional audio preparation)
-> transcription (optional audio-to-text)
-> normalization (optional text canonicalization)
-> scoring (metric computation)
-> report.json + pipeline_description.json
Input parsing and format checks belong to the first node that consumes the input. They are documented internal steps, not separate pipeline stages.
A pipeline_id names an exact computation as
task.language.metric.node_version.... For example:
asr.en.wer.whisper_norm_english_v1.wenet_wer_v1
metric stays globally canonical, such as wer, cer, spk_sim, dnsmos,
wv_mos, or utmos. When two routes report the same metric, their
pipeline_id differs because their node chain differs. Multi-metric requests
are bundles whose identities contain their atomic member pipelines.
The committed catalog can be drawn as one map: every atomic pipeline is a colored ribbon that flows from its task through the applicable frontend, transcription, normalization, and scoring nodes into a report.
docs/atlas/index.html is the interactive version of
the same map, with search, per-task filtering, hover tracing, and a catalog
table; open it in a browser from a local clone. Both views are generated from
docs/pipeline_catalog.jsonl:
python scripts/generate_pipeline_atlas.pySURE-EVALUATION is currently installed from source, not from PyPI. The commands below create an isolated environment and run a committed, text-only ASR example that does not download a model or create a node-local environment.
git clone https://github.com/PigeonDan1/sure-evaluation.git
cd sure-evaluation
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .
python -m pip check
sure-eval doctorDiscover all English ASR WER routes. The table marks the default, prints every
exact pipeline_id, shows the ordered computation nodes and required inputs,
and identifies optional setup:
sure-eval metric routes asr --language en --metric werSelect one exact route and write its executable pipeline JSON:
sure-eval metric describe asr \
--pipeline-id asr.en.wer.whisper_norm_english_v1.wenet_wer_v1 \
--output .sure-eval-demo/pipeline.jsonInspect the setup plan and validate only the environments used by that exact pipeline:
sure-eval env setup --pipeline .sure-eval-demo/pipeline.json --dry-run
sure-eval env check --pipeline .sure-eval-demo/pipeline.jsonRun the pipeline and inspect the structured report:
sure-eval metric run \
--pipeline .sure-eval-demo/pipeline.json \
--ref-file examples/readme/asr_en_ref.txt \
--hyp-file examples/readme/asr_en_hyp.txt \
--output-dir .sure-eval-demo/asr-en-wer \
--validate-env
python -m json.tool .sure-eval-demo/asr-en-wer/report.json
python -m json.tool .sure-eval-demo/asr-en-wer/pipeline_description.jsonThe same exact pipeline_id appears in the discovery result, pipeline JSON,
run summary, report.json, and pipeline_description.json.
Use canonical metric names to discover alternatives, then copy the exact
pipeline_id you intend to run:
# Three Mandarin ASR CER normalization/scoring routes
sure-eval metric routes asr --language zh --metric cer
# Three TTS speaker-similarity backends
sure-eval metric routes tts --language zh --metric spk_sim
# Three KWS input contracts for the same macro_recall metric
sure-eval metric routes kws --metric macro_recall
# Stable machine-readable discovery output
sure-eval metric routes tts --language zh --metric cer --jsonFor a pipeline with optional models or tools, prepare only its declared nodes:
sure-eval metric describe tts \
--pipeline-id tts.zh.cer.qwen3_asr_1_7b_v1.punctuation_strip_norm_v1.wenet_cer_v1 \
--output .sure-eval-demo/tts-qwen.json
sure-eval env setup --pipeline .sure-eval-demo/tts-qwen.json --dry-run
sure-eval env setup --pipeline .sure-eval-demo/tts-qwen.json
sure-eval env download --node transcription/qwen3_asr_1_7b --dry-run
sure-eval env download --node transcription/qwen3_asr_1_7b
sure-eval env check --pipeline .sure-eval-demo/tts-qwen.jsonSee Installation and
Environment Management before running heavyweight
nodes. Environment setup installs declared tools; model assets are downloaded
separately and should be reviewed with env download --dry-run. Checkpoints,
caches, and node-local virtual environments remain local and are excluded from
packages and Git.
| Task | Canonical metrics | Guide |
|---|---|---|
| ASR | WER, CER, MER | ASR |
| S2TT | BLEU, BLEU-char, chrF, XCOMET-XL, BLEURT-20 | S2TT |
| SD | DER | SD |
| SV | EER, minDCF | SV |
| SA-ASR | cpWER, DER companion result | SA-ASR |
| TTS | CER, WER, speaker similarity, DNSMOS, WV-MOS, UTMOS, KOMOS (Korean) | TTS |
| TTA | FAD, FD, KL, IS, CLAP score, AudioBox Aesthetics CE/CU/PC/PQ | TTA |
| VC | CER, WER, speaker similarity, DNSMOS, WV-MOS, UTMOS | VC |
| SE | SI-SDR, STOI/eSTOI, PESQ, LSD, nSRMR, LPS, DNSMOS P.835/P.808 components, WV-MOS, UTMOS, NISQA DIM, SQUIM objective predictions | SE |
| RIR estimation | NPM, RT30/RT60 MAE, DRR/C50/C80/EDC, IACC full/early/late, ITD/ILD/IPD/DOA errors | RIR estimation |
| TSE | SI-SDR, speaker similarity, DNSMOS, WV-MOS, UTMOS, CER, WER | TSE |
| Classification / SER / GR | Accuracy | Classification |
| SLU | Accuracy, LLM Judge | SLU |
| KWS | Per-keyword/macro recall at 0.1 FA/hour, trigger latency; optional ROC/AUC | KWS |
| VAD | F1, false alarm, miss, NIST DCF, ROC AUC | VAD |
| FULL_DUPLEX | IRD, FSED, ERT, EIT from saved conversation timelines | Full-duplex timing |
| DOA | MAE, RMSE, threshold accuracy, localization precision/recall/F1 | DOA |
| LID | Spoken-language label accuracy | LID |
Each task guide defines its input contract, registered metrics, exact pipeline IDs, nodes, and run examples. The generated pipeline catalog contains all registered atomic route instances for declared language profiles plus curated multi-metric bundles.
Routes are declared in
src/sure_eval/evaluation/tasks/<task>/routes.yaml. Node metadata lives in
src/sure_eval/evaluation/nodes/<stage>/<name>/manifest.yaml; optional runtime
requirements live beside it in node_env.yaml. Adding a route changes routing
and identity metadata, while its computation remains owned by its versioned
nodes.
To contribute a task, metric, route, or node tool, start with Contributing. It directs each PR type to a focused manual and the repository PR template. Agents should also follow the Agent Contract.
Community pipelines can be shared and explored on Open Bench, where usage and community feedback help collaborators identify widely adopted evaluation routes.
Create an evaluation pipeline that is truly yours, then make it reviewable and shareable.
MIT License. See LICENSE.