Open ASR Leaderboard

This repository contains the code for the Open ASR Leaderboard. The leaderboard is a Gradio Space that allows users to compare the accuracy of ASR models on a variety of datasets. The leaderboard is hosted at hf-audio/open_asr_leaderboard.

Datasets

The Open ASR Leaderboard evaluates models on a diverse set of publicly available ASR benchmarks hosted on the Hugging Face Hub. These datasets cover a wide range of domains, languages, and recording conditions to provide a fair and comprehensive comparison across models.

• Main Test Sets (English, short-form): The main benchmark datasets used for evaluation (short-form English) are available here.

• Chunked sets with session-level references: Earnings22-Cleaned-AA-chunked splits 6 earnings calls into 341 chunks, with one reference transcript per call in the parent Earnings22-Cleaned-AA set. Models transcribe each chunk as usual; the scorer concatenates a call's chunk predictions before computing WER against the call-level reference. Run with --dataset_path ArtificialAnalysis/Earnings22-Cleaned-AA-chunked --dataset earnings22_cleaned_aa_chunked --split test.

• English, long-form: The ASR Longform benchmark dataset includes earnings21 and earnings22. We also evaluate on CORAAL, but it is stored as a separate dataset since it has multiple splits.

• Multilingual Benchmark: The ASR Multilingual benchmark dataset includes FLEURS, MCV and MLS. For Hindi, data from Voice Arena is used. The Armenian FLEURS and MCV test sets are hosted under Metric-AI/open-asr-leaderboard-multilingual-datasets.

• Private datasets: After submitting a model to the leaderboard, the maintainers will evaluate on private sets, as described here.

Evaluate a model (as of 24 July 2026)

English short-form, English long-form, and multilingual evaluations use Hugging Face Jobs to guarantee reproducibility: every run executes a Docker image on the same hardware, to minimize environment and driver differences.

For API models see here.

Jobs are launched on the following hardware (flavor in HF Jobs terminology):

name             pretty name             cpu       ram      storage   accelerator               cost/min  cost/hour
h200             Nvidia H200             23 vCPU   256 GB   3000 GB   1x H200 (141 GB)          $0.0833   $5.00

Example costs for a full run over the main public datasets for English short-form:

• $2.92 for nvidia/parakeet-tdt-0.6b-v3
• $4.75 for openai/whisper-large-v3-turbo
• $5.58 for Qwen/Qwen3-ASR-1.7B

Each model family has its own Docker image with the necessary software requirements. The evaluation configurations are hosted as HF Spaces.

To launch an evaluation:

1. Hugging Face Hub setup

   • Create an account at https://huggingface.co/ and add credits for HF Jobs: https://huggingface.co/settings/billing
   • Create a WRITE token and copy it.
   • Create a Storage Bucket to store results: https://huggingface.co/new-bucket

2. One-time local setup

A local setup is needed to launch the evaluation and score with the repo's normalizer.

# Clone the repository
git clone git@github.com:huggingface/open_asr_leaderboard.git
cd open_asr_leaderboard

# Create a minimal conda environment (no GPU required locally)
conda create -n leaderboard_jobs python=3.10 -y
conda activate leaderboard_jobs
pip install -r requirements/requirements_jobs.txt
huggingface-cli login   # paste your WRITE token when prompted

3. Launch an evaluation 🚀

# Open the relevant submit_jobs script, uncomment the models/datasets you want, then run:
RESULTS_BUCKET="<your-bucket>" HF_TOKEN=hf_... bash qwen/submit_jobs.sh

# Jobs are submitted in parallel (one per dataset). The script waits for all
# jobs to finish, syncs results from the bucket, and prints a CSV summary.

# Billing to org
ORG_NAME="<org-name>" RESULTS_BUCKET="<your-bucket>" HF_TOKEN=hf_... bash qwen/submit_jobs.sh

For long-form evaluations (Earnings21, Earnings22, and the eight CORAAL subsets), use the submit_jobs_*_longform.sh script of the model's family, e.g.:

RESULTS_BUCKET="<your-bucket>" HF_TOKEN=hf_... bash transformers/submit_jobs_whisper_longform.sh

Local evaluation

For contributors who want to test locally, the requirements/ folder contains per-family dependency files. The Dockerfiles in the HF Spaces can also be used to build a local container.

Each model family has a run_eval.py entry point, launched on HF Jobs by a submit_jobs*.sh script (e.g. transformers/submit_jobs_whisper.sh). run_eval.py outputs a JSONL file with predictions and prints WER and RTFx after completion.

Trade-off plots

Pareto plots of WER against RTFx and model size are available directly on the leaderboard.

Benchmark fitting

The benchmark_fitting folder contains scripts to measure whether a model overfits reference transcripts in the test sets. See benchmark_fitting/README.md for more information.

Contributing a model or dataset

Please follow the pull request template; it contains a submission checklist and guidelines.

Miscellaneous

Normalizer changes

After a normalization change, existing JSONL manifests can be rescored without rerunning inference: manifests store the raw text and pred_text, and normalization is applied at scoring time.

# dry run over every row of every results file
python scripts/rescore_all.py

# some models / files only, keeping the updated CSVs locally for inspection
python scripts/rescore_all.py --models openai/whisper-large-v3 nvidia/canary-1b-v2 \
    --targets english multilingual --language de --out_dir /tmp/rescored

# re-use results already synced locally, and open the PRs
python scripts/rescore_all.py --skip_sync --open_pr

Citation

@misc{srivastav2025openasrleaderboardreproducible,
      title={Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation}, 
      author={Vaibhav Srivastav and Steven Zheng and Eric Bezzam and Eustache Le Bihan and Nithin Koluguri and Piotr Żelasko and Somshubra Majumdar and Adel Moumen and Sanchit Gandhi},
      year={2025},
      eprint={2510.06961},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.06961}, 
}