Skip to content

feat(se): add OpenBench speech enhancement metrics - #39

Merged
PigeonDan1 merged 2 commits into
X-LANCE:mainfrom
Yj-Xiong:feat/se-openbench-metrics
Nov 2, 2026
Merged

PigeonDan1 merged 2 commits into
X-LANCE:mainfrom
Yj-Xiong:feat/se-openbench-metrics

Conversation

@Yj-Xiong

Copy link
Copy Markdown

Start Here

Read Contributing. If the category is unclear, use
Add Evaluation Capabilities.

## Summary

Add reusable, opt-in Speech Enhancement (SE) metric pipelines. These metrics
are general SE metrics and are not limited to dereverberation.

Added metrics:

- eSTOI
- LSD
- LPS
- nSRMR
- DNSMOS P.808 MOS
- DNSMOS P.835 SIG
- DNSMOS P.835 BAK
- TorchAudio SQUIM objective STOI, PESQ, and SI-SDR predictions
- NISQA DIM MOS, NOI, DIS, COL, and LOUD predictions

The existing default SE bundle and existing metric behavior remain unchanged.
POLQA/P.863 is intentionally not registered because the supplied white paper
does not provide a redistributable licensed scoring implementation.

## PR Type

Select one guide:

- [ ] [New task](docs/pr_guides/new_task.md)
- [x] [New metric](docs/pr_guides/new_metric.md)
- [ ] [New pipeline route](docs/pr_guides/new_route.md)
- [ ] [Node/tool/version change](docs/pr_guides/node_change.md)
- [ ] [Maintenance](docs/pr_guides/maintenance.md)

Guide followed:

- `docs/pr_guides/new_metric.md`
- `docs/contributing.md`

## Evaluation Identity

- Reported `metric`:
  `estoi`, `lsd`, `lps`, `nsrmr`, `dnsmos_p808`, `dnsmos_sig`,
  `dnsmos_bak`, `squim_stoi`, `squim_pesq`, `squim_si_sdr`,
  `nisqa_mos`, `nisqa_noi`, `nisqa_dis`, `nisqa_col`, `nisqa_loud`
- `execution_metrics`, if different:
  Same as the reported metric for all new routes.
- Added or changed `pipeline_id`:
  - `se.any.estoi.estoi_v1`
  - `se.any.lsd.lsd_v1`
  - `se.any.lps.lps_v1`
  - `se.any.nsrmr.nsrmr_v1`
  - `se.any.dnsmos_p808.dnsmos_p808_v1`
  - `se.any.dnsmos_sig.dnsmos_sig_v1`
  - `se.any.dnsmos_bak.dnsmos_bak_v1`
  - `se.any.squim_stoi.squim_stoi_v1`
  - `se.any.squim_pesq.squim_pesq_v1`
  - `se.any.squim_si_sdr.squim_si_sdr_v1`
  - `se.any.nisqa_mos.nisqa_mos_v1`
  - `se.any.nisqa_noi.nisqa_noi_v1`
  - `se.any.nisqa_dis.nisqa_dis_v1`
  - `se.any.nisqa_col.nisqa_col_v1`
  - `se.any.nisqa_loud.nisqa_loud_v1`
- Default route changed: no
- Score comparability changed: no

## Changed Declarations

- Task manifest: yes
- Task routes: yes
- Node manifest: yes
- `node_env.yaml`: yes
- Node README follows `docs/node_readme_template.md`: yes
- Catalog regenerated: yes
- Catalog discovery task or curated bundle changed:
  yes, atomic SE routes were added; no task or default bundle was changed
- Documentation coverage checked against `docs/contributing.md`:
  yes. Root EN/ZH READMEs, the SE task guide, task-local README, all added
  node READMEs, and generated atlas artifacts were updated. The task index was
  unchanged because SE already exists as a registered task.

## Validation

Exact commands run:

```bash
PYTHONPATH=src /hpc_stor01/home/yujie.xiong_sx/AutoResearch/SURE-SE/work/harness_runtime/bin/python -m pytest -q tests/test_se_openbench_metrics.py

PYTHONPATH=src /hpc_stor01/home/yujie.xiong_sx/AutoResearch/SURE-SE/work/harness_runtime/bin/python -m pytest -q --ignore=tests/test_evaluation_env_check.py

PYTHONPATH=src /hpc_stor01/home/yujie.xiong_sx/AutoResearch/SURE-SE/work/harness_runtime/bin/python scripts/generate_pipeline_catalog.py

PYTHONPATH=src /hpc_stor01/home/yujie.xiong_sx/AutoResearch/SURE-SE/work/harness_runtime/bin/python scripts/generate_pipeline_atlas.py

git diff --exit-code docs/pipeline_catalog.jsonl \
  docs/atlas/index.html \
  docs/atlas/pipeline_atlas.svg \
  docs/atlas/pipeline_atlas_dark.svg

git diff --check

Results:

  • SE-focused tests: 10 passed
  • Portable CI approximation: 545 passed, 67 skipped
  • Catalog and atlas regeneration: no diff
  • Whitespace validation: passed

Exact pipeline identity check:

sure-eval metric routes se --metric estoi --json

sure-eval metric describe se \
  --metrics estoi,lsd \
  --output /tmp/se_pipeline.json \
  --json

sure-eval env setup \
  --pipeline /tmp/se_pipeline.json \
  --dry-run \
  --json

sure-eval metric run \
  --pipeline /tmp/se_pipeline.json \
  --samples-jsonl /path/to/samples.jsonl \
  --output-dir /tmp/se_report \
  --json

The generated report.json and pipeline_description.json preserve the exact
selected pipeline identity.

For heavyweight nodes, real A10 smoke tests were completed for DNSMOS, NISQA
DIM, SQUIM objective predictions, and LPS. Checkpoint-backed environment
assertions for unrelated existing nodes require their pre-existing model assets.

Safety

  • No .venv, checkpoints, generated reports, local cache paths, or credentials committed
  • No private absolute paths in committed config or docs
  • Score-affecting behavior is declared in route/node identity
  • Existing default route behavior is tested and unchanged

@PigeonDan1

Copy link
Copy Markdown
Collaborator

Thanks for the implementation. After reviewing the route definitions, SE task pipeline, provider dispatch, documentation, local regression results, and the green Core workflow, I consider the overall direction and the added metrics acceptable.

There is one concrete issue I would like you to address before merge:

Required: incorrect metric metadata in multi-metric pipeline slots

For example:

build_pipeline_spec("se", metric="estoi,lsd")

currently describes the slots as:

scoring   -> scoring/estoi, metric="estoi"
scoring_2 -> scoring/lsd,   metric="estoi"

The second slot should identify lsd. This does not currently change the executed score because execution uses the selected metrics/member pipeline IDs, but it makes the pipeline JSON metadata inaccurate and can mislead downstream inspection or tooling. The issue appears to come from the shared _node_slots() path using only the first selected route when populating every scoring slot. Please fix or explicitly redesign that metadata and add a regression test covering multi-metric scoring slots. Please also verify the existing TTS/VC/TSE/SE bundles after the change.

Optional robustness suggestion

The shared provider normalization layer could reject non-finite values (NaN/Inf) before aggregation, so injected providers cannot produce a non-JSON score. The concrete new model providers already perform finite checks, so I consider this a robustness improvement rather than a merge blocker.

The remaining reviewed items do not require changes for this PR: the default SE bundle is unchanged, the new routes are opt-in, the documentation and model/license limitations are described, and the current Core checks pass. After the slot metadata fix, please update the PR head and rerun Core; I will review the updated result before merge.

@PigeonDan1
PigeonDan1 merged commit 204ffe8 into X-LANCE:main Nov 2, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants