Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
063f3c0
feat: add RIR evaluation task and spatial metrics
PigeonDan1 Oct 13, 2026
33655e2
fix: accept numpy RIR references
PigeonDan1 Oct 13, 2026
260d314
feat: extend RIR spatial evaluation metrics
PigeonDan1 Oct 17, 2026
cf2382e
fix: expose RIR spatial metric aliases
PigeonDan1 Oct 17, 2026
8041b12
docs: document RIR scoring node contracts
PigeonDan1 Oct 17, 2026
333eec2
feat: align RIR evaluation with Rec-BIR metrics
PigeonDan1 Oct 17, 2026
82c3f60
fix: match Rec-BIR lag-invariant NPM
PigeonDan1 Oct 17, 2026
e54a434
feat: support ch0-relative multichannel spatial metrics
PigeonDan1 Oct 17, 2026
32a19f4
docs: name the task RIR estimation
PigeonDan1 Oct 17, 2026
8154867
feat: report ITD in milliseconds
PigeonDan1 Oct 17, 2026
ed41e18
feat: add cross-channel base RIR bundle
PigeonDan1 Oct 17, 2026
f1f57a1
feat: add multichannel spatial bundle without IACC
PigeonDan1 Oct 17, 2026
9ce8323
feat: record reference and prediction RIR metrics
PigeonDan1 Oct 19, 2026
adecb58
fix: align RIR sample aggregation and document node contracts
PigeonDan1 Oct 19, 2026
e805dd7
feat: align RIR review branch with upstream main
PigeonDan1 Oct 19, 2026
b590195
merge: sync evaluation engine with upstream main
PigeonDan1 Oct 21, 2026
bbac6c9
feat: add RT30 and EDC RIR metrics
PigeonDan1 Oct 21, 2026
43116b5
chore: regenerate pipeline catalog for RIR metrics
PigeonDan1 Oct 21, 2026
9ac40a8
fix: bound RIR NPM lag correlation memory
PigeonDan1 Oct 21, 2026
0fdc5d2
feat: add full early late IACC RIR metrics
PigeonDan1 Oct 21, 2026
019ac30
perf: use FFT for bounded RIR NPM correlation
PigeonDan1 Oct 21, 2026
532a8d1
test: cover DOA in binaural RIR bundle
PigeonDan1 Oct 21, 2026
755bfa6
chore: refresh RIR bundle presets and docs
PigeonDan1 Oct 22, 2026
e317803
Merge remote-tracking branch 'origin/main' into feat/rir-estimation-o…
PigeonDan1 Oct 23, 2026
b8c1ddc
docs: complete IACC node readmes
PigeonDan1 Oct 23, 2026
efd4137
test: complete RIR route and input contract coverage
PigeonDan1 Oct 26, 2026
6545382
fix(rir): reject undefined channel scores and preserve input provenance
PigeonDan1 Oct 27, 2026
2228686
test(rir): verify known delays and documented preprocessing
PigeonDan1 Oct 27, 2026
1e7cdb2
docs: refresh the pipeline atlas and check generated views in Core
PigeonDan1 Oct 27, 2026
ae16abc
Merge main after KOMOS and refresh the combined pipeline atlas
PigeonDan1 Oct 27, 2026
474bbf0
Merge documentation freshness checks and regenerate the RIR atlas
PigeonDan1 Oct 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/workflows/core.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,15 +80,16 @@ jobs:
- uses: astral-sh/setup-uv@v5
with:
enable-cache: true
- name: Regenerate catalog
- name: Regenerate catalog and atlas
run: |
uv run \
--python 3.11 \
--extra dev \
--extra canonical \
python scripts/generate_pipeline_catalog.py
- name: Catalog must be committed
run: git diff --exit-code docs/pipeline_catalog.jsonl
python scripts/generate_pipeline_atlas.py
- name: Catalog and atlas must be committed
run: git diff --exit-code docs/pipeline_catalog.jsonl docs/atlas/index.html docs/atlas/pipeline_atlas.svg docs/atlas/pipeline_atlas_dark.svg

cli-smoke:
name: CLI smoke
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,7 @@ packages and Git.
| TTA | FAD, FD, KL, IS, CLAP score, AudioBox Aesthetics CE/CU/PC/PQ | [TTA](docs/tasks/tta.md) |
| VC | CER, WER, speaker similarity, DNSMOS, WV-MOS, UTMOS | [VC](docs/tasks/vc.md) |
| SE | SI-SDR, STOI, PESQ, DNSMOS, WV-MOS, UTMOS | [SE](docs/tasks/se.md) |
| RIR estimation | NPM, RT30/RT60 MAE, DRR/C50/C80/EDC, IACC full/early/late, ITD/ILD/IPD/DOA errors | [RIR estimation](docs/tasks/rir.md) |
| TSE | SI-SDR, speaker similarity, DNSMOS, WV-MOS, UTMOS, CER, WER | [TSE](docs/tasks/tse.md) |
| Classification / SER / GR | Accuracy | [Classification](docs/tasks/classification.md) |
| SLU | Accuracy | [SLU](docs/tasks/slu.md) |
Expand Down
1 change: 1 addition & 0 deletions README_ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,7 @@ node-local 虚拟环境只保留在本地,不会进入安装包或 Git。
| TTA | FAD、FD、KL、IS、CLAP score、AudioBox Aesthetics CE/CU/PC/PQ | [TTA](docs/tasks/tta.md) |
| VC | CER、WER、说话人相似度、DNSMOS、WV-MOS、UTMOS | [VC](docs/tasks/vc.md) |
| SE | SI-SDR、STOI、PESQ、DNSMOS、WV-MOS、UTMOS | [SE](docs/tasks/se.md) |
| RIR estimation | NPM、RT30/RT60 MAE、DRR/C50/C80/EDC、IACC 全窗/早期/晚期、ITD/ILD/IPD/DOA 误差 | [RIR estimation](docs/tasks/rir.md) |
| TSE | SI-SDR、说话人相似度、DNSMOS、WV-MOS、UTMOS、CER、WER | [TSE](docs/tasks/tse.md) |
| Classification / SER / GR | Accuracy | [Classification](docs/tasks/classification.md) |
| SLU | Accuracy | [SLU](docs/tasks/slu.md) |
Expand Down
12 changes: 6 additions & 6 deletions docs/atlas/index.html

Large diffs are not rendered by default.

20 changes: 10 additions & 10 deletions docs/atlas/pipeline_atlas.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 10 additions & 10 deletions docs/atlas/pipeline_atlas_dark.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 3 additions & 2 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ sure-eval env setup --pipeline pipeline.json --dry-run --json

Required PR checks are intentionally portable: they must pass on a clean GitHub
runner without model checkpoints, secrets, or node-local virtual environments.
They run `ruff`, whitespace checks, catalog freshness, package build/install
They run `ruff`, whitespace checks, catalog/atlas freshness, package build/install
smoke tests, CLI smoke tests, and the test suite excluding checkpoint-backed
environment assertions.

Expand All @@ -104,7 +104,8 @@ git diff --check
uv run --extra dev --extra audio --extra canonical pytest -q --ignore=tests/test_evaluation_env_check.py
uv run --extra dev --extra audio --extra canonical pytest tests/test_evaluation_env_check.py -q -k "in_process or pip_runtime or dry_run or doctor or cache_dir or se_pip_runtime_nodes_are_declared"
uv run --extra dev --extra canonical python scripts/generate_pipeline_catalog.py
git diff --exit-code docs/pipeline_catalog.jsonl
python scripts/generate_pipeline_atlas.py
git diff --exit-code docs/pipeline_catalog.jsonl docs/atlas/index.html docs/atlas/pipeline_atlas.svg docs/atlas/pipeline_atlas_dark.svg
```

Checkpoint-backed node-local checks live in the optional Heavy Nodes workflow.
Expand Down
19 changes: 19 additions & 0 deletions docs/pipeline_catalog.jsonl

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/tasks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ multi-metric bundles, see [Pipeline Catalog](../pipeline_catalog.md).
- [TTA — Text-to-Audio](./tta.md)
- [VC — Voice Conversion](./vc.md)
- [SE — Speech Enhancement](./se.md)
- [RIR — RIR Estimation and Spatial Audio](./rir.md)
- [TSE — Target Speaker Extraction](./tse.md)
- [Classification / SER / GR](./classification.md)
- [KWS — Keyword Spotting](./kws.md)
Expand Down
132 changes: 132 additions & 0 deletions docs/tasks/rir.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# RIR Estimation and Spatial Audio

The RIR estimation task evaluates predicted room impulse responses against reference RIRs.
Rows use `prediction_rir` and `reference_rir` paths and align by `sample_id`.
The audio files may be WAV/FLAC or NumPy `.npy`/`.npz` arrays. NumPy arrays
must provide `sample_rate` in the row because they do not carry an embedded
rate. The files must have matching sample rates and channel counts; spatial
metrics require at least two channels. Each pair is truncated to the shorter
file before scoring. NPM searches one common lag within +/-256 samples;
acoustic metrics begin each channel near its own strongest impulse; spatial
metrics use the channel signals as provided. Rows must refer to the same
source, room, microphone positions, and channel order. Each metric averages
its per-sample scores with equal weight. Every `per_sample` entry records
reference-side and prediction-side values with the unit. Multi-channel
spatial entries include each channel-0-relative pair; ILD/IPD additionally
report the valid frequency-bin count and absolute-error sum used for the mean.

An undefined RT30, RT60, DRR, clarity, or reference EDC estimate fails the
evaluation with the sample and channel identified; channels are never silently
dropped from the score. IACC and ITD also reject undefined correlation (silent
or constant channels, or an empty selected window). A silent prediction can
still receive an EDC MAE against a valid reference; its optional correlation
diagnostic can be undefined. Reports retain input paths, effective sample
rate, channel layout/count, geometry metadata, original lengths, and the
paired length in `details.rows`.

The truncation policy measures only the shared duration; it does not penalize
a missing tail. ITD and DOA retain the fixed +/-1 ms search window, including
for microphone arrays. Array delays outside that window are not supported by
this protocol and the reported angle is not a general array-localization
estimate. Choose recordings compatible with these limits before comparing
scores. NPM's separate +/-256-sample alignment is unchanged.

The evaluator has two channel branches. The single-channel branch accepts one
channel and computes the base NPM, RT30, RT60, DRR, C50, C80, and EDC metrics. The
multi-channel branch computes the same base metrics before adding spatial
metrics. IACC is restricted to an exact binaural pair. ITD, ILD, IPD, and DOA
accept two or more channels and report one pair for every `chN` relative to
`ch0`; the metric score is the mean over those pairs. Set `channel_layout` to
`mono`, `multi_channel`, or `binaural_lr` when the layout is known; `auto` is
used when it is omitted.

## Metrics

| Metric | Meaning | Direction |
|---|---|---|
| `rir_npm` | normalized projection misalignment in dB | lower is better |
| `rt30_mae` | measured 30 dB decay-time error in seconds | lower is better |
| `rt60_mae` | absolute RT60 error in seconds | lower is better |
| `rir_drr_mae` | direct-to-reverberant ratio error in dB | lower is better |
| `rir_c50_mae` | C50 error in dB | lower is better |
| `rir_c80_mae` | C80 error in dB | lower is better |
| `rir_edc_mae` | energy decay curve MAE above -40 dB in dB | lower is better |
| `rir_iacc_mae` | compatibility alias for full-window IACC error | lower is better |
| `rir_iacc_full_mae` | full-window interaural cross-correlation coefficient error in a +/-1 ms window | lower is better |
| `rir_iacc_early_mae` | IACC from the ch0 direct peak through 80 ms | lower is better |
| `rir_iacc_late_mae` | IACC after the ch0 direct peak plus 80 ms | lower is better |
| `rir_itd_mae` | interaural time difference error in ms in a +/-1 ms window | lower is better |
| `rir_ild_mae` | ch0-relative interchannel level error in dB | lower is better |
| `rir_ipd_mae` | ch0-relative interchannel phase error in radians | lower is better |
| `rir_doa_mae` | ch0-relative TDOA-derived DOA error in degrees | lower is better |

The default bundle is:

```text
rir.any.multi.rir_npm.rir_npm_v1__rt30_mae.rt30_mae_v1__rt60_mae.rt60_mae_v1__rir_drr_mae.rir_drr_mae_v1__rir_c50_mae.rir_c50_mae_v1__rir_c80_mae.rir_c80_mae_v1__rir_edc_mae.rir_edc_mae_v1
```

The bundle's top-level `score` is `rir_npm`; all metric scores are in
`details.results`. This is a reporting convention, not a combined score.

For cross-channel base evaluation, the curated bundle is:

```text
rir.any.multi.rir_npm.rir_npm_v1__rt30_mae.rt30_mae_v1__rt60_mae.rt60_mae_v1__rir_drr_mae.rir_drr_mae_v1__rir_c50_mae.rir_c50_mae_v1__rir_c80_mae.rir_c80_mae_v1__rir_edc_mae.rir_edc_mae_v1
```

It accepts the monaural and multi-channel branches because it contains only
the shared base metrics. The existing atomic routes remain available for
single-metric evaluation. Add the spatial bundle separately when the samples
meet the IACC two-channel contract.

For multi-channel spatial evaluation with more than two channels, use the
registered spatial bundle without IACC:

```text
rir.any.multi.rir_npm.rir_npm_v1__rt30_mae.rt30_mae_v1__rt60_mae.rt60_mae_v1__rir_drr_mae.rir_drr_mae_v1__rir_c50_mae.rir_c50_mae_v1__rir_c80_mae.rir_c80_mae_v1__rir_edc_mae.rir_edc_mae_v1__rir_itd_mae.rir_itd_mae_v1__rir_ild_mae.rir_ild_mae_v1__rir_ipd_mae.rir_ipd_mae_v1__rir_doa_mae.rir_doa_mae_v1
```

This bundle reports the remaining spatial metrics as channel-0-relative pairs;
IACC is intentionally absent because it is defined only for an exact binaural
pair.

For the binaural spatial chain, start with the same NPM, RT30, RT60, DRR,
C50, C80, and EDC base metrics, then select the registered bundle containing
`rir_iacc_full_mae`, `rir_iacc_early_mae`, `rir_iacc_late_mae`, `rir_itd_mae`,
`rir_ild_mae`, `rir_ipd_mae`, and `rir_doa_mae`. Its pipeline ID is:

```text
rir.any.multi.rir_npm.rir_npm_v1__rt30_mae.rt30_mae_v1__rt60_mae.rt60_mae_v1__rir_drr_mae.rir_drr_mae_v1__rir_c50_mae.rir_c50_mae_v1__rir_c80_mae.rir_c80_mae_v1__rir_edc_mae.rir_edc_mae_v1__rir_iacc_full_mae.rir_iacc_full_mae_v1__rir_iacc_early_mae.rir_iacc_early_mae_v1__rir_iacc_late_mae.rir_iacc_late_mae_v1__rir_itd_mae.rir_itd_mae_v1__rir_ild_mae.rir_ild_mae_v1__rir_ipd_mae.rir_ipd_mae_v1__rir_doa_mae.rir_doa_mae_v1
```

Its top-level score is `rir_npm`; spatial scores remain separate entries in
`details.results`.

Example row:

```json
{"sample_id":"room001_utt001","prediction_rir":"pred.wav","reference_rir":"ref.wav","sample_rate":16000,"channel_layout":"mono"}
```

All three IACC window metrics require exactly two channels. The other spatial metrics use
channel 0 as the reference channel and report `pair_count` plus per-channel
pair details. ITD uses a normalized cross-correlation search over +/-1 ms.
`rir_doa_mae` additionally requires `metadata.microphone_spacing_m`; for
channel `N` the default spacing is `N * microphone_spacing_m`, or an explicit
`channel_spacings_m` list can be supplied. Its optional `sound_speed_mps`
defaults to 343 m/s.

The scoring nodes run in-process with the package's `audio` extra. They do
not need model checkpoints or node-local environments. A model may generate
the predicted RIR in a separate environment such as `usddps-recrir`; scoring
only consumes the resulting audio files.

## CLI

```bash
sure-eval metric routes rir --metric npm --json
sure-eval metric routes rir --metric iacc --json
sure-eval metric describe rir --metric npm --output rir_pipeline.json --json
sure-eval metric run --pipeline rir_pipeline.json --samples-jsonl rir_samples.jsonl --output-dir /tmp/rir_eval --json
```
55 changes: 55 additions & 0 deletions scripts/generate_pipeline_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
"doa",
"lid",
"sv",
"rir",
)

# Multi pipelines are curated presets. Atomic pipelines are always discovered
Expand All @@ -59,6 +60,60 @@
("se", {"metrics": ("si-sdr", "stoi", "pesq", "dnsmos", "wv-mos", "utmos")}),
("tse", {"language": "zh", "metrics": ("si_sdr", "sim/wavlm-large", "dnsmos")}),
("sv", {"metrics": ("eer", "min_dcf")}),
("rir", {"metrics": ("rir_npm", "rt60_mae")}),
(
"rir",
{
"metrics": (
"rir_npm",
"rt30_mae",
"rt60_mae",
"rir_drr_mae",
"rir_c50_mae",
"rir_c80_mae",
"rir_edc_mae",
)
},
),
(
"rir",
{
"metrics": (
"rir_npm",
"rt30_mae",
"rt60_mae",
"rir_drr_mae",
"rir_c50_mae",
"rir_c80_mae",
"rir_edc_mae",
"rir_iacc_full_mae",
"rir_iacc_early_mae",
"rir_iacc_late_mae",
"rir_itd_mae",
"rir_ild_mae",
"rir_ipd_mae",
"rir_doa_mae",
)
},
),
(
"rir",
{
"metrics": (
"rir_npm",
"rt30_mae",
"rt60_mae",
"rir_drr_mae",
"rir_c50_mae",
"rir_c80_mae",
"rir_edc_mae",
"rir_itd_mae",
"rir_ild_mae",
"rir_ipd_mae",
"rir_doa_mae",
)
},
),
)


Expand Down
39 changes: 38 additions & 1 deletion src/sure_eval/evaluation/audio_samples.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
from sure_eval.evaluation.tasks.vc.types import VCSample
from sure_eval.evaluation.tasks.se.types import SESample
from sure_eval.evaluation.tasks.tta.types import TTASample
from sure_eval.evaluation.tasks.rir.types import RIRSample


class SampleJsonlError(ValueError):
Expand Down Expand Up @@ -201,6 +202,40 @@ def load_tta_samples_jsonl(
return samples


def load_rir_samples_jsonl(
path: str | Path, *, metrics: tuple[str, ...] | list[str] | None = None
) -> list[RIRSample]:
rows = _read_rows(Path(path))
_validate_common(rows)
samples: list[RIRSample] = []
for row in rows:
line_no = row["_line_no"]
_require_fields(row, line_no, ("sample_id", "prediction_rir", "reference_rir"))
for field in ("prediction_rir", "reference_rir"):
if not isinstance(row[field], str):
_fail(line_no, f"{field} must be a path string")
metadata = _metadata(row, line_no)
sample_rate = row.get("sample_rate", metadata.get("sample_rate"))
if sample_rate is not None:
try:
sample_rate = int(sample_rate)
except (TypeError, ValueError):
_fail(line_no, "sample_rate must be an integer")
samples.append(
RIRSample(
prediction_rir=_resolve_existing_path(row["prediction_rir"], path, line_no, "prediction_rir"),
reference_rir=_resolve_existing_path(row["reference_rir"], path, line_no, "reference_rir"),
observed_audio=_resolve_optional_path(row.get("observed_audio"), path, line_no, "observed_audio"),
reference_audio=_resolve_optional_path(row.get("reference_audio"), path, line_no, "reference_audio"),
sample_rate=sample_rate,
channel_layout=str(row.get("channel_layout", metadata.get("channel_layout", "auto"))),
sample_id=str(row["sample_id"]),
metadata=metadata,
)
)
return samples


def _read_rows(path: Path) -> list[dict[str, Any]]:
if not path.exists():
raise SampleJsonlError(f"samples_jsonl file not found: {path}")
Expand Down Expand Up @@ -268,7 +303,9 @@ def _resolve_path(value: Any, jsonl_path: str | Path) -> Path:


def _metadata(row: dict[str, Any], line_no: int) -> dict[str, Any]:
metadata = row.get("metadata") or {}
metadata = row.get("metadata")
if metadata is None:
metadata = {}
if not isinstance(metadata, dict):
_fail(line_no, "metadata must be an object when present")
return dict(metadata)
Expand Down
2 changes: 1 addition & 1 deletion src/sure_eval/evaluation/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ def run_metric_pipeline(
None, "--match-tolerance-sec", help="KWS event matching tolerance in seconds (default 1.0)"
),
samples_jsonl: Optional[str] = typer.Option(
None, "--samples-jsonl", help="TTS/TTA/VC/SE/TSE samples JSONL file"
None, "--samples-jsonl", help="TTS/TTA/VC/SE/TSE/RIR samples JSONL file"
),
device: str = typer.Option(
"cuda", "--device", help="Device passed to audio model runtime builders"
Expand Down
14 changes: 13 additions & 1 deletion src/sure_eval/evaluation/cli_adapters.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
"Check selected node directories for pyproject.toml or uv.lock when preparing a run."
)

AUDIO_SAMPLE_TASKS = {"tts", "vc", "se", "speech_enhancement", "tse", "tta"}
AUDIO_SAMPLE_TASKS = {"tts", "vc", "se", "speech_enhancement", "tse", "tta", "rir"}
SE_TASKS = {"se", "speech_enhancement"}
SE_DEFAULT_METRICS = ("si-sdr", "stoi", "pesq", "dnsmos", "wv-mos", "utmos")
SV_DEFAULT_METRICS = ("eer", "min_dcf")
Expand Down Expand Up @@ -615,6 +615,13 @@ def _describe_kwargs(
if pipeline_id:
kwargs["pipeline_id"] = pipeline_id
return kwargs
if task == "rir":
kwargs = {}
if metric:
kwargs["metrics"] = split_metric_csv(metric)
if pipeline_id:
kwargs["pipeline_id"] = pipeline_id
return kwargs
return {}


Expand Down Expand Up @@ -876,6 +883,11 @@ def _audio_sample_kwargs(

samples = load_tta_samples_jsonl(samples_jsonl, metrics=metrics)
runtime = build_tta_runtime(metrics=metrics, device=device, cache_dir=cache_dir)
elif task == "rir":
from sure_eval.evaluation.audio_samples import load_rir_samples_jsonl

samples = load_rir_samples_jsonl(samples_jsonl, metrics=metrics)
runtime = {}
else:
return {}
payload = {"samples": samples}
Expand Down
Loading
Loading