Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ packages and Git.
| TTA | FAD, FD, KL, IS, CLAP score, AudioBox Aesthetics CE/CU/PC/PQ | [TTA](docs/tasks/tta.md) |
| VC | CER, WER, speaker similarity, DNSMOS, WV-MOS, UTMOS | [VC](docs/tasks/vc.md) |
| SE | SI-SDR, STOI/eSTOI, PESQ, LSD, nSRMR, LPS, DNSMOS P.835/P.808 components, WV-MOS, UTMOS, NISQA DIM, SQUIM objective predictions | [SE](docs/tasks/se.md) |
| RIR estimation | NPM, RT30/RT60 MAE, DRR/C50/C80/EDC, IACC full/early/late, ITD/ILD/IPD/DOA errors | [RIR estimation](docs/tasks/rir.md) |
| RIR estimation | NPM, RT30/RT60 MAE, DRR/C50/C80/EDC, octave-band and broadband EDT, ISO 3382 EDT/C80/D50/Ts JND, IACC full/early/late, ITD/ILD/IPD/DOA errors | [RIR estimation](docs/tasks/rir.md) |
| TSE | SI-SDR, speaker similarity, DNSMOS, WV-MOS, UTMOS, CER, WER | [TSE](docs/tasks/tse.md) |
| Classification / SER / GR | Accuracy | [Classification](docs/tasks/classification.md) |
| SLU | Accuracy | [SLU](docs/tasks/slu.md) |
Expand Down
2 changes: 1 addition & 1 deletion README_ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ node-local 虚拟环境只保留在本地,不会进入安装包或 Git。
| TTA | FAD、FD、KL、IS、CLAP score、AudioBox Aesthetics CE/CU/PC/PQ | [TTA](docs/tasks/tta.md) |
| VC | CER、WER、说话人相似度、DNSMOS、WV-MOS、UTMOS | [VC](docs/tasks/vc.md) |
| SE | SI-SDR、STOI/eSTOI、PESQ、LSD、nSRMR、LPS、DNSMOS P.835/P.808 分量、WV-MOS、UTMOS、NISQA DIM、SQUIM 预测指标 | [SE](docs/tasks/se.md) |
| RIR estimation | NPM、RT30/RT60 MAE、DRR/C50/C80/EDC、IACC 全窗/早期/晚期、ITD/ILD/IPD/DOA 误差 | [RIR estimation](docs/tasks/rir.md) |
| RIR estimation | NPM、RT30/RT60 MAE、DRR/C50/C80/EDC、倍频带与宽带 EDT、ISO 3382 EDT/C80/D50/Ts JND、IACC 全窗/早期/晚期、ITD/ILD/IPD/DOA 误差 | [RIR estimation](docs/tasks/rir.md) |
| TSE | SI-SDR、说话人相似度、DNSMOS、WV-MOS、UTMOS、CER、WER | [TSE](docs/tasks/tse.md) |
| Classification / SER / GR | Accuracy | [Classification](docs/tasks/classification.md) |
| SLU | Accuracy | [SLU](docs/tasks/slu.md) |
Expand Down
12 changes: 6 additions & 6 deletions docs/atlas/index.html

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/atlas/pipeline_atlas.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/atlas/pipeline_atlas_dark.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 7 additions & 1 deletion docs/pipeline_catalog.jsonl

Large diffs are not rendered by default.

57 changes: 48 additions & 9 deletions docs/tasks/rir.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,11 @@ Rows use `prediction_rir` and `reference_rir` paths and align by `sample_id`.
The audio files may be WAV/FLAC or NumPy `.npy`/`.npz` arrays. NumPy arrays
must provide `sample_rate` in the row because they do not carry an embedded
rate. The files must have matching sample rates and channel counts; spatial
metrics require at least two channels. Each pair is truncated to the shorter
file before scoring. NPM searches one common lag within +/-256 samples;
acoustic metrics begin each channel near its own strongest impulse; spatial
metrics use the channel signals as provided. Rows must refer to the same
metrics require at least two channels. NPM searches one common lag within
`+/-256` samples over the full union support. Acoustic metrics independently
start each prediction/reference channel 2.5 ms before its strongest impulse,
then use their shared duration; spatial metrics keep the original channel
timing. Rows must refer to the same
source, room, microphone positions, and channel order. Each metric averages
its per-sample scores with equal weight. Every `per_sample` entry records
reference-side and prediction-side values with the unit. Multi-channel
Expand All @@ -24,8 +25,8 @@ diagnostic can be undefined. Reports retain input paths, effective sample
rate, channel layout/count, geometry metadata, original lengths, and the
paired length in `details.rows`.

The truncation policy measures only the shared duration; it does not penalize
a missing tail. ITD and DOA retain the fixed +/-1 ms search window, including
The acoustic shared-duration policy measures only the common duration; it does
not penalize a missing tail. ITD and DOA retain the fixed +/-1 ms search window, including
for microphone arrays. Array delays outside that window are not supported by
this protocol and the reported angle is not a general array-localization
estimate. Choose recordings compatible with these limits before comparing
Expand All @@ -45,14 +46,19 @@ used when it is omitted.
| Metric | Meaning | Direction |
|---|---|---|
| `rir_npm` | normalized projection misalignment in dB | lower is better |
| `rt30_mae` | measured 30 dB decay-time error in seconds | lower is better |
| `rt30_mae` | RT30 error in seconds, with the -5 to -35 dB EDC slope extrapolated to 60 dB | lower is better |
| `rt60_mae` | absolute RT60 error in seconds | lower is better |
| `rir_drr_mae` | direct-to-reverberant ratio error in dB | lower is better |
| `rir_c50_mae` | C50 error in dB | lower is better |
| `rir_c80_mae` | C80 error in dB | lower is better |
| `rir_edc_mae` | energy decay curve MAE above -40 dB in dB | lower is better |
| `rir_iacc_mae` | compatibility alias for full-window IACC error | lower is better |
| `rir_iacc_full_mae` | full-window interaural cross-correlation coefficient error in a +/-1 ms window | lower is better |
| `rir_edt_m_mae` | 500 Hz/1 kHz octave-band EDT MAE in seconds | lower is better |
| `rir_edt_broadband_mae` | broadband EDT MAE in seconds | lower is better |
| `rir_edt_jnd_mae` | EDT error divided by 5% reference EDT (`x_jnd`) | lower is better |
| `rir_c80_jnd_mae` | C80 error divided by 1 dB (`x_jnd`) | lower is better |
| `rir_d50_jnd_mae` | D50 error divided by 0.05 (`x_jnd`) | lower is better |
| `rir_ts_jnd_mae` | centre-time error divided by 10 ms (`x_jnd`) | lower is better |
| `rir_iacc_full_mae` | full-window interaural cross-correlation coefficient error in a +/-1 ms window; `iacc`, `rir_iacc`, and `rir_iacc_mae` are compatibility selectors | lower is better |
| `rir_iacc_early_mae` | IACC from the ch0 direct peak through 80 ms | lower is better |
| `rir_iacc_late_mae` | IACC after the ch0 direct peak plus 80 ms | lower is better |
| `rir_itd_mae` | interaural time difference error in ms in a +/-1 ms window | lower is better |
Expand All @@ -69,6 +75,39 @@ rir.any.multi.rir_npm.rir_npm_v1__rt30_mae.rt30_mae_v1__rt60_mae.rt60_mae_v1__ri
The bundle's top-level `score` is `rir_npm`; all metric scores are in
`details.results`. This is a reporting convention, not a combined score.

## ISO 3382 EDT and JND routes

`rir_edt_m_mae` follows the leaderboard protocol: detect the first broadband
sample within 20 dB of the peak, filter at 500 Hz and 1 kHz with a zero-phase
Butterworth octave-band filter, fit the initial 0 to -10 dB Schroeder decay,
extrapolate to 60 dB, average the two band values per channel, and report
seconds of absolute error. `rir_edt_broadband_mae` is an optional strict atomic
route. It performs the same decay fit on the full-band signal with one
broadband onset shared by the RIR's channels and does not apply a JND threshold. It is not part of the
leaderboard or JND bundle because some measured RIRs cross -10 dB within fewer
than four samples, which is insufficient for a meaningful broadband EDT fit.

The four `*_jnd_mae` routes use the informative BS EN ISO 3382-1:2009 Annex A
Table A.1 thresholds: EDT 5% of the reference EDT, C80 1 dB, D50 0.05, and Ts
10 ms. Each route reports a dimensionless mean error in `x_jnd`, plus
`absolute_mae`, `mean_jnd`, `within_one_jnd_percent`, `valid_samples`, and
`total_samples` in its result details. All four routes support one or more
channels. Channel errors are averaged; a sample counts as within one JND only
when every channel passes. This channel aggregation and pass-rate definition
are benchmark rules, not ISO-prescribed auditorium reporting.

The standard's frequency averaging is retained for all four JND parameters:
500 Hz and 1 kHz are averaged per channel. The implementation uses a sixth
order Butterworth prototype and zero-phase filtering; IEC 61260 certification,
measured background levels, source calibration, and auditorium measurement
conditions are not claimed. Existing `rir_edc_mae` remains an energy-curve
error in dB and must not be interpreted as EDT.

The JND routes are the ISO-parameter benchmark subset. The base `rir_c50_mae`
and `rir_c80_mae` routes are broadband Rec-BIR comparison metrics, while NPM,
DRR, EDC, and the spatial metrics are model-evaluation extensions; none of
these routes is a complete ISO 3382 auditorium measurement report.

For cross-channel base evaluation, the curated bundle is:

```text
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ diarization = [
audio = [
"soundfile>=0.12.0",
"librosa>=0.10.0",
"scipy>=1.6.0",
]
se = [
"pystoi>=0.4.1",
Expand Down
1 change: 1 addition & 0 deletions scripts/generate_pipeline_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@
)
},
),
("rir", {"metrics": ("rir_edt_jnd_mae", "rir_c80_jnd_mae", "rir_d50_jnd_mae", "rir_ts_jnd_mae")}),
)


Expand Down
21 changes: 17 additions & 4 deletions src/sure_eval/evaluation/cli_adapters.py
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ def list_metric_routes(
routes, _ = load_task_routes(route_task)
requested_language = _normalize_language_filter(language)
requested_metric = canonical_metric(metric) if metric else None
requested_metric_tokens = _metric_filter_tokens(metric) if metric else set()
language_sensitive = bool(routes.get("language_sensitive", False))
if not language_sensitive and requested_language not in {None, "n/a"}:
raise ValueError(f"Task {task!r} is language-independent; omit --language or use any")
Expand All @@ -146,10 +147,7 @@ def list_metric_routes(
route = dict(configured_route)
if not _route_matches_task_alias(route, requested_task):
continue
if (
requested_metric
and canonical_metric(str(route.get("metric") or "")) != requested_metric
):
if requested_metric_tokens and not requested_metric_tokens.intersection(_route_metric_tokens(route)):
continue
route_language = _concrete_route_language(
route,
Expand Down Expand Up @@ -291,6 +289,21 @@ def _route_matches_task_alias(route: dict[str, Any], requested_task: str) -> boo
return normalize_task(str(route_alias)) == requested_task


def _metric_filter_tokens(metric: str) -> set[str]:
"""Return raw and canonical forms accepted by route selection."""

raw = str(metric).strip().lower().replace("-", "_")
return {raw, canonical_metric(raw)}


def _route_metric_tokens(route: dict[str, Any]) -> set[str]:
tokens: set[str] = set()
for value in (route.get("metric"), *(route.get("aliases") or ())):
raw = str(value).strip().lower().replace("-", "_")
tokens.update((raw, canonical_metric(raw)))
return tokens


def _route_matches_language(route: dict[str, Any], requested_language: str | None) -> bool:
if requested_language is None:
return True
Expand Down
52 changes: 52 additions & 0 deletions src/sure_eval/evaluation/nodes/scoring/rir_c80_jnd_mae/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# ISO 3382 C80 JND Error

## Purpose

Compares predicted and reference C80 in the benchmark's ISO 3382 octave-band
protocol. It reports a C80 error in multiples of the 1 dB JND; it is not a
complete auditorium measurement report.

## Task Scenarios

- `rir/any/rir_c80_jnd_mae` selects this atomic scoring node.
- The node is also available in the RIR ISO 3382 JND metric set.

## Input

- Schema: `rir_samples_jsonl`.
- Required roles: `sample_id`, `prediction_rir`, `reference_rir`.
- NumPy inputs require row-level `sample_rate`; rows align by `sample_id`.

## Output

- Schema: `rir_c80_jnd_mae_score`.
- Lower is better; the score unit is `x_jnd`.
- Details retain per-channel prediction/reference C80, absolute error, JND
multiple, pass flag, absolute MAE, and within-one-JND sample percentage.

## Versioned Computation

- Node `scoring/rir_c80_jnd_mae`, version `v1`.
- Detect one broadband onset per RIR, filter at 500 Hz and 1 kHz with the
zero-phase Butterworth octave-band filters, and compute the 80 ms early/late
energy ratio in each band.
- Average the two band values per channel, divide absolute error by 1 dB,
average channels then samples, and require every channel for a sample pass.

## Runtime and Assets

- Runtime: `in_process`.
- Requires NumPy, SoundFile, and SciPy from `sure-evaluation[audio]`.
- No checkpoint, binary, or node-local environment is used.

## Source and References

- Local implementation: `sure_eval.evaluation.tasks.rir.metrics`.
- Threshold: BS EN ISO 3382-1:2009 Annex A Table A.1, applied as a benchmark
comparison rule.

## Limitations

The channel aggregation and pass-rate rule are benchmark conventions. Missing
or undefined band energy fails the strict route; IEC 61260 certification,
background-noise correction, and auditorium calibration are not claimed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from sure_eval.evaluation.tasks.rir.metrics import rir_c80_jnd_mae

__all__ = ["rir_c80_jnd_mae"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
id: scoring/rir_c80_jnd_mae
version: v1
stage: scoring
language_sensitive: false
input_schema: rir_samples_jsonl
output_schema: rir_c80_jnd_mae_score
implementation: sure_eval.evaluation.tasks.rir.metrics.rir_c80_jnd_mae
metric: rir_c80_jnd_mae
backend: scipy-rir
internal_stages: [audio_load, direct_onset, octave_band_filter, c80_fit, jnd_normalization, mean_aggregation]
used_by: [RIR ISO 3382 JND]
notes: [Lower is better. Unit is x_jnd. Supports one or more channels. C80 JND is 1 dB.]
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from sure_eval.evaluation.tasks.rir.metrics import rir_c80_jnd_mae

__all__ = ["rir_c80_jnd_mae"]
52 changes: 52 additions & 0 deletions src/sure_eval/evaluation/nodes/scoring/rir_d50_jnd_mae/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# ISO 3382 D50 JND Error

## Purpose

Compares predicted and reference D50 in the benchmark's ISO 3382 octave-band
protocol. It reports a D50 error in multiples of the 0.05 JND; it is not a
complete auditorium measurement report.

## Task Scenarios

- `rir/any/rir_d50_jnd_mae` selects this atomic scoring node.
- The node is also available in the RIR ISO 3382 JND metric set.

## Input

- Schema: `rir_samples_jsonl`.
- Required roles: `sample_id`, `prediction_rir`, `reference_rir`.
- NumPy inputs require row-level `sample_rate`; rows align by `sample_id`.

## Output

- Schema: `rir_d50_jnd_mae_score`.
- Lower is better; the score unit is `x_jnd`.
- Details retain per-channel prediction/reference D50, absolute error, JND
multiple, pass flag, absolute MAE, and within-one-JND sample percentage.

## Versioned Computation

- Node `scoring/rir_d50_jnd_mae`, version `v1`.
- Detect one broadband onset per RIR, filter at 500 Hz and 1 kHz with the
zero-phase Butterworth octave-band filters, and compute the first 50 ms
energy divided by total energy in each band.
- Average the two band values per channel, divide absolute error by 0.05,
average channels then samples, and require every channel for a sample pass.

## Runtime and Assets

- Runtime: `in_process`.
- Requires NumPy, SoundFile, and SciPy from `sure-evaluation[audio]`.
- No checkpoint, binary, or node-local environment is used.

## Source and References

- Local implementation: `sure_eval.evaluation.tasks.rir.metrics`.
- Threshold: BS EN ISO 3382-1:2009 Annex A Table A.1, applied as a benchmark
comparison rule.

## Limitations

The channel aggregation and pass-rate rule are benchmark conventions. Missing
or undefined band energy fails the strict route; IEC 61260 certification,
background-noise correction, and auditorium calibration are not claimed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from sure_eval.evaluation.tasks.rir.metrics import rir_d50_jnd_mae

__all__ = ["rir_d50_jnd_mae"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
id: scoring/rir_d50_jnd_mae
version: v1
stage: scoring
language_sensitive: false
input_schema: rir_samples_jsonl
output_schema: rir_d50_jnd_mae_score
implementation: sure_eval.evaluation.tasks.rir.metrics.rir_d50_jnd_mae
metric: rir_d50_jnd_mae
backend: scipy-rir
internal_stages: [audio_load, direct_onset, octave_band_filter, d50_fit, jnd_normalization, mean_aggregation]
used_by: [RIR ISO 3382 JND]
notes: [Lower is better. Unit is x_jnd. Supports one or more channels. D50 JND is 0.05.]
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from sure_eval.evaluation.tasks.rir.metrics import rir_d50_jnd_mae

__all__ = ["rir_d50_jnd_mae"]
6 changes: 3 additions & 3 deletions src/sure_eval/evaluation/nodes/scoring/rir_drr_mae/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ then averages samples equally.
## Versioned Computation

Node `scoring/rir_drr_mae`, version `v1`, starts each channel 2.5 ms before
its strongest impulse. It divides energy from the strongest impulse through
the following 2.5 ms by later energy, converts the ratio with `10*log10`,
and compares prediction with reference.
its strongest impulse. It divides energy in the symmetric window from 2.5 ms
before through 2.5 ms after the strongest impulse by later energy, converts
the ratio with `10*log10`, and compares prediction with reference.

## Runtime and Assets

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Broadband EDT MAE

## Purpose

Reports full-band EDT error in seconds. This optional route is a model
comparison diagnostic and is distinct from ISO octave-band EDT.

## Task Scenarios

- `rir/any/rir_edt_broadband_mae` selects this atomic scoring node.
- It is optional and is not part of the leaderboard ISO JND bundle.

## Input

- Schema: `rir_samples_jsonl`.
- Required roles: `sample_id`, `prediction_rir`, `reference_rir`.
- NumPy inputs require row-level `sample_rate`; rows align by `sample_id`.

## Output

- Schema: `rir_edt_broadband_mae_score`.
- Lower is better; the score unit is seconds.
- Details retain per-channel prediction/reference EDT and absolute errors.

## Versioned Computation

- Node `scoring/rir_edt_broadband_mae`, version `v1`.
- Detect one broadband onset shared by the RIR's channels, compute the full-band
Schroeder EDC, fit 0 to -10 dB, extrapolate the slope to 60 dB, and average
channel errors then samples.

## Runtime and Assets

- Runtime: `in_process`.
- Requires NumPy and the optional audio loader; no checkpoint or binary.
- No node-local environment is used.

## Source and References

- Local implementation: `sure_eval.evaluation.tasks.rir.metrics`.
- The companion `rir_edt_m_mae` route follows the RIR leaderboard's octave-band
benchmark protocol.

## Limitations

This route has no ISO Table A.1 JND normalization. Undefined decay fits fail
the strict route; short RIRs may not contain enough samples for a stable fit.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from sure_eval.evaluation.tasks.rir.metrics import rir_edt_broadband_mae

__all__ = ["rir_edt_broadband_mae"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
id: scoring/rir_edt_broadband_mae
version: v1
stage: scoring
language_sensitive: false
input_schema: rir_samples_jsonl
output_schema: rir_edt_broadband_mae_score
implementation: sure_eval.evaluation.tasks.rir.metrics.rir_edt_broadband_mae
metric: rir_edt_broadband_mae
backend: numpy-rir
internal_stages: [audio_load, shared_direct_onset, schroeder_decay, edt_fit, mean_aggregation]
used_by: [RIR room acoustics]
notes: [Lower is better. Unit is seconds. Full-band 0 to -10 dB decay fit extrapolated to 60 dB.]
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from sure_eval.evaluation.tasks.rir.metrics import rir_edt_broadband_mae

__all__ = ["rir_edt_broadband_mae"]
Loading
Loading