Skip to content

Feat/rir iso3382 jnd v3 - #43

Open
Yj-Xiong wants to merge 3 commits into
X-LANCE:mainfrom
Yj-Xiong:feat/rir-iso3382-jnd-v3
Open

Yj-Xiong wants to merge 3 commits into
X-LANCE:mainfrom
Yj-Xiong:feat/rir-iso3382-jnd-v3

Conversation

@Yj-Xiong

@Yj-Xiong Yj-Xiong commented Nov 10, 2026 •

Copy link
Copy Markdown

Start Here## Start Here

Read Contributing. This contribution follows the New Metric PR Guide; the fixes to existing scoring nodes are also checked against the Node Change PR Guide.

Summary

  • Add six RIR metrics: 500 Hz/1 kHz octave-band EDT MAE, optional broadband EDT MAE, and EDT/C80/D50/Ts errors normalized by their individual JND thresholds. Each JND result includes absolute MAE, mean JND multiple, per-sample channel details, and the metric-specific within-one-JND rate. No combined pass rate is introduced.
  • Align existing RIR scoring with the documented leaderboard protocol: NPM uses full union support and a shared ±256-sample lag; RT30 fits the −5 to −35 dB EDC slope and extrapolates to 60 dB; DRR uses a ±2.5 ms direct-sound window; C50/C80 use the exact early/late split; IACC/ITD use per-lag overlap normalization. The implementation keeps channel-relative timing for spatial metrics.
  • Remove the duplicate registered rir_iacc_mae route. The iacc, rir_iacc, and rir_iacc_mae selectors resolve to rir_iacc_full_mae; full/early/late IACC remain separate metrics and require exactly two channels. Other spatial metrics accept two or more channels and average each chN versus ch0 pair within a sample.

The RIR evaluation chain has not been released to online users. This PR retains
the existing v1 node and pipeline identities as the first release protocol.
Internal results produced with the earlier implementation must be recomputed
before comparison with results from this branch.

The RIR row contract is unchanged: prediction_rir and reference_rir are required, rows align by sample_id, NumPy inputs also require sample_rate, and DOA additionally requires microphone-spacing metadata. Acoustic and ISO parameter metrics accept one or more channels. An undefined channel result fails the strict route rather than being silently excluded.

Metric definitions and aggregation

For the ISO parameter routes, each predicted/reference RIR has one broadband direct onset (the first sample within 20 dB of its peak). A zero-phase, sixth-order-prototype Butterworth octave-band filter is applied at 500 Hz and 1 kHz. Band parameter values are averaged arithmetically per channel before the predicted/reference absolute error is taken. Channel errors are averaged per sample; sample scores receive equal weight.

Parameter Band value JND divisor Absolute-error unit
EDT Fit the 0 to −10 dB Schroeder EDC slope and extrapolate −60/slope 0.05 × reference EDT seconds
C80 10 log10(E[0,80 ms) / E[80 ms,end)) 1 dB dB
D50 E[0,50 ms) / E[0,end) 0.05 ratio
Ts 1000 × Σ(t·h²) / Σ(h²) from onset 10 ms ms

The reported *_jnd_mae score is the mean of abs(prediction − reference) / JND divisor, in x_jnd; lower is better. within_one_jnd_percent counts samples for which every channel is within that metric's threshold, divided by the number of successfully scored samples. The route fails if any channel is undefined, so a successful report has valid_samples == total_samples. These channel and pass-rate rules are benchmark conventions. The filter, background-noise treatment, and calibration are not claimed as a complete ISO 3382 auditorium measurement.

PR Type

Select one guide:

Guide followed: New Metric; Node Change requirements are relevant to the corrections of existing scoring nodes.

Evaluation Identity

  • Reported metric: rir_edt_m_mae, rir_edt_broadband_mae, rir_edt_jnd_mae, rir_c80_jnd_mae, rir_d50_jnd_mae, rir_ts_jnd_mae. The JND bundle reports metric=multi with four separate results.
  • execution_metrics, if different: atomic routes use the corresponding canonical metric. The JND bundle executes rir_edt_jnd_mae, rir_c80_jnd_mae, rir_d50_jnd_mae, and rir_ts_jnd_mae. Short names such as edt_jnd are selectors only.
  • Added atomic pipeline_id values:
rir.any.rir_edt_m_mae.rir_edt_m_mae_v1
rir.any.rir_edt_broadband_mae.rir_edt_broadband_mae_v1
rir.any.rir_edt_jnd_mae.rir_edt_jnd_mae_v1
rir.any.rir_c80_jnd_mae.rir_c80_jnd_mae_v1
rir.any.rir_d50_jnd_mae.rir_d50_jnd_mae_v1
rir.any.rir_ts_jnd_mae.rir_ts_jnd_mae_v1
  • Added JND bundle pipeline_id:
rir.any.multi.rir_edt_jnd_mae.rir_edt_jnd_mae_v1__rir_c80_jnd_mae.rir_c80_jnd_mae_v1__rir_d50_jnd_mae.rir_d50_jnd_mae_v1__rir_ts_jnd_mae.rir_ts_jnd_mae_v1
  • Removed duplicate atomic pipeline_id: rir.any.rir_iacc_mae.rir_iacc_mae_v1. Its legacy selectors target rir.any.rir_iacc_full_mae.rir_iacc_full_mae_v1.
  • Default route changed: no selection change. The default bundle still selects the same metrics, but some scores change under the corrected formulas.
  • Score comparability changed: yes. Existing NPM, RT30/RT60, EDC, DRR, C50/C80, IACC, ITD, and ITD-derived DOA values may differ from earlier reports. Compare scores only when the route/node version and input protocol match.

Changed Declarations

  • Task manifest: yes; the six metrics and their input contracts are registered, and the duplicate IACC contract is removed.
  • Task routes: yes; six atomic routes are added, and rir_iacc_mae becomes a compatibility selector for rir_iacc_full_mae.
  • Node manifest: yes for six new scoring nodes; the broadband EDT stage description is corrected.
  • node_env.yaml: n/a; all affected RIR scoring nodes run in_process without checkpoints or binaries. SciPy is declared in the package's audio extra.
  • Node README follows docs/node_readme_template.md: yes for the six new nodes (all required sections); existing affected node descriptions retain v1 for the pre-release protocol decision noted above.
  • Catalog regenerated: yes; 229 total rows, including 20 RIR atomic routes and 5 curated RIR bundles.
  • Catalog discovery task or curated bundle changed: the discovery task list is unchanged (rir was already registered); one four-metric JND bundle is added.
  • Documentation coverage checked against docs/contributing.md: README.md, README_ZH.md, docs/tasks/rir.md, the task-local README, affected node READMEs, catalog, and atlas are updated. The task index already links to RIR and needs no change.

Validation

Executed on the v3 branch (7dd2b8a):

OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 PYTHONPATH=src python -m pytest -q tests/test_rir_task.py
# 63 passed
ruff check .
# All checks passed
git diff --check origin/main...HEAD
PYTHONPATH=src python scripts/generate_pipeline_catalog.py
python scripts/generate_pipeline_atlas.py
# 229 catalog rows; generated atlas matches the committed files

A small synthetic two-channel RIR fixture with 4096 samples at 16 kHz, a direct impulse, and an exponential tail was generated as follows:

python - <<'PYCODE'
import json
from pathlib import Path
import numpy as np
import soundfile as sf

out = Path('/tmp/rir-pr-v3-description-smoke-261110')
out.mkdir(parents=True, exist_ok=True)
rir = np.zeros((4096, 2), dtype=np.float64)
rir[8, 0], rir[8, 1] = 1.0, 0.8
tail = np.exp(-np.arange(4088) / 600.0)
rir[8:, 0] += 0.05 * tail
rir[8:, 1] += 0.04 * tail
sf.write(out / 'reference.wav', rir, 16000, subtype='FLOAT')
sf.write(out / 'prediction.wav', 0.9 * rir, 16000, subtype='FLOAT')
row = {'sample_id': 'rir-jnd-smoke', 'prediction_rir': 'prediction.wav', 'reference_rir': 'reference.wav', 'sample_rate': 16000, 'channel_layout': 'binaural_lr'}
(out / 'samples.jsonl').write_text(json.dumps(row) + '\n', encoding='utf-8')
PYCODE

The following exact route chain completed successfully and preserved the same pipeline_id in report.json and pipeline_description.json:

PYTHONPATH=src sure-eval metric routes rir --metric rir_edt_jnd_mae --json
PYTHONPATH=src sure-eval metric describe rir --pipeline-id rir.any.rir_edt_jnd_mae.rir_edt_jnd_mae_v1 --output /tmp/rir-pr-v3-description-smoke-261110/pipeline.json --json
PYTHONPATH=src sure-eval env setup --pipeline /tmp/rir-pr-v3-description-smoke-261110/pipeline.json --dry-run --json
PYTHONPATH=src sure-eval env check --pipeline /tmp/rir-pr-v3-description-smoke-261110/pipeline.json --json
PYTHONPATH=src sure-eval metric run --pipeline /tmp/rir-pr-v3-description-smoke-261110/pipeline.json --samples-jsonl /tmp/rir-pr-v3-description-smoke-261110/samples.jsonl --output-dir /tmp/rir-pr-v3-description-smoke-261110/report --validate-env --json
# run status: ok; report pipeline_id: rir.any.rir_edt_jnd_mae.rir_edt_jnd_mae_v1

The broader local CLI/contracts/environment subset had 165 passed, 11 skipped, and 4 failures tied to unavailable BLEURT, XCOMET, and Paraformer checkpoints in unrelated node-local environments. RIR-focused and exact-route checks passed. The portable full-suite CI run is pending the PR.

For heavyweight nodes: n/a. RIR scoring is in-process; the exact route's environment setup planned no actions and env check returned ok.

Safety

  • No .venv, checkpoints, generated reports, local cache paths, or credentials committed.
  • No private absolute paths in committed config or docs.
  • Score-affecting behavior is declared in route/node identity.
  • Existing default route behavior is tested or explicitly changed: selection is unchanged; the corrected scores and comparability change are stated above and covered by focused tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants