Skip to content

feat/rir-estimation-openbench - #34

Merged
PigeonDan1 merged 31 commits into
X-LANCE:mainfrom
Yj-Xiong:feat/rir-estimation-openbench
Oct 29, 2026
Merged

PigeonDan1 merged 31 commits into
X-LANCE:mainfrom
Yj-Xiong:feat/rir-estimation-openbench

Conversation

@Yj-Xiong

@Yj-Xiong Yj-Xiong commented Oct 23, 2026 •

Copy link
Copy Markdown

Summary

Evaluate predicted room impulse responses against references through a new language-independent RIR task. Scoring consumes WAV/FLAC/NumPy RIR files without running a generation model.

Adds 15 atomic metrics and four curated bundles: NPM, measured RT30/RT60 errors, DRR, C50/C80, EDC, binaural IACC full/early/late, and channel-0-relative ITD/ILD/IPD/DOA. Channel or pair errors are averaged within each sample, then samples receive equal weight.

PR Type

  • New task

Guides: docs/pr_guides/new_task.md, docs/node_readme_template.md, and the Open-Bench Pipeline contribution guide.

Evaluation Identity and Input

  • Task selector: rir; report task: RIR estimation.
  • Canonical metric: the selected atomic metric, or multi for a bundle.
  • Default bundle: rir_npm, rt30_mae, rt60_mae, rir_drr_mae, rir_c50_mae, rir_c80_mae, rir_edc_mae.
  • Bundle top-level score: the first metric (rir_npm by default); component scores remain separate in details.results.
  • Atomic IDs follow rir.any.<metric>.<metric>_v1; discovery and the catalog list exact bundle IDs.
  • Existing task defaults are unchanged.

JSONL rows require sample_id, prediction_rir, and reference_rir. Files must have matching rates/channel counts; NumPy rows also require sample_rate. Relative paths resolve against the manifest. Optional channel_layout validates mono, multichannel, or exact binaural layouts. DOA requires positive finite microphone spacing and sound speed; explicit channel spacings must cover every channel-0 pair.

Scoring Correctness

  • Undefined RT30/RT60/DRR/C50/C80 or reference EDC estimates fail with the affected sample/channel, instead of silently dropping a bad channel and returning a misleading perfect score.
  • Silent/constant-channel correlations, empty IACC windows, and unestimable ITD fail explicitly.
  • A silent prediction can still receive EDC MAE against a valid reference; undefined optional correlation is JSON null and does not remove the channel.
  • Reports retain paths, effective sample rate, layout/count, geometry metadata, original lengths, and paired length in details.rows.

Accepted v1 Protocol Boundaries

The maintainer has reviewed and accepted these documented boundaries for v1:

  1. Pairing truncates to the shorter RIR. This measures only shared duration and does not penalize a missing predicted tail. Length diagnostics now expose this.
  2. IACC, ITD, and TDOA-derived DOA retain a fixed +/-1 ms lag window. Known-delay tests verify in-range numbers for different microphone spacings; general array delays beyond this range remain unsupported. A geometry-based DOA search requires a separate protocol decision.
  3. NPM retains one common lag within +/-256 samples and its existing normalized projection definition.

These definitions were not silently changed while fixing invalid-input handling. Scores from the earlier rir_nmse/rir_corr prototype are not directly comparable.

Documentation and Declarations

Task manifest/routes, scoring manifests/node READMEs, public task guide/index, task-local README, and both homepage RIR entries are included. The branch also includes the KOMOS/documentation changes and atlas freshness checks. Its combined catalog has 196 rows, with RIR and KOMOS present in the atlas.

Scoring runs in-process with NumPy and the audio extra, without node-local environments or checkpoints.

Validation

Python 3.11, dev/audio/canonical/giga extras:

  • Combined portable suite after integrating KOMOS: 595 passed, 54 skipped.
  • Environment metadata: 15 passed, 16 deselected.
  • RIR-focused tests: 54 passed, covering bad prediction/reference channels, undefined correlations, empty late windows, invalid geometry, provenance, truncation diagnostics, known ITD, and exact DOA values at different spacings.
  • All 27 new invalid-input regression cases fail against the original metric implementation and pass with the fix.
  • Ruff, whitespace, deterministic catalog/atlas regeneration, wheel/sdist build, and packaged node assets pass.
uv run --python 3.11 --extra dev --extra audio --extra canonical --extra giga pytest -q --ignore=tests/test_evaluation_env_check.py
uv run --python 3.11 --extra dev --extra audio --extra canonical --extra giga pytest tests/test_evaluation_env_check.py -q -k "in_process or pip_runtime or dry_run or doctor or cache_dir or se_pip_runtime_nodes_are_declared"
uv run --extra dev ruff check .
uv run --extra canonical python scripts/generate_pipeline_catalog.py
python scripts/generate_pipeline_atlas.py
git diff --exit-code docs/pipeline_catalog.jsonl docs/atlas/index.html docs/atlas/pipeline_atlas.svg docs/atlas/pipeline_atlas_dark.svg
git diff --check

Final head: 474bbf0d08d2119ce003214a00011f9f60d2e11c. GitHub Core passed all seven jobs: Python 3.10/3.11/3.12, lint, catalog/atlas freshness, CLI smoke, and package build. The maintainer has completed the protocol review and approved this head for merge within the documented limits.

Final maintainer validation (2026-10-29)

The merge with current main produces exactly the tested head tree. Rechecked: 54 RIR tests, Ruff, catalog/atlas freshness, and whitespace pass; all seven GitHub Core jobs on 474bbf0 pass. The shared-duration comparison and fixed +/-1 ms spatial search are accepted as explicit v1 limits. Broader array-delay support or a different length policy should be introduced through an explicitly versioned protocol.

@PigeonDan1 PigeonDan1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 474bbf0d08d2119ce003214a00011f9f60d2e11c for merge under the documented v1 protocol.

The previously confirmed scoring defects are fixed: unscorable channels and undefined correlations fail explicitly, geometry is validated, and reports retain input/geometry/length diagnostics. The shared-duration truncation policy, separate NPM alignment, and fixed +/-1 ms ITD/DOA window are explicit in the task and node documentation. These are acceptable boundaries for this version; the spatial metrics remain unsuitable for array delays outside that window. A broader geometry-based protocol can be versioned separately.

Validation: the existing combined portable suite passed 595 tests with 54 optional skips, and the final GitHub Core passed all seven jobs including Python 3.10/3.11/3.12. Rechecked today: 54 RIR tests, Ruff, 196-row catalog freshness, atlas freshness, and whitespace pass. The merge result with current main is byte-for-byte the tested PR tree, with no conflicts. Existing task defaults remain unchanged.

The protocol review hold is resolved; approve for merge within these documented limits.

@PigeonDan1
PigeonDan1 merged commit ab5598a into X-LANCE:main Oct 29, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants