Repository navigation
feat/rir-estimation-openbench - #34
Conversation
PigeonDan1
left a comment
There was a problem hiding this comment.
Reviewed head 474bbf0d08d2119ce003214a00011f9f60d2e11c for merge under the documented v1 protocol.
The previously confirmed scoring defects are fixed: unscorable channels and undefined correlations fail explicitly, geometry is validated, and reports retain input/geometry/length diagnostics. The shared-duration truncation policy, separate NPM alignment, and fixed +/-1 ms ITD/DOA window are explicit in the task and node documentation. These are acceptable boundaries for this version; the spatial metrics remain unsuitable for array delays outside that window. A broader geometry-based protocol can be versioned separately.
Validation: the existing combined portable suite passed 595 tests with 54 optional skips, and the final GitHub Core passed all seven jobs including Python 3.10/3.11/3.12. Rechecked today: 54 RIR tests, Ruff, 196-row catalog freshness, atlas freshness, and whitespace pass. The merge result with current main is byte-for-byte the tested PR tree, with no conflicts. Existing task defaults remain unchanged.
The protocol review hold is resolved; approve for merge within these documented limits.
Summary
Evaluate predicted room impulse responses against references through a new language-independent RIR task. Scoring consumes WAV/FLAC/NumPy RIR files without running a generation model.
Adds 15 atomic metrics and four curated bundles: NPM, measured RT30/RT60 errors, DRR, C50/C80, EDC, binaural IACC full/early/late, and channel-0-relative ITD/ILD/IPD/DOA. Channel or pair errors are averaged within each sample, then samples receive equal weight.
PR Type
Guides:
docs/pr_guides/new_task.md,docs/node_readme_template.md, and the Open-Bench Pipeline contribution guide.Evaluation Identity and Input
rir; report task:RIR estimation.metric: the selected atomic metric, ormultifor a bundle.rir_npm,rt30_mae,rt60_mae,rir_drr_mae,rir_c50_mae,rir_c80_mae,rir_edc_mae.rir_npmby default); component scores remain separate indetails.results.rir.any.<metric>.<metric>_v1; discovery and the catalog list exact bundle IDs.JSONL rows require
sample_id,prediction_rir, andreference_rir. Files must have matching rates/channel counts; NumPy rows also requiresample_rate. Relative paths resolve against the manifest. Optionalchannel_layoutvalidates mono, multichannel, or exact binaural layouts. DOA requires positive finite microphone spacing and sound speed; explicit channel spacings must cover every channel-0 pair.Scoring Correctness
nulland does not remove the channel.details.rows.Accepted v1 Protocol Boundaries
The maintainer has reviewed and accepted these documented boundaries for v1:
These definitions were not silently changed while fixing invalid-input handling. Scores from the earlier
rir_nmse/rir_corrprototype are not directly comparable.Documentation and Declarations
Task manifest/routes, scoring manifests/node READMEs, public task guide/index, task-local README, and both homepage RIR entries are included. The branch also includes the KOMOS/documentation changes and atlas freshness checks. Its combined catalog has 196 rows, with RIR and KOMOS present in the atlas.
Scoring runs in-process with NumPy and the audio extra, without node-local environments or checkpoints.
Validation
Python 3.11, dev/audio/canonical/giga extras:
Final head:
474bbf0d08d2119ce003214a00011f9f60d2e11c. GitHub Core passed all seven jobs: Python 3.10/3.11/3.12, lint, catalog/atlas freshness, CLI smoke, and package build. The maintainer has completed the protocol review and approved this head for merge within the documented limits.Final maintainer validation (2026-10-29)
The merge with current main produces exactly the tested head tree. Rechecked: 54 RIR tests, Ruff, catalog/atlas freshness, and whitespace pass; all seven GitHub Core jobs on
474bbf0pass. The shared-duration comparison and fixed +/-1 ms spatial search are accepted as explicit v1 limits. Broader array-delay support or a different length policy should be introduced through an explicitly versioned protocol.