Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Start Here## Start Here
Read Contributing. This contribution follows the New Metric PR Guide; the fixes to existing scoring nodes are also checked against the Node Change PR Guide.
Summary
rir_iacc_maeroute. Theiacc,rir_iacc, andrir_iacc_maeselectors resolve torir_iacc_full_mae; full/early/late IACC remain separate metrics and require exactly two channels. Other spatial metrics accept two or more channels and average eachchNversusch0pair within a sample.The RIR evaluation chain has not been released to online users. This PR retains
the existing
v1node and pipeline identities as the first release protocol.Internal results produced with the earlier implementation must be recomputed
before comparison with results from this branch.
The RIR row contract is unchanged:
prediction_rirandreference_rirare required, rows align bysample_id, NumPy inputs also requiresample_rate, and DOA additionally requires microphone-spacing metadata. Acoustic and ISO parameter metrics accept one or more channels. An undefined channel result fails the strict route rather than being silently excluded.Metric definitions and aggregation
For the ISO parameter routes, each predicted/reference RIR has one broadband direct onset (the first sample within 20 dB of its peak). A zero-phase, sixth-order-prototype Butterworth octave-band filter is applied at 500 Hz and 1 kHz. Band parameter values are averaged arithmetically per channel before the predicted/reference absolute error is taken. Channel errors are averaged per sample; sample scores receive equal weight.
−60/slope0.05 × reference EDT10 log10(E[0,80 ms) / E[80 ms,end))E[0,50 ms) / E[0,end)1000 × Σ(t·h²) / Σ(h²)from onsetThe reported
*_jnd_maescore is the mean ofabs(prediction − reference) / JND divisor, inx_jnd; lower is better.within_one_jnd_percentcounts samples for which every channel is within that metric's threshold, divided by the number of successfully scored samples. The route fails if any channel is undefined, so a successful report hasvalid_samples == total_samples. These channel and pass-rate rules are benchmark conventions. The filter, background-noise treatment, and calibration are not claimed as a complete ISO 3382 auditorium measurement.PR Type
Select one guide:
Guide followed: New Metric; Node Change requirements are relevant to the corrections of existing scoring nodes.
Evaluation Identity
metric:rir_edt_m_mae,rir_edt_broadband_mae,rir_edt_jnd_mae,rir_c80_jnd_mae,rir_d50_jnd_mae,rir_ts_jnd_mae. The JND bundle reportsmetric=multiwith four separate results.execution_metrics, if different: atomic routes use the corresponding canonical metric. The JND bundle executesrir_edt_jnd_mae,rir_c80_jnd_mae,rir_d50_jnd_mae, andrir_ts_jnd_mae. Short names such asedt_jndare selectors only.pipeline_idvalues:pipeline_id:pipeline_id:rir.any.rir_iacc_mae.rir_iacc_mae_v1. Its legacy selectors targetrir.any.rir_iacc_full_mae.rir_iacc_full_mae_v1.Changed Declarations
rir_iacc_maebecomes a compatibility selector forrir_iacc_full_mae.node_env.yaml: n/a; all affected RIR scoring nodes runin_processwithout checkpoints or binaries. SciPy is declared in the package'saudioextra.docs/node_readme_template.md: yes for the six new nodes (all required sections); existing affected node descriptions retainv1for the pre-release protocol decision noted above.rirwas already registered); one four-metric JND bundle is added.docs/contributing.md:README.md,README_ZH.md,docs/tasks/rir.md, the task-local README, affected node READMEs, catalog, and atlas are updated. The task index already links to RIR and needs no change.Validation
Executed on the v3 branch (
7dd2b8a):A small synthetic two-channel RIR fixture with 4096 samples at 16 kHz, a direct impulse, and an exponential tail was generated as follows:
The following exact route chain completed successfully and preserved the same
pipeline_idinreport.jsonandpipeline_description.json:PYTHONPATH=src sure-eval metric routes rir --metric rir_edt_jnd_mae --json PYTHONPATH=src sure-eval metric describe rir --pipeline-id rir.any.rir_edt_jnd_mae.rir_edt_jnd_mae_v1 --output /tmp/rir-pr-v3-description-smoke-261110/pipeline.json --json PYTHONPATH=src sure-eval env setup --pipeline /tmp/rir-pr-v3-description-smoke-261110/pipeline.json --dry-run --json PYTHONPATH=src sure-eval env check --pipeline /tmp/rir-pr-v3-description-smoke-261110/pipeline.json --json PYTHONPATH=src sure-eval metric run --pipeline /tmp/rir-pr-v3-description-smoke-261110/pipeline.json --samples-jsonl /tmp/rir-pr-v3-description-smoke-261110/samples.jsonl --output-dir /tmp/rir-pr-v3-description-smoke-261110/report --validate-env --json # run status: ok; report pipeline_id: rir.any.rir_edt_jnd_mae.rir_edt_jnd_mae_v1The broader local CLI/contracts/environment subset had 165 passed, 11 skipped, and 4 failures tied to unavailable BLEURT, XCOMET, and Paraformer checkpoints in unrelated node-local environments. RIR-focused and exact-route checks passed. The portable full-suite CI run is pending the PR.
For heavyweight nodes: n/a. RIR scoring is in-process; the exact route's environment setup planned no actions and
env checkreturnedok.Safety
.venv, checkpoints, generated reports, local cache paths, or credentials committed.