Repository navigation
Feat/etc canonicalization - #40
Conversation
|
Reviewed head Please address the following before merge. These findings concern the adapter's evaluation contract, beyond the happy-path tests. 1. Required: multiline model output can produce an incorrect zero WER
I reproduced this through the real In the second case the normalizer reports one row per side, but the scorer sees two utterances per side and only one reference token. No content words were removed or substituted; changing internal whitespace alone corrupted the sample boundaries and score. The real subprocess adapter also accepts a JSON prediction containing Please guarantee that each canonicalized text remains within its original sample row. Define how internal CR/LF/tab characters are handled: either normalize them to safe intra-row whitespace or reject invalid output with an explicit sample error. Do not drop the remainder of the text. Preserve original keys, row counts, and a trace of any serialization normalization. Upstream JSONL safely contains multiline strings; this extra protection is needed for the key-text adaptation. Add regression tests for the example above, CRLF, internal tabs, and empty output. A malformed model output must not silently yield a successful zero error rate. Verify both score and utterance/token counts. 2. Required: handle generation budget exhaustion explicitly
The default budget matches upstream, but general ASR inputs can exceed the benchmark's transcript lengths. Please define and test a long-input policy: detect an incomplete/budget-limited generation and fail explicitly, or implement a documented configurable-budget/retry approach. A flag that still allows a partial transcript to be scored as complete would not resolve the issue. Record effective generation settings and document the supported limit. This does not require another pipeline stage or unbounded generation. 3. Input integrity: do not silently discard nonblank malformed rows
Please reject nonblank malformed rows with the filename/line number, or explicitly expose the dropped-row count and policy. Some existing adapters have similar behavior, so this is not a claim of a new regression in all default ASR routes. Blank lines can be ignored; a valid 4. Small documentation corrections
CI and next reviewPlease add focused regression coverage to the existing portable suite: model-output serialization, key/row preservation, normal EOS versus token-budget exhaustion, and malformed/empty input handling. These can use mocked subprocesses/models without GPU or checkpoint downloads. Keep the existing three-version Core, catalog/atlas freshness, CLI, and package checks; no extra workflow is needed just to add these cases. After updating the head, rerun Core and provide the corrected examples/results for re-review. The present green CI and the nine-row real-model smoke are useful, but do not cover these failure cases. I have not modified this PR's implementation or merged it. |
|
Re-reviewed head
The new regressions detect the original defects. Core on this head is 7/7 green; local validation passed 617 portable tests / 56 skipped, 87 focused tests / 11 skipped, and the environment metadata subset. Catalog/atlas freshness, Ruff, wheel/sdist, and packaged node assets also pass. Two items remain before merge: 1. Required: preserve empty and whitespace-only hypotheses independently of model output
Using the actual subprocess adapter and ASR run/report path with controlled subprocess responses: The latter report records There is also a regression for a whitespace-only hypothesis: with Please handle transcripts with no non-whitespace content deterministically: preserve the original key and row, keep the effective text empty, and preferably exclude these items from model inference. When processing mixed empty/nonempty batches, restore the original order. Preserve the explicit failure policy for genuinely nonempty input that produces empty output. Required regression coverage in the existing portable suite:
These tests should exercise the actual adapter boundary rather than assuming an empty model response. The existing multiline, truncation and malformed-row regressions should continue passing. 2. Required small documentation fix: broken task README linkIn That resolves to a nonexistent location from this README. The correct relative target is: Please check the changed relative links when updating the docs. Validation and next reviewPlease update the PR's validation results to the new head and close the currently unclosed code block in the PR description. Add the empty-input regressions to the existing Core test suite and rerun Core on the updated commit; an additional workflow or GPU job is unnecessary for these cases. The earlier requested multiline, generation-budget and malformed-row fixes are accepted. Raw model-text retention and richer model provenance remain optional improvements. I will keep the PR unmerged until the remaining input invariant and documentation link are corrected and the new head passes review/CI. This feedback does not modify your implementation. |
Summary
Add
normalization/etc_canonicalization, an optional ASR normalization nodebased on the evaluation-side transcript canonicalization protocol from
ETC-Bench.
ETC-Bench maps speech-equivalent written transcript variants to a shared
canonical form before reference and hypothesis comparison. Reference and
hypothesis are processed independently by the same canonicalizer.
PR Type
Guide followed:
docs/pr_guides/new_route.mddocs/pr_guides/node_change.mdEvaluation Identity
metric: existingcerorwerexecution_metrics, if different: n/aasr.ar.cer.etc_canonicalization_ar_v1.wenet_cer_v1asr.ja.cer.etc_canonicalization_ja_v1.wenet_cer_v1asr.zh.cer.etc_canonicalization_zh_v1.wenet_cer_v1asr.bn.wer.etc_canonicalization_bn_v1.wenet_wer_v1asr.de.wer.etc_canonicalization_de_v1.wenet_wer_v1asr.en.wer.etc_canonicalization_en_v1.wenet_wer_v1asr.fa.wer.etc_canonicalization_fa_v1.wenet_wer_v1asr.fr.wer.etc_canonicalization_fr_v1.wenet_wer_v1asr.hi.wer.etc_canonicalization_hi_v1.wenet_wer_v1asr.id.wer.etc_canonicalization_id_v1.wenet_wer_v1asr.ru.wer.etc_canonicalization_ru_v1.wenet_wer_v1asr.vi.wer.etc_canonicalization_vi_v1.wenet_wer_v1Changed Declarations
node_env.yaml: yesdocs/node_readme_template.md: yescatalog, and generated atlas updated.
Validation