Skip to content

Feat/etc canonicalization - #40

Merged
Jason040114 merged 8 commits into
X-LANCE:mainfrom
Jason040114:feat/etc-canonicalization
Nov 9, 2026
Merged

Jason040114 merged 8 commits into
X-LANCE:mainfrom
Jason040114:feat/etc-canonicalization

Conversation

@Jason040114

Copy link
Copy Markdown
Collaborator

Summary

Add normalization/etc_canonicalization, an optional ASR normalization node
based on the evaluation-side transcript canonicalization protocol from
ETC-Bench.

ETC-Bench maps speech-equivalent written transcript variants to a shared
canonical form before reference and hypothesis comparison. Reference and
hypothesis are processed independently by the same canonicalizer.

PR Type

  • New task
  • New metric
  • New pipeline route
  • Node/tool/version change
  • Maintenance

Guide followed:

  • docs/pr_guides/new_route.md
  • docs/pr_guides/node_change.md

Evaluation Identity

  • Reported metric: existing cer or wer
  • execution_metrics, if different: n/a
  • Added pipeline IDs:
    • asr.ar.cer.etc_canonicalization_ar_v1.wenet_cer_v1
    • asr.ja.cer.etc_canonicalization_ja_v1.wenet_cer_v1
    • asr.zh.cer.etc_canonicalization_zh_v1.wenet_cer_v1
    • asr.bn.wer.etc_canonicalization_bn_v1.wenet_wer_v1
    • asr.de.wer.etc_canonicalization_de_v1.wenet_wer_v1
    • asr.en.wer.etc_canonicalization_en_v1.wenet_wer_v1
    • asr.fa.wer.etc_canonicalization_fa_v1.wenet_wer_v1
    • asr.fr.wer.etc_canonicalization_fr_v1.wenet_wer_v1
    • asr.hi.wer.etc_canonicalization_hi_v1.wenet_wer_v1
    • asr.id.wer.etc_canonicalization_id_v1.wenet_wer_v1
    • asr.ru.wer.etc_canonicalization_ru_v1.wenet_wer_v1
    • asr.vi.wer.etc_canonicalization_vi_v1.wenet_wer_v1
  • Default route changed: no
  • Score comparability changed: yes, only when an ETC route is explicitly selected

Changed Declarations

  • Task manifest: no
  • Task routes: yes
  • Node manifest: yes
  • node_env.yaml: yes
  • Node README follows docs/node_readme_template.md: yes
  • Catalog regenerated: yes
  • Catalog discovery task or curated bundle changed: no
  • Documentation coverage: ASR task documentation, node README, generated
    catalog, and generated atlas updated.

Validation

PYTHONPATH=src python -m pytest \
  tests/test_etc_canonicalization_node.py \
  tests/test_asr_pipeline_nodes.py \
  tests/test_asr_canonical_route.py \
  tests/test_pipeline_catalog_identity.py \
  tests/test_evaluation_scripts_contracts.py -q

python -m ruff check \
  src/sure_eval/evaluation/nodes/normalization/etc_canonicalization \
  tests/test_etc_canonicalization_node.py

git diff --check

Results:

- 81 passed, 11 skipped
- Ruff passed
- Catalog and atlas regenerated successfully

Exact route verification:

sure-eval metric describe asr \
  --pipeline-id asr.en.wer.etc_canonicalization_en_v1.wenet_wer_v1 \
  --output /tmp/asr_etc.json

sure-eval env check --pipeline /tmp/asr_etc.json --json

## Real Model Smoke Test

Using a user-prepared ETC-Qwen3.5-0.8B checkpoint:

The recipe calls for two eggs.
The recipe calls for 2 eggs.

Both outputs became:

The recipe calls for 2 eggs.

A nine-row multi-phenomenon SURE pipeline smoke test covered numeric,
currency, decimal, biological symbol, mathematical symbol, URL,
abbreviation, British/American spelling, and diacritic variants.

pipeline_id:
asr.en.wer.etc_canonicalization_en_v1.wenet_wer_v1

rows:
ref=9, hyp=9

score:
WER=0.0

## Usage

After cloning:

sure-eval env setup --node normalization/etc_canonicalization
export SURE_ETC_CANONICALIZATION_MODEL=/path/to/etc-qwen3.5-0.8b

sure-eval metric describe asr \
  --pipeline-id asr.en.wer.etc_canonicalization_en_v1.wenet_wer_v1 \
  --output /tmp/asr_etc.json

sure-eval metric run --pipeline /tmp/asr_etc.json \
  --ref-file ref.txt \
  --hyp-file hyp.txt \
  --output-dir /tmp/asr-etc-run

The model checkpoint, node-local environment, caches, and generated reports are
user-managed runtime assets.

Copilot AI balanced review requested due to automatic review settings November 2, 2026 06:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@PigeonDan1

Copy link
Copy Markdown
Collaborator

Reviewed head 94fe662ac9bfa8d9276fd63e29f296823427832d. The optional normalization-node design is appropriate, and existing default ASR routes remain unchanged. I checked the pinned ETC-Bench source: the system prompt matches upstream. Core is green (7/7); local Python 3.11 validation passed 611 portable tests with 56 optional skips, plus 81 focused tests with 11 skips.

Please address the following before merge. These findings concern the adapter's evaluation contract, beyond the happy-path tests.

1. Required: multiline model output can produce an incorrect zero WER

canonicalize_batch() validates the output count and string types, but .strip() leaves internal newlines intact. _normalize_file() then writes the model output directly into key<TAB>text files. The scorer interprets subsequent physical lines as additional utterances.

I reproduced this through the real scripts.asr.run -> report.json path with an injected canonicalizer:

Input reference:   utt1<TAB>a b
Input hypothesis:  utt1<TAB>a c

Single-line outputs: ref="a b",  hyp="a c"  -> WER=0.5
Multiline outputs:   ref="a\nb", hyp="a\nc" -> WER=0.0, reported as success

In the second case the normalizer reports one row per side, but the scorer sees two utterances per side and only one reference token. No content words were removed or substituted; changing internal whitespace alone corrupted the sample boundaries and score. The real subprocess adapter also accepts a JSON prediction containing a\nb.

Please guarantee that each canonicalized text remains within its original sample row. Define how internal CR/LF/tab characters are handled: either normalize them to safe intra-row whitespace or reject invalid output with an explicit sample error. Do not drop the remainder of the text. Preserve original keys, row counts, and a trace of any serialization normalization. Upstream JSONL safely contains multiline strings; this extra protection is needed for the key-text adaptation.

Add regression tests for the example above, CRLF, internal tabs, and empty output. A malformed model output must not silently yield a successful zero error rate. Verify both score and utterance/token counts.

2. Required: handle generation budget exhaustion explicitly

model_runtime.py fixes max_new_tokens=256, then decodes and returns without checking whether generation ended normally. Using the actual runtime function with a fake model returning 256 tokens and no EOS, I confirmed it is accepted as normal text with no truncation indication. This is a contract probe, not a claim that I reproduced long-text failure with real weights.

The default budget matches upstream, but general ASR inputs can exceed the benchmark's transcript lengths. Please define and test a long-input policy: detect an incomplete/budget-limited generation and fail explicitly, or implement a documented configurable-budget/retry approach. A flag that still allows a partial transcript to be scored as complete would not resolve the issue. Record effective generation settings and document the supported limit. This does not require another pipeline stage or unbounded generation.

3. Input integrity: do not silently discard nonblank malformed rows

_normalize_file() currently ignores every line without a tab, and the new test explicitly expects a malformed row to disappear. With one valid matching row plus a malformed second row on each side, the run returns WER=0.0 and reports only the valid row; the trace does not disclose the dropped input.

Please reject nonblank malformed rows with the filename/line number, or explicitly expose the dropped-row count and policy. Some existing adapters have similar behavior, so this is not a claim of a new regression in all default ASR routes. Blank lines can be ignored; a valid key<TAB> empty hypothesis must remain an evaluation sample.

4. Small documentation corrections

  • In docs/tasks/asr.md, the ETC heading was inserted immediately after ## Canonical ITN Pipelines, so the existing Canonical ITN body now appears under ETC. Keep those as separate complete sections.
  • Add an ETC entry/link to the task-local ASR README, following the existing documentation-coverage guidance. No new homepage task row is needed.
  • Document the generation limit and the possibility of content changes by a learned canonicalizer. Optionally improve trace reproducibility with model revision/hash, prompt hash, and effective decoding parameters.

CI and next review

Please add focused regression coverage to the existing portable suite: model-output serialization, key/row preservation, normal EOS versus token-budget exhaustion, and malformed/empty input handling. These can use mocked subprocesses/models without GPU or checkpoint downloads. Keep the existing three-version Core, catalog/atlas freshness, CLI, and package checks; no extra workflow is needed just to add these cases.

After updating the head, rerun Core and provide the corrected examples/results for re-review. The present green CI and the nine-row real-model smoke are useful, but do not cover these failure cases. I have not modified this PR's implementation or merged it.

@PigeonDan1

Copy link
Copy Markdown
Collaborator

Re-reviewed head 016856477bc216ac2e1c9f9b89cf8a27421544ce. The main fixes are effective:

  • The original multiline counterexample now returns WER 0.5, with one utterance per side and two reference tokens. CRLF/tab output also stays within its original row.
  • Calling the actual runtime with a controlled fake model confirms normal EOS and EOS at the budget succeed, while 256 generated tokens without EOS fail explicitly.
  • Nonblank rows without a tab now raise an error with the file and line number.
  • The public ASR guide now separates ETC from Canonical ITN and documents the generation limit.

The new regressions detect the original defects. Core on this head is 7/7 green; local validation passed 617 portable tests / 56 skipped, 87 focused tests / 11 skipped, and the environment metadata subset. Catalog/atlas freshness, Ruff, wheel/sdist, and packaged node assets also pass.

Two items remain before merge:

1. Required: preserve empty and whitespace-only hypotheses independently of model output

_normalize_file() still sends every transcript, including an empty string, to the model. The new test_etc_preserves_valid_empty_input_row makes its fake canonicalizer return empty strings, so it only verifies the cooperative-output case.

Using the actual subprocess adapter and ASR run/report path with controlled subprocess responses:

Reference:  utt1<TAB>hello
Hypothesis: utt1<TAB>

Model returns "" for empty input:      WER=1.0, one deletion
Model returns "hello" for empty input: WER=0.0, successful report

The latter report records original_text="" and normalized_text="hello". This demonstrates an adapter constraint gap; I am not claiming the real checkpoint was observed producing this particular word. An empty recognition result has no transcript content to canonicalize and should remain empty regardless of generated output.

There is also a regression for a whitespace-only hypothesis: with original_text=" " and an empty normalized result, the new if not normalized_text and original_text condition raises canonicalizer returned empty output for nonempty input. This should instead remain an empty hypothesis and contribute deletion errors.

Please handle transcripts with no non-whitespace content deterministically: preserve the original key and row, keep the effective text empty, and preferably exclude these items from model inference. When processing mixed empty/nonempty batches, restore the original order. Preserve the explicit failure policy for genuinely nonempty input that produces empty output.

Required regression coverage in the existing portable suite:

  1. Nonempty reference + empty hypothesis yields WER=1.0, one utterance per side, one reference token and one deletion, even if a fake model would generate nonempty text for an empty input.
  2. The same holds for whitespace-only hypotheses; they are not rejected as nonempty input.
  3. Mixed empty/nonempty batches preserve keys, order and row counts, with only nonempty transcripts passed to inference if using that approach.

These tests should exercise the actual adapter boundary rather than assuming an empty model response. The existing multiline, truncation and malformed-row regressions should continue passing.

2. Required small documentation fix: broken task README link

In src/sure_eval/evaluation/tasks/asr/README.md, the new ETC link uses:

../../src/sure_eval/evaluation/nodes/normalization/etc_canonicalization/README.md

That resolves to a nonexistent location from this README. The correct relative target is:

../../nodes/normalization/etc_canonicalization/README.md

Please check the changed relative links when updating the docs.

Validation and next review

Please update the PR's validation results to the new head and close the currently unclosed code block in the PR description. Add the empty-input regressions to the existing Core test suite and rerun Core on the updated commit; an additional workflow or GPU job is unnecessary for these cases.

The earlier requested multiline, generation-budget and malformed-row fixes are accepted. Raw model-text retention and richer model provenance remain optional improvements. I will keep the PR unmerged until the remaining input invariant and documentation link are corrected and the new head passes review/CI. This feedback does not modify your implementation.

@Jason040114
Jason040114 merged commit abfdd68 into X-LANCE:main Nov 9, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants