Skip to content

Validate Whisper timestamp token bounds - #33824

Merged
Akshay Sonawane (apsonawane) merged 3 commits into
mainfrom
fix/whisper-token-bounds
Nov 3, 2026
Merged

Akshay Sonawane (apsonawane) merged 3 commits into
mainfrom
fix/whisper-token-bounds

Conversation

@apsonawane

Copy link
Copy Markdown
Contributor

This pull request strengthens input validation for Whisper model token IDs and improves test coverage for these checks. The main focus is on ensuring that special token IDs (such as EOS and timestamp tokens) are within valid vocabulary bounds, preventing out-of-range errors during generation. Additionally, new unit tests are added to verify these validations.

Validation Improvements:

  • Added explicit checks in BeamSearchParameters::ValidateWhisperTimestampTokenId and TimestampLogitsProcessor::Process to ensure that eos_token_id, end_of_text_token_id, and timestamp token IDs are within the valid vocabulary range, throwing informative errors if not. [1] [2]

Test Enhancements:

  • Introduced new unit tests in beam_search_test.cc to verify that invalid token IDs are correctly rejected and valid boundary values are accepted, including tests for out-of-range EOS and timestamp token IDs.
  • Added a TestSequences helper class and MakeTimestampProcessor factory function to facilitate testing of the logits processor.
  • Included necessary headers for new tests and processor usage. [1] [2]

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Beam-search tests cannot compile because non-const span references bind to temporaries, and four timestamp-validation tests leave eos_token_id uninitialized, making validation unreliable.

2 open issues
What changed in this PR

This PR strengthens vocabulary-bound checks in ONNX Runtime’s Whisper generation code to reject invalid EOS and timestamp token IDs before unsafe score access.

Changes:

  • Adds EOS validation to Whisper beam-search parameters.
  • Checks end-of-text and sequence timestamp IDs during logits processing.
  • Adds tests for invalid IDs, valid boundary values, and unchanged scores on rejection.
File Description
onnxruntime/​test/​contrib_ops/​beam_search_test.cc Adds test helpers and token-bound validation cases.
onnxruntime/​contrib_ops/​cpu/​transformers/​logits_processor.h Validates end-of-text and sequence timestamp IDs before processing each beam.
onnxruntime/​contrib_ops/​cpu/​transformers/​beam_search_parameters.cc Enforces vocabulary bounds for Whisper EOS IDs.

Review effort: Balanced

@titaiwangms Ti-Tai Wang (titaiwangms) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found in a bounded static review of the diff, Whisper parameter validation, timestamp logits processing, and added tests. The added EOS and sequence-token bounds checks precede the corresponding score writes; the tests include out-of-range values and a valid vocabulary boundary.

Validation scope: static inspection only; I did not build or run the tests or sanitizers.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

EOS validation tests omit negative IDs in both test paths and the accepted zero boundary in the parameter test, leaving lower-bound regressions undetected.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Low severity Test negative and zero EOS IDs

onnxruntime/​test/​contrib_ops/​beam_search_test.cc:163

Both new EOS guards reject negative IDs, but the parameter and processor tests only exercise upper-bound failures, so removing either >= 0 check would go unnoticed. Add -1 to both invalid-EOS loops (including line 163) and cover 0 as an accepted lower boundary in the parameter test.

2 resolved since last review

Review effort: Max

@apsonawane
Akshay Sonawane (apsonawane) merged commit 585d6a6 into main Nov 3, 2026
96 of 97 checks passed
@apsonawane
Akshay Sonawane (apsonawane) deleted the fix/whisper-token-bounds branch November 3, 2026 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants