Repository navigation
Conversation
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
jinyouzhi
marked this pull request as ready for review
November 20, 2026 06:01
jinyouzhi
requested review from
loadams,
tjruwase and
tohtana
as code owners
November 20, 2026 06:01
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
delock
self-requested a review
November 20, 2026 06:28
delock
approved these changes
Nov 20, 2026
delock
enabled auto-merge
November 20, 2026 10:04
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #9126.
Root cause
The segment-KI rollout change eagerly initialized
DeepSpeedStaticCachefrom global configuration head counts. That allocation can disagree with AutoTP's local KV heads, causing either a copy shape failure or silent wrong-token generation. The same change also passed a boolean 4-D mask directly to attention implementations that expect additive masks, while graph/Python token selection bypassed the repetition penalty. Together these account for the nightly static-graph failures intests/unit/v1/test_hybrid_engine_rollout_v1.py.The batched graph step did not need the full-step argmax kernel because Python already selects each row's token. The kernel is now explicitly limited to batch size one; the b=1 EOS check remains amortized over 16 replays because at most 15 trailing steps are padded away, avoiding a synchronization on every replay. Batched decoding continues checking after every step because its dead work is multiplied by the number of active rows.
Duplicate-work check
#9077 is primarily a static graph-reuse performance PR, not the dedicated #9126 fix, but its rebase onto #8602 independently absorbed the same local-KV, additive-mask, and repetition-penalty corrections while preserving that performance work. #9127 remains the isolated hotfix for the nightly regression. If #9127 is merged first, #9077 should rebase onto it and drop the duplicated correctness hunks; its remaining scope is graph reuse and lifecycle. #9079 is stacked on #9077 and separately handles whole-step graph capture, including replacement of the Python token-history loop.
Validation
Hardware: 1x Intel Arc Pro B70 (16 devices visible), Level Zero XPU; driver version is not exposed by the environment. Software: torch
2.14.0.dev20260804+xpu, Transformers5.17.0.The broad two-file run produced 34 passes and 17 skips before the added direct regressions; its remaining failures were unsupported Level Zero SDPA/distributed operations (
UR_RESULT_ERROR_UNSUPPORTED_FEATURE), not assertion regressions in the changed path.The new direct regressions assert that visible positions are additive zero and masked positions are
finfo.min, and that the full-step kernel is not mounted for repetition penalties or batched decode. Existing end-to-end tests compare generated tokens against Hugging Face and cover the penalty result on CUDA CI.Compatibility coverage
The per-step
torch.catused by repetition penalty remains O(n²), matching the existing continuous rollout implementation. Optimizing only this static path is small (roughly 30–60 minutes plus a focused test), but unifying it with variable-length continuous rollout would take roughly 2–4 hours and require broader compaction/padding/CUDA validation. It is intentionally deferred from this correctness fix and can be handled as a small follow-up performance PR, primarily benefitingrepetition_penalty != 1; the default penalty path is unaffected.AI assistance was used for investigation, implementation, and test preparation; the submitting human reviewed the final diff and is responsible for the DCO sign-off.