Repository navigation
Fix HybridEngineRollout KV caches under autocast when keys and values differ in dtype - #9124
ArchanaChetan07 wants to merge 1 commit into
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Thanks for the fix and the clear reproduction test! Reviewed against master @ The fix looks correct. Blocker: needs a rebase. The PR currently conflicts with master ( Non-blocking suggestions:
Thanks! |
Under torch.autocast an fp32 Llama-style model produces fp32 keys (the rotary embedding multiplies the bf16 key projection by fp32 cos/sin) but bf16 values. DeepSpeedStaticLayer sizes its buffers from the first keys it sees and then rejected the bf16 values, and HF StaticCache, used for the static graph and continuous prefills, rejects them the same way. Cast incoming key/value states to the cache dtype in DeepSpeedStaticLayer.update, and route both prefills through _create_prefill_cache: models whose layers are all full attention prefill into a lazily sized DeepSpeedStaticCache; sliding-window and linear-attention models keep HF StaticCache, since DeepSpeedStaticCache models every layer as full attention and hybrid GDN slots bind to the HF cache. Fixes deepspeedai#9102 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ArchanaChetan07 <apatil@sandiego.edu>
9031d60 to
39e0fd5
Compare
|
Thanks for the careful review! I rebased onto master ( Rebase. After #8602, hybrid models bind their GDN slots to the HF 1. Continuous-batch staging cache. Good catch: the 2. Static-cache unit test. Added 3. GPU coverage. I can't add labels here. If a maintainer could add the GPU label or trigger the CUDA run, the graph-capture cases would get upstream coverage. Locally, on a T1000 (sm_75, torch 2.6.0+cu124, transformers 5.12.1), the rollout and static-cache suites give 240 passed, 6 failed. The same 6 fail on master on this machine with byte-identical assertion output ( 4. #9077. Noted. In One tradeoff I flagged in the description: the cast can also narrow (e.g. fp16 parameters under bf16 autocast, where the decode cache is allocated in fp16). Master raised there; this rounds to the parameter dtype. Happy to restrict it to widening if you'd prefer. |
Fixes #9102.
Root cause
The issue attributes the failure to caches being created with the parameter dtype. Tracing it on master, the cause is slightly different: under
torch.autocast(dtype=bf16)an fp32 Llama model produces keys and values in different dtypes:k_proj/v_projrun under autocast and return bf16.apply_rotary_pos_embmultiplies the bf16 keys by the fp32cos/sin(the rotary module runs with autocast disabled), so the keys are promoted back to fp32. The values stay bf16.Caches that take their dtype from the first keys (fp32) then reject the bf16 values:
DeepSpeedStaticLayer.update(continuous-batching decode, with or without graph capture):Index put requires the source and destination dtypes match, got Float ... BFloat16.StaticCache.update, used for the prefill of the static graph path (_generate_graph) and of the continuous paths (_continuous_prefill_cache, for models that set_supports_cache_class):index_copy_(): self and source expected to have the same dtype ... Float ... BFloat16. HFStaticCachecannot hold fp32 keys and bf16 values at once, whatever dtype it is allocated in.Allocating the caches in the autocast dtype would not fix this, because the keys really are fp32.
Change (+27 / -3 in
deepspeed/)DeepSpeedStaticLayer.update: cast the incoming key/value states to the cache dtype before writing..to()returns the same tensor when the dtypes already match, so graph capture and the non-autocast paths are unchanged.HybridEngineRollout._create_prefill_cache, used by both prefill sites (_generate_graph,_continuous_prefill_cache):sliding_window, everylayer_typesentryfull_attention) prefill into a lazily sizedDeepSpeedStaticCache, which is sized from the prefill keys so AutoTP-local KV heads still work. With no write position set, it writes the prompt at[0, prompt_len)and gives a single prefill forward the same mask sizes and offsets as a fresh HFStaticCache.StaticCache, exactly as on master.DeepSpeedStaticCachemodels every layer as full attention, so a sliding-window model prefilled into it would silently attend outside the window. Hybrid models also need the HF cache because their GDN slots are bound to its layers (feat(rollout): segment-based kernel injection for hybrid-engine greedy decode #8602).Model families by path (default configs, transformers 5.12 and 5.23):
Behaviour without autocast
To check this doesn't change behaviour anywhere else, I ran the same 30 rollout scenarios on master and on this branch and compared the generated tokens (or the raised error):
_supports_cache_class.All 30 are identical. The sliding-window runs use prompts longer than the window, and they still raise the same error they raise on master.
Tradeoff
The cast can also narrow. When the decode cache is allocated eagerly in the parameter dtype and that dtype is narrower than the incoming states (e.g. fp16 parameters under bf16 autocast), keys and values are now rounded to that dtype where master raised an error. That only happens under autocast and matches the model's own parameter precision; I can restrict the cast to widening and raise otherwise if you prefer.
Relation to other PRs
_create_static_cachecall lines in_generate_graph/_continuous_prefill_cachethere, so any rebase against Reuse the static rollout decode graph across calls #9077 should be trivial.generate()uses HFDynamicCache(concatenation handles mixed dtypes) and already worked.Tests
Every new test fails on master. Reverting either half of the fix makes some of them fail:
test_rollout_under_autocast_matches_hf_generate: 6 cases = {static graph, continuous graph, continuous eager} × {default,_supports_cache_class=True}. Greedy rollout of an fp32 tiny Llama under bf16 autocast, token-for-token vs HFgenerate()under the same autocasttest_static_layer_accepts_values_in_a_different_dtype_than_keys: fp32 keys + bf16 values through both write paths (prefill and per-row decode positions)test_prefill_cache_keeps_hf_static_cache_unless_full_attention_only: routing for full-attention, sliding-window and linear-attention configsSuites:
pytest unit/v1/test_hybrid_engine_rollout_v1.py unit/utils/test_static_cache.py unit/runtime/rolloutDS_ACCELERATOR=cpupytest --forked -n 4 unit/runtime/rollout unit/utils/test_static_cache.py unit/v1/test_hybrid_engine_rollout_v1.pyThe 6 GPU failures fail on master on the same machine with byte-identical assertion output:
test_static_graph_uses_prefill_kv_shape[1,2,4]test_static_graph_matches_native_llama_generation[eager]test_decode_repetition_penalty_matches_hf_generate[0.0-True-None]and[0.7-True-None]They look specific to this setup (sm_75 with bf16 emulation, torch 2.6, transformers 5.12).
pre-commit run --files <changed files>passes.Not validated: bf16-native GPUs (sm_80+), fp16 autocast, hybrid (GDN) models, ZeRO-3 / AutoTP rollout under autocast, and multi-GPU. Without a GPU, upstream CI only runs the continuous-eager cases, the static-cache tests and the routing test; the graph-capture cases need a CUDA run.
AI assistance
AI assistance (Claude Code) was used to reproduce the bug, trace the root cause, draft the change and tests, and run the commands above. I reviewed every changed line and ran the tests myself.
🤖 Generated with Claude Code