Repository navigation
Conversation
…k support Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
vocab_parallel_linear_logprobs, an opt-in primitive that computes FP32per-token selected log-probabilities and optional entropy directly from hidden
states and resident vocabulary-sharded LM-head parameters.
Refs #8790 and the third request in
#8942 (comment).
positions in the LM head while retaining them as backbone context.
recompute projection tiles in backward without retaining full local logits.
uneven/empty vocabulary shards. Return unreduced scores; the caller owns
causal alignment, masking policy and loss.
Scope and related work
Existing vocab-parallel scoring ops consume materialized logits; fused-linear
CE does not expose selected-token scores and entropy. This complements the
sequence chunking in deepspeedai/DeepSpeedExamples#1023 and the rollout/GRPO
work in #9037; it does not change either architecture or implement a GRPO loss.
Duplicate searches found no existing streamed-scoring PR.
Hidden gradients are reduced inside this primitive exactly once: pass original
decoder hidden states, not the head's TP backward-reduction mapping.
An opt-in
allow_empty_shardsargument extends the existing metadata helper;existing callers keep their previous behavior.
Validation
Hardware: 2 x NVIDIA L20, physical GPUs 0 and 3, driver
580.173.02.Software: Python
3.10.22, PyTorch2.14.1+cu130, Transformers5.17.0,pytest
9.1.1.git diff --checkCUDA coverage includes TP2 FP32/BF16 scores and hidden/head/bias gradients,
uneven/empty shards, all-masked/zero-token batches and collective input errors.
A TP1 ZeRO-0 Engine test matches dense loss, gradients and parameters over
three AdamW updates. Additional FP32 integration tests cover:
shards, and ZeRO-0/2. Over three SGD updates, scores, reconstructed shared
gradients/weights and final-normalization gradients/weights match the dense
reference. The test verifies that AutoTP preserves the shared Parameter.
including an all-masked microbatch on one rank. Full averaged gradients and
parameters match the dense reference over three AdamW updates.
The 10 skips are existing JSD cases whose world-size list includes four ranks;
only the requested two GPUs were visible. Both test commands completed
successfully: 94.71 seconds for the final new suite, 291.60 seconds for regression.
Draft limitations
Only resident linear heads are supported; ZeRO-3 parameters are rejected.
No automatic trainer integration, sequence-sharded hidden states, quantized
heads, higher-order gradients or implicit model-specific/sampling transforms.
The distribution is full-vocabulary softmax after temperature; matching rollout
scores requires a separately agreed probability contract.
Bounded projection tiles do not bound total memory: compaction retains the scored
hidden states, and head gradients accumulate in a full local-shard FP32 buffer.
Eager tile launches, per-token-chunk collectives and input-validation
synchronization are additional costs that need end-to-end measurement.
Not qualified: new scoring with FP16 loss scaling, BF16 ZeRO-2 integration,
ZeRO-1, offload, complete external GRPO/TRL consumers, other accelerators,
or combined TP x DP, PP/SP/EP.
Full-step throughput, peak memory, token capacity and convergence have not
been benchmarked; no improvement in those metrics is claimed. Those
measurements and a real consumer remain follow-ups before ready-for-review.
Potential follow-ups (not implemented in this PR)
These are separate directions for discussion and qualification, not claims of
support or commitments to expand this PR:
integration with Add a multi-turn GRPO example with tool calling, and port OPSD to the per-sample rollout API DeepSpeedExamples#1023, keeping rollout and GRPO
loss in the example. Compare full-step throughput, peak memory and token
capacity against its existing sequence-chunked head and a sequence-chunked
vocab-parallel baseline; validate training numerics before claiming a benefit.
0ac80058as the initial contract reference. Potential additions include logit scaling
and soft-capping in the correct order, TP-global
support_idsnormalization,multiple
score_ids,log_sum_sq_probs,mean_logits,is_top1, and anexplicit FP32 projection option. Match padding, tie-breaking, numerical
stability and differentiable/non-differentiable output semantics, with dense
and TRL-reference output/gradient tests at TP1/TP2.
alignment, loss normalization, MoE auxiliary loss and PEFT/generation behavior
in TRL or its integration layer, rather than in this Core primitive. Backend
selection must be TP-aware: an unsupported path must not silently normalize
over only a local vocabulary shard. Full TRL fused-head compatibility is not
provided by the current primitive.
head materialization/release across forward and recomputed backward.
Separately qualify BF16 ZeRO-2 integration, FP16 loss scaling, offload and
combined parallel layouts; define contracts for non-plain heads before
claiming support.
independent initially; consider consolidating common streamed-projection
machinery when more consumers justify it. Streamed dense KL/JSD would be a
separate primitive, potentially relevant to
Add vocabulary-parallel KL support to OPSD DeepSpeedExamples#1019: selected-token scores and entropy cannot
replace its exact full-vocabulary distillation loss. Profile the actual
bottleneck before prioritizing that extension.
Prepared with GitHub Copilot CLI assistance.