Repository navigation
Conversation
…tion group A ZeRO-3 engine whose data-parallel group is a subset of the world (two trainer layouts living in the same processes, or a trainer next to inference-only ranks) hung or failed its trace checks: the checks broadcast from global rank 0 over the world and the barriers waited for every rank. They now run over the parameters' partition group; with the default group nothing changes. Signed-off-by: RunguoLi <li19107254665@gmail.com>
…age relay, pipelined execution, ZeRO-3 training-state move relayout.py: RelayoutPlan moves weights between two sharded layouts in GPU memory. Both sides are declared per rank as affine pieces; the source and destination ranks may be disjoint, equal or overlapping, and a weight may have several holders (the trainer's ZeRO-3 partitions, surviving shards of the previous layout, replicas). - Planning runs identically on every rank. Holders are preferred in the order: the destination rank itself, its node, the caller's layout order, the least loaded. - topology="relay": a block that crosses nodes is fetched once per node and copied to the node's other ranks in a second stage; spread=True has the replicas of a node fetch bands of each large block and exchange them. - Execution: one all_to_all_single per bucket of weights and stage; over buckets the stages are pipelined (stage one of a bucket travels with stage two of the bucket before). affine_plan.py: the same moves planned through the affine IR's plan_transfer (planner="affine"), a relay being two plans through a relay map whose shards are bands of the destinations themselves. Weights it does not express yet (holders in two layouts, partial targets) fall back to the rectangle planner per weight. zero3_state.py: Zero3StateMove moves a ZeRO-3 engine's fp32 master partitions, optimizer moments and step counters to another ZeRO-3 engine of the same model over other ranks, from the optimizer storage maps; release_training_state / restore_training_state give back and re-allocate the inactive engine's memory. Tests come with the next commit (they use the native rollout's layouts). Signed-off-by: RunguoLi <li19107254665@gmail.com>
…a; tests of the relayout and the state move native/: a small tensor-parallel decoder whose weights are views of one buffer per rank, allocated when the layout is created and written in place, so CUDA graphs captured over the first version of the weights stay valid for later ones. NativeRollout syncs it from a ZeRO-3 trainer and changes its ranks or tensor-parallel degree through RelayoutPlan; a replica whose ranks and degree are unchanged keeps its buffer, KV cache and graphs. build_rollout takes engine="native"; RolloutBatch gains an optional response_logprobs. Tests: test_relayout.py (both planners: ZeRO-3 to tensor parallel, degree changes, a lost rank refilled, two nodes flat / relay / spread, pipelined buckets, the planners' inter-node bytes compared, fallbacks, contradictory maps), test_native_decoder.py, test_native_rollout.py (sync, re-placement, the trainer switch with Zero3StateMove). Signed-off-by: RunguoLi <li19107254665@gmail.com>
|
Thanks @LiRunGuo, this is great to see running end to end. A DP4 → DP2 → DP4 state move in 0.6–0.8 s with every move digest-checked is the comparison the RFC needed against the checkpoint path: the same Qwen3-0.6B ZeRO-3 changes through a universal checkpoint took me 46 s (DP4 → DP2) and 51 s (DP2 → DP4) on 4× A100, most of it the relaunch. To your two questions, as the RFC's author: The split: +1.
The relay-map formulation: yes. Bands of the destinations as relay shards is the alternative already recorded under open question 1 in #9026, from your earlier comment: no partial targets and no relay buffer. On the The choice. I'll take what is a flag here: whether a weight goes direct, through a relay, or through a relay with spread, and the thresholds ( Before and after. I can run your state move on the same 4× A100 node as my checkpoint-path numbers, with the same model and changes, so there is one comparison on one machine. I'll update #9026's body with the relay-map formulation and the split once @0z5a has weighed in. |
|
@LiRunGuo, thanks for the split at the end of the description. A note on step 2 (the training-state move rewritten on The description says no open PR covers a live move of the optimizer state. deepspeedai/DeepSpeedExamples#1021 (open since 10-22) already does this on the IR, beyond ZeRO-3:
Suggestion. Since step 2 is a rewrite anyway, could it target this wider matrix rather than ZeRO-3 only? The IR already describes every one of these layouts. What differs per case is:
One gap the TP cases expose. A second engine with another TP size overwrites the process-global groups in This comment was drafted with AI assistance and checked by me against #9074's description and the #1021 comments. |
|
Thanks @Achyuthan-S. One machine, both paths. Thanks for the offer. The benchmark script is not in the PR and I'm not publishing it yet, so I can't hand you the run for now. I have the same run queued on a 4× A100 node and will post it here when it finishes; that is another machine than yours, so it puts the same GPU model next to your checkpoint numbers, not one machine. @jinyouzhi's run on #9026 already gives one pair on one machine, measured independently (4× L20, ZeRO-1 DP4 → ZeRO-3 DP4: 0.62 s in the running processes against 27.4 s through a universal checkpoint). The difference to this PR is the check: sums there, a bitwise digest here. Per-weight planning. Right:
The choice. Good, it is yours. Data for it, on 2 nodes × 4 H100 with Qwen3-8B (a ZeRO-3 trainer on node 0, replicas on node 1): I ran 30 configurations with both planners: three syncs from the trainer (into TP4, TP2 × 2 and TP1 × 4) and seven transitions (four layout changes, read from the old replica or from both sources), each direct, relayed, and relayed with spread.
Both planners run on this PR's executor, not on #8809. I can post the per-configuration table (bytes inside and across nodes, times) on #9026 as points for §3. The two-layout fallback. In those runs 217 of 399 weights planned from both sources fell back to the rectangle planner. I'll rerun that case on #9052, and on #9070 for the column case, and report how many each removes. #9076 now carries its regression test and is marked ready for review. |
NativeRollout is the destination the relayout and state-move tests write into, not a rollout engine proposed for users. build_rollout is as on master again; the tests construct NativeRollout directly. Signed-off-by: RunguoLi <li19107254665@gmail.com>
|
The 4× A100 run finished: A100 80GB PCIe, one node, the same DP4 → DP2 → DP4 move inside the same four processes, six rounds per planner, a training step after every move. The node's other four GPUs were running other jobs at the time, so the spread is wide.
Medians of 6, with min and max.
|
|
Thanks @LiRunGuo, the A100 run is the comparison I was after, and 48 of 48 bitwise with the flipped bit caught for both models is reassuring. To read it next to mine: of my 46 s (DP4 → DP2) and 51 s (DP2 → DP4), save + convert + load is about 15 s and 18 s. The rest is the relaunch, about 32 s and 33 s, and 23 s of that is building the model: on that cluster, importing transformers from a venv on the NFS home directory takes 21–25 s by itself. As @jinyouzhi's 10 s relaunch on #9026 shows, that part depends on the environment, so the part that carries over between machines is the 15–18 s through the checkpoint. The policy's inputs. Right, a The two-layout fallback. #9052 has the Codex fixes. Now that #9011 has merged I'm rebasing it onto master (the only conflict is that both add a helper at the same spot in Per-engine groups. Agreed with @jinyouzhi that a move across TP or EP sizes inside the same processes needs each engine to hold its own groups. I'll list #9081 in #9026 as what state moves beyond DP changes depend on. |
…to its accumulation steps The move copied micro_steps from the source. A destination that accumulates over more micro-batches than the source then started mid-window: after one step of a source with one micro-batch per step, a destination with two stepped on its first micro-batch alone. The counter is now rounded up to a multiple of the destination's accumulation steps (the engine's boundary is (micro_steps + 1) % gradient_accumulation_steps == 0). Signed-off-by: RunguoLi <li19107254665@gmail.com>
… state over other data-parallel ranks in place Zero3StateMove moves a trainer's state between two engines, one per layout, built in advance. Zero3Repartition keeps one engine and changes its data-parallel ranks at a step boundary: it re-points the engine's references to its data-parallel group, allocates buffers of the new partition sizes (one per dtype, as ZeRO-3 holds the low-precision and gradient partitions) and points the existing tensor objects at them (so the optimizer state, the hooks and the parameter coordinator stay valid), recomputes the cached sizes and the gradient accumulation steps (the global batch is kept), moves the fp32 master partitions and the optimizer's moments with a RelayoutPlan from the old storage maps to the new ones, refreshes the low-precision partitions from the fp32 master, and releases the state of the ranks that leave. A rank that leaves keeps the engine and can come back. Every new member restarts its parameter fetch trace, so that ranks which never stepped in the engine record the next step together with the others. The plan is cached per source layout and optimizer state kinds (an optimizer has no moments before its first step). Supported: same-process changes of a ZeRO-3 engine whose parameters are all trained. Not supported: offloaded state, frozen parameters, hpZ, MiCS, sequence parallelism, quantized weights, and a process that joins after the engine was built. Tests (tests/unit/runtime/rollout/test_zero3_repartition.py, 4 ranks): one engine trains on four ranks, shrinks to two, trains, grows back to four, trains, and shrinks again with the cached plan; every micro-batch is the whole batch on every rank, so each layout computes the same gradient exactly, and after every phase the fp32 master and Adam moments equal bit for bit those of a plain engine that stays on four ranks (rect and affine planners, fp32 and bf16). A second test shrinks before the first step, grows back ranks that never stepped, and shrinks again with the plan made before the first step. Not covered by a test: a model with keep_in_fp32_modules (partitions in two dtypes). Signed-off-by: RunguoLi <li19107254665@gmail.com>
… member has one The parameter coordinator's fetch trace does not depend on the partition sizes. The re-partition restarted it on every new member at every change, so the first training step after each change ran without prefetching (on four RTX 3090s with Qwen3-0.6B: 1.01 s instead of 0.82 s on two ranks, 0.94 s instead of 0.69 s on four). It is needed only when a member has no complete trace (it has not stepped in this engine yet), since that member would compare its trace with the others' in a collective the ranks with a complete trace do not join. The members now exchange whether their trace is complete with the information they already gather, and all of them restart only if one has none. `trace_restarted` says which happened. Tests: after a shrink of ranks that all stepped, the trace stays complete; a grow that brings back ranks that never stepped restarts it. Signed-off-by: RunguoLi <li19107254665@gmail.com>
…s on every rank With data_types.keep_in_fp32_modules, ZeRO-3 puts the kept parameters in sub-groups of their own whose fp32 master is the weight partition itself. The install pointed these masters with ZeRO-3's _point_aliased_fp32_partitions_at_lp_buffer, which slices the training dtype's buffer only: a kept parameter's master would view the bf16 buffer (past its end in the test). Each aliased master now views its own sub-group's new partition and is again that sub-group's flat, as ZeRO-3 builds it. The parameters' process group, hpZ and Muon were checked after the first collective and only on the new layout's members, so the ranks that leave waited in the next collective instead of failing (zero.Init partitions over its data_parallel_group, the world when none is given, also when given an mpu). They are now checked with the other preconditions, on every rank, and raised on all of them together. Test: test_parameters_kept_in_fp32 keeps a logit scale in fp32 beside the bf16 weights of a tiny Qwen3 built under zero.Init, shrinks the engine from four ranks to two, trains, grows it back and trains, against an engine that trains on four ranks throughout. The moves keep the state bit for bit, the state equals the other engine's after every phase, the partitions keep both dtypes, and the kept parameter's forward value is its fp32 master. Without the fix the new members fail in the install. Signed-off-by: RunguoLi <li19107254665@gmail.com>
… partitions again Upstream deepspeedai#8981 zeroes, at each accumulation boundary, the gradient partitions that the window did not write. It finds them from a list of the parameters with data in this rank's partition (_params_with_grad_partition), which ZeRO-3 makes once. The list depends on the layout, and a re-partition kept it: a parameter that had only padding on a rank before and has data there after was left out, and when it got no gradient in a window, the rank applied its last gradient again. With gradient clipping (on by default), that gradient also enters the global norm, so every parameter's update changes. The install now resets the list (when the optimizer has one), and ZeRO-3 makes it again at the next boundary. test_unused_parameter_after_a_change: a two-element parameter, with data on ranks 0 and 1 of four, moves to ranks [2, 3] (data on both), which train a step that uses it and one that does not; the engine must match one that stays on four ranks. On this branch merged with master the test fails without the reset and passes with it; on the branch's own base, before deepspeedai#8981, it passes either way. Signed-off-by: RunguoLi <li19107254665@gmail.com>
Status: draft for discussion, not for merge as it is. Part of RFC #9026, which has not been accepted yet. I'm opening it so that the two-stage relay, the pipelined execution and the training-state move can be discussed on code that runs, next to #9045. The split I'd propose for upstream is at the end; each piece would be its own PR after the RFC is settled.
This PR does not propose a new rollout engine. The
NativeRolloutin the third commit is only the destination its tests write into; it is not registered withbuild_rolloutand stays out of upstream. For the hybrid engine, the re-shard at the training/rollout boundary (ZeRO-3 partitions into the inference layout ofHybridEngineRollout) is #9045'sAffineWeightSync, which takes maps only and can be called fromHybridEngine.eval(). What is here are the mechanisms beyond a fixed trainer on one node: the trainer's own layout changing with its optimizer state, ZeRO-3 over a subset of the ranks (#9076), and moves across nodes.What it is. Moving model state between two sharded layouts in GPU memory, without a checkpoint and without gathering a whole parameter on any rank:
Zero3StateMove), or inside one engine that changes its data-parallel ranks in place (Zero3Repartition, commits 6 to 8).The eight commits
[zero]ZeRO-3's rank-agreement checks and barriers run over the partition group. 31 lines added and 19 removed in five core files (stage3.py,partition_parameters.py,partitioned_param_coordinator.py,parameter_offload.py,zero/utils.py).assert_ints_same_as_other_ranks(ints, group=None)compares over the world, and an engine's defaultdp_process_groupholds every rank.[rollout]relayout.py,affine_plan.py,zero3_state.py(new files; no existing file changes).RelayoutPlan: both sides are declared per rank as affine pieces. Source and destination ranks may be disjoint, equal or overlapping, and a weight may have several holders. Planning runs identically on every rank from the gathered descriptions; execution is oneall_to_all_singleper bucket of weights and stage.topology="relay": a block that crosses nodes is fetched once per node and copied to the node's other ranks in a second stage.spread=Truehas the replicas of a node fetch bands of each large block and exchange them, so the traffic uses every NIC of the node. Over buckets the stages are pipelined: stage one of a bucket travels with stage two of the bucket before.planner="affine"(affine_plan.py): the same moves planned byplan_transfer. A relay is two plans through a relay map whose shards are bands of the destinations themselves, so there is no relay buffer. Two cases still fall back, per weight, to the module's own rectangle planner: holders in two layouts (waiting for Describe a parameter held in several layouts as one affine map #9052) and partial targets (RFC [RFC] Two-stage relay for cross-node weight sync, priced by a calibrated cost model #9026, open question 1).Zero3StateMove(src_engine, dst_engine): moves the training state throughzero_optimizer_storage_affine_maps([AffineIR] Affine storage maps for live ZeRO parameters and optimizer states #8885).release_training_state/restore_training_stategive back and re-allocate the inactive engine's memory.[rollout]native/and the tests. A small tensor-parallel decoder whose weights are views of one buffer per rank, written in place, so CUDA graphs captured over one version of the weights stay valid for the next.NativeRolloutis the destination the tests move weights into.RolloutBatchgains an optionalresponse_logprobs.[rollout]NativeRolloutleavesbuild_rollout, which is as on master again. The tests constructNativeRolloutdirectly.[rollout]Zero3StateMovealigns the destination's micro-step counter to the destination's accumulation steps. It copied the source's counter, so a destination that accumulates over more micro-batches than the source started mid-window: after one step of a source with one micro-batch per step, a destination with two stepped on its first micro-batch alone.[rollout]Zero3Repartition(inzero3_state.py, and a new test file). One ZeRO-3 engine changes its data-parallel ranks at a step boundary, with no second engine.RelayoutPlan, cached per source layout and optimizer state kinds, and refreshes the low-precision partitions from the fp32 master. The ranks that leave release their state; they keep the engine and can come back.[rollout]Zero3Repartitionkeeps the parameter fetch trace when every new member has a complete one.[rollout]Zero3Repartitionwith parameters kept in fp32, and its preconditions checked on every rank.data_types.keep_in_fp32_modules(Keep the parameters a model names for fp32 in fp32 under ZeRO-3 #8843), the kept parameters form sub-groups of their own whose fp32 master is the weight partition itself. The sixth commit pointed these masters with ZeRO-3's_point_aliased_fp32_partitions_at_lp_buffer, which slices the training dtype's buffer only, so a kept parameter's master would view the bf16 buffer. Each aliased master now views its own sub-group's new partition.zero.Initwith anmpubut nodata_parallel_groupis partitioned over the world, not over the engine's group.How it relates to open work
AffineWeightSync) is the same weight move on the IR alone (plan_transfers+execute_transfers), validated on one node. This draft holds what [rollout] Affine weight-sync building block: ZeRO storage into a tensor-parallel inference copy, shard to shard #9045 leaves out: the relay, the pipelined execution, holders in two layouts, and the training state.native/arena.pyduplicates [rollout] Affine weight-sync building block: ZeRO storage into a tensor-parallel inference copy, shard to shard #9045'sWeightArena; only one of them should land.AutoTPRollout) is a tensor-parallel rollout engine already under review.NativeRolloutoverlaps with it and is not proposed to land next to it. It is here because the relayout and trainer-switch tests run through it.schedule_transfers) is what the pipelined execution should be built on: with the schedule exposed, stage two of one bucket can share a collective with stage one of the next. The executor inrelayout.pydoes this with its own bucketing today.Zero3StateMoveneeds aTrainerMPUper engine, made on every rank, with the engine over all ranks built first (the build order is [BUG] ZeRO-3 ids come from process-wide counters, so ranks that built different models fail the rank-agreement checks #9098).Zero3Repartitionre-pointsgroups.mpuand the engine's group references by hand.test_zero3_repartition.pyandTestTrainerSwitchcan serve as stage 2's multi-engine checks.Validation
One node with 4× RTX 3090 (24 GB), driver 595.91.07, torch 2.13.0+cu130, NCCL 2.29.7. Commands from
tests/, each aspython -m pytest -q -rfEs <files>. The rows marked head ran on the head commit, on another node with the same GPUs and software (driver 595.58.03). The others ran on the sixth commit; the seventh and eighth change onlyZero3Repartitionand its test.unit/runtime/rollout/test_zero3_repartition.py(4 ranks), headunit/runtime/rollout/test_relayout.py,test_native_decoder.py,test_native_rollout.pytest_relayout.py,test_native_rollout.py, headunit/runtime/rolloutunit/checkpoint/test_affine_shard_map.py,test_affine_transfer_plan.py,test_affine_transfer_executor.py,test_affine_transfer_executor_external_group.py,test_affine_replica_policy.py,test_zero_affine.pyunit/v1/zero/test_zero.py,unit/v1/zero/test_zero_init_pg_contradiction.py,unit/runtime/zero/test_zero_context.pytest_hybrid_engine_rollout.py. They need thetransformer_inferenceextension, which does not build in this environment: its CUDA toolkit has nocuda_profiler_api.h. With that header taken from the machine's CUDA 13.0 install, the file passes (72 passed). Neither test touches this change.TestNativeRolloutApartandTestTrainerSwitchintest_native_rollout.py.TestTrainerSwitchalso compares training across two trainer layouts with a plain-torch AdamW reference.TestTrainerSwitchAccumulationmoves the state into an engine that accumulates two micro-batches per step. It fails without the fifth commit (its first micro-batch is taken as a boundary).test_zero3_repartition.py, main test. One engine trains on four ranks, shrinks to two, trains, grows back to four, trains, and shrinks again with the cached plan.test_zero3_repartition.py, second test. The engine shrinks before its first step, then grows back ranks that never stepped (the trace case of commit 7).test_zero3_repartition.py, third test (commit 8). A tiny Qwen3 is built underzero.Initwith a logit scale kept in fp32 beside its bf16 weights. It shrinks from four ranks to two, trains, grows back and trains, against an engine that trains on four ranks throughout.Zero3Repartitionin eleven ways, one at a time, and each break makes these tests fail or hang (head). The breaks: moving only the fp32 master; keeping the old gradient positions; keeping the old partition sizes; keeping stale step counters on a rank that comes back; not aligning the micro-step counter; never restarting the trace; always restarting it; leaving the state kinds out of the plan cache key; one low-precision buffer in the training dtype for every parameter; one gradient buffer for every parameter; ZeRO-3's alias helper instead of the per-sub-group alias (the code before commit 8).End to end, a ZeRO-3 trainer goes DP4 → DP2 → DP4 inside the same four processes (Qwen3-0.6B, bf16, Adam; 310 parameters; 6.66 GiB of fp32 master weights and two moments). Both paths ran, three rounds each per planner, with a training step after every move (sixth commit):
Zero3StateMove)Zero3StateMove)plan_transfer)Zero3Repartition)Zero3Repartition)plan_transfer)What I could not run, and known limits
get_accelerator()anddeepspeed.comm, but the native decoder's graph capture is CUDA-specific.keep_in_fp32_modules).Zero3Repartitionkeeps one buffer per dtype, as ZeRO-3 does; the third test covers one parameter kept in fp32 beside bf16 weights.NativeRolloutassumes one dtype.Zero3Repartitionchanges ranks only among the processes that built the engine. It does not support offloaded state, frozen parameters (e.g. a LoRA base), hpZ, MiCS, sequence parallelism or quantized weights. No test saves a checkpoint after a change.Zero3Repartitionchecks its preconditions before its first collective, on every rank. An error after that point on some ranks only (in the install) still leaves the other ranks waiting.Zero3Repartitionsets ZeRO-3 internals directly: the group references, including the parameter coordinator's private process group, and the cached partition sizes. A change to those internals needs a change here. [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081's per-engine groups would remove the group part.NativeRollout.bind(NotImplementedError).all_to_all_singlemessage. Replaying the same plans on CPU is exact for all 707 weights.torch.distributed.all_to_all_singlewith self-describing data, no DeepSpeed, no model.NCCL_IB_QPS_PER_CONNECTION=4orNCCL_NCHANNELS_PER_NET_PEER=8.affine-transfer.rstfrom [rollout] Affine weight-sync building block: ZeRO storage into a tensor-parallel inference copy, shard to shard #9045).Size, and the split I'd propose. 2,709 new lines under
deepspeed/and 1,476 in tests is too much for one review. For upstream I would not send it in this form:plan_transfers+execute_transfers, as [rollout] Affine weight-sync building block: ZeRO storage into a tensor-parallel inference copy, shard to shard #9045 does for weights), with DP4 → DP2 → DP4 as its test.Zero3Repartition). It needs no second engine and no engine build order, and it uses less memory. Its group handling would follow [RFC] Cleanup: move process-wide parallel-group globals to per-engine groups — follow-up to #8241 / #8851 #9081's per-engine groups.Zero3StateMovewould remain only as a second path in that test, or stay out.plan_transferscalls through a relay map. The planner needs nothing new; the open questions are where the relay-map constructor lives and how the choice meets the cost model of [AffineIR] Calibrate affine transfer links per path (Plan analysis & calibration Part-2) #8966 / [AffineIR] Analyze affine transfer batches before execution (Plan analysis & calibration Part-1) #8967.NativeRolloutstays out; the destination upstream is feat(rollout): add AutoTP rollout engine with in-memory weight sync #9023's engine through [rollout] Affine weight-sync building block: ZeRO storage into a tensor-parallel inference copy, shard to shard #9045's sync.Three questions for the RFC:
affine_plan.pywhat [RFC] Two-stage relay for cross-node weight sync, priced by a calibrated cost model #9026 should describe?AI assistance: AI assistance was used. The code, tests and this description were drafted with an AI coding agent (Claude Code). The submitter reviewed every changed line and ran the tests above.
Maintenance contact: @LiRunGuo. cc @Achyuthan-S @0z5a @jinyouzhi