Repository navigation
Description
Describe the bug
ZeRO-3 numbers parameters and modules with two counters that live for the whole process: Init.param_id (partition_parameters.py, set in _convert_to_deepspeed_param) and the mutable default count=[0] of DeepSpeedZeRoOffload._register_deepspeed_module (parameter_offload.py). The checks that ranks agree compare these ids as they are: the coordinator's trace checks in reset_step (partitioned_param_coordinator.py), the bucket check in stage3.py under enable_sanity_checks, and all_gather_coalesced in safe mode. So if one rank has built more ZeRO-3 parameters or modules than another earlier in the process, the same model gets different ids on the two ranks, and the first check fails although both ranks hold the same model.
This comes up when ZeRO-3 runs on some ranks before an engine over all of them, for example a trainer on some ranks and a rollout or reference model on others. @LiRunGuo hit it in #9074 (see #9076), which avoids it by building the engine over all ranks first.
To Reproduce
Master at e35282b, CPU, gloo, torch 2.8.0. Rank 1 alone builds a model under zero.Init, then both ranks train one engine:
# DS_ACCELERATOR=cpu torchrun --nproc_per_node 2 repro.py
import torch
import deepspeed
import deepspeed.comm as dist
class Net(torch.nn.Module):
def __init__(self):
super().__init__()
self.layers = torch.nn.Sequential(torch.nn.Linear(8, 8), torch.nn.Linear(8, 8))
def forward(self, x):
return self.layers(x).square().mean()
deepspeed.init_distributed(dist_backend="gloo")
rank = dist.get_rank()
own_groups = [dist.new_group([r]) for r in range(dist.get_world_size())]
# Rank 1 alone builds a model under ZeRO-3, on a group of its own.
if rank == 1:
with deepspeed.zero.Init(data_parallel_group=own_groups[rank]):
Net()
# Then both ranks train one model together.
config = {
"train_micro_batch_size_per_gpu": 2,
"zero_allow_untested_optimizer": True,
"zero_optimization": {
"stage": 3,
"stage3_param_persistence_threshold": 0
},
}
model = Net()
engine, _, _, _ = deepspeed.initialize(model=model,
model_parameters=model.parameters(),
optimizer=torch.optim.SGD(model.parameters(), lr=0.1),
config=config)
print(f"rank {rank}: ds_ids {[p.ds_id for p in model.parameters()]}", flush=True)
for step in range(2):
loss = engine(torch.randn(2, 8))
engine.backward(loss)
engine.step()
print(f"rank {rank}: step {step} done", flush=True)rank 1: ds_ids [4, 5, 6, 7]
rank 0: ds_ids [0, 1, 2, 3]
rank 1: step 0 done
rank 0: step 0 done
File "deepspeed/runtime/zero/partitioned_param_coordinator.py", line 269, in reset_step
File "deepspeed/runtime/zero/utils.py", line 125, in assert_ints_same_as_other_ranks
RuntimeError: Detected a disagreement on list contents between rank0 and rank1:
list length: 8
rank0: [0, 1, 2, 3, 2, 3, 0, 1]
rank1: [4, 5, 6, 7, 6, 7, 4, 5]
The check runs when the second step starts, after the first step has recorded the trace. When both ranks build the extra model, both get ids [4, 5, 6, 7] and training runs.
The module counter fails the same way. On #9076 (29223f9), which runs these checks over the partition group so that ZeRO-3 can train on ranks 1 and 2 of three, training there first and then one engine over all three ranks fails the module check (partitioned_param_coordinator.py:266): rank 0 has [0, 2, 2, 2, 3, 4, ...], ranks 1 and 2 have [4, 6, 6, 6, 7, 8, ...]. On master the engine on ranks 1 and 2 never gets that far: while it is built, it waits for rank 0 at a barrier over the world (stage3.py, _create_fp16_partitions_with_defragmentation), which is what #9076 fixes. For this run the engine over all ranks needs an mpu of its own; without one, ranks 1 and 2 keep the first engine's groups.mpu and use a different data-parallel group than rank 0 (#9081).
Expected behavior
The checks pass for the same model on every rank, whatever else each process built before, and still fail for a different set or order.
Possible fix
The ids themselves should stay unique within a process: other code keys on them, for example the NVMe parameter swapper names its files {ds_id}_param.tensor.swp. The checks could instead compare each id's position among the engine's own ids, as @LiRunGuo suggested on #9076. The engine knows its modules and parameters, so it can map each id to its index among them once, at init, and the checks compare those indices. I'm happy to send this with a test (ZeRO-3 on some ranks, then an engine over all of them, on CPU) if the direction looks right.
Activity
LiRunGuo commentedon Nov 17, 2026
Confirming from #9074: we hit the parameter-id case in our first DP4 ↔ DP2 test. There the two-rank engine was built before the four-rank one, and the four-rank engine's first trace check failed with a list-contents disagreement. The workaround is the build order noted on
TrainerMPU: the engine over every rank first. That ordering is there only for this reason, so the position-based comparison would let it go.A related case for whoever takes this, from re-partitioning one engine in place (
Zero3Repartition, now in #9074):reset_steponly runs on ranks without a complete trace. It is a collective, so after a grow that brought such ranks back, the first step failed with a list-length disagreement.Zero3Repartitionrestarts the trace on every member, so that all of them record the next step together.I can run a fix against #9074's tests on 4 GPUs.
Achyuthan-S commentedon Nov 19, 2026
Thanks @LiRunGuo, that's a useful second case. Comparing positions wouldn't cover it: there the ids agree, and what differs is whether a rank has a trace at all. As you describe it, the check in
reset_stepruns only on ranks without a complete trace, and it is a collective, so ranks that kept their trace and ranks that rejoined without one disagree. Making it a collective that every rank of the group joins, whatever its trace state, sounds like the right fix for that half.@jinyouzhi, since you've put the per-engine fix for this in scope for #9081, next to stage 2, I'm happy for it to go there rather than as a separate PR from me. Both halves are per-engine state, and stage 2 is where the engine owns it. The repro above and the module-counter run on #9076 can become the tests. I'll review and run the fix on CPU, and @LiRunGuo has offered #9074's tests on 4 GPUs. If you'd rather have the position-based change on its own first, say so and I'll open it.
jinyouzhi commentedon Nov 19, 2026
Thanks @Achyuthan-S , and thanks @LiRunGuo for the trace-state case.
#9098 should stay a separate PR from you, not part of #9120. #9120 changes which groups an engine uses; it doesn't touch the ids or the checks (
param_id,_register_deepspeed_module'scount,reset_step, theenable_sanity_checksbucket check, the safe-mode check inall_gather_coalesced), and it is already large. "In scope next to stage 2" in #9081 meant in scope for the RFC as its own track, not inside #9120; sorry for the confusion.So please go ahead with your own PR: the position-based comparison, with the repro above as the test, and the trace-state half too (every rank of the group joins the check in
reset_step, whatever its trace state). The two PRs don't block each other: only the end-to-end run from #9076 (ZeRO-3 on some ranks, then an engine over all of them without its ownmpu) also needs #9120, and I'll run it once both are up. Happy to review.