Repository navigation
Describe a parameter held in several layouts as one affine map - #9052
Achyuthan-S wants to merge 3 commits into
Conversation
be8531f to
110ff4b
Compare
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 110ff4b8b3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if len(piece.shape) == len(logical_shape) and all( | ||
| size <= 1 or stride == expected | ||
| for size, stride, expected in zip(piece.shape, piece.source_strides, strides)): | ||
| origin = _coordinate(piece.source_offset, strides, logical_shape) |
There was a problem hiding this comment.
Reject blocks that cross logical-axis bounds
When an otherwise executable piece wraps across a logical-axis boundary (for example shape=(1, 4), source_offset=0, and source_strides=(3, 1) for a (2, 3) parameter), matching row-major strides and a valid starting coordinate make this branch classify it as axis-aligned even though origin + shape exceeds the logical shape. _cut then models offset 3 as nonexistent coordinate (0, 3) rather than (1, 0), and the returned union advertises different locations for the two real copies; both validate_coverage() and plan_transfer() reject it. Check that LogicalBox(origin, piece.shape) is contained in the logical parameter before accepting it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Thanks, confirmed: for a (2, 3) parameter the (1, 4) run from offset 0 was cut as the box (0, 0)+(1, 4), and the union came back with a piece that validate_coverage and plan_transfer then rejected. _block_of now refuses a block that doesn't fit inside the parameter, the same way it refuses other pieces that aren't blocks, and test_a_union_refuses_layouts_it_cannot_merge covers it. Fixed in e97f115 (45652c6 before the rebase onto master).
| for other, holders in blocks: | ||
| overlap = block.intersection(other) | ||
| if not overlap.is_empty: | ||
| parts.append((overlap, piece.locations | holders)) |
There was a problem hiding this comment.
Refine replica boundaries within each input map
When one valid input has replicas with different piece boundaries—for example location 0 stores one (2, 3) box while location 1 stores two (1, 3) boxes in swapped shard rows, all declaring {0, 1}—_blocks_held contains a full box plus its halves. This loop appends overlaps against the original block rather than a common disjoint refinement, so unioning a third full-copy layout gives that location the full box and both halves; affine_map.validate() then raises because 12 elements are assigned to a 6-element shard. Include boundaries within each input map when building the disjoint refinement before emitting pieces.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Thanks, confirmed: that input made the union raise "Rank 10 holds a shard of 6 elements but its pieces account for 12". Every piece is now cut into the disjoint refinement of all the blocks both layouts hold, including one layout's own replica edges, so each part appears once and lists every holder. test_a_union_cuts_replicas_that_split_a_block_differently covers it. On 900 random layouts the holders match the previous commit for every element, and every plan is exact or refused exactly as before. Fixed in e97f115 (45652c6 before the rebase onto master).
45652c6 to
e97f115
Compare
e97f115 to
d175096
Compare
jinyouzhi
left a comment
There was a problem hiding this comment.
The cutting looks right to me, and checking holders and extract against each layout separately is a good oracle. The new tests pass locally (CPU). Two things:
1. Union cost grows quadratically with the number of partitions. _disjoint_blocks intersects every block with every part found so far, and _cut intersects every piece with every part. A (4096, 4096) parameter, ZeRO-3 partitions plus an 8-way row split:
| partitions | pieces | union_affine_maps |
|---|---|---|
| 25 | 160 | 0.02 s |
| 50 | 308 | 0.08 s |
| 100 | 604 | 0.29 s |
| 200 | 1196 | 1.12 s |
At 200 partitions, about 2/3 of the time is in _cut and 1/3 in _disjoint_blocks, nearly all of it in LogicalBox.intersection (545k calls). Extrapolating, 1000 partitions is about 28 s for one parameter; a run at that size had not finished after 15 minutes. With a few hundred parameters per model and a data-parallel size in the hundreds, the union alone would take minutes, and it isn't covered by PlanBudget. Since the parts are already sorted, an interval index on axis 0 (bisect on the row range, then intersect only the candidates) should make both loops close to linear for the row/flat layouts this targets. A timing check at a few hundred partitions would also be worth adding to the PR description.
script
import time
from deepspeed.checkpoint.affine import contiguous_split_map, nest_affine_maps, replicated_map, union_affine_maps
from deepspeed.checkpoint.zero_affine import zero_storage_affine_map
shape = (4096, 4096)
numel = shape[0] * shape[1]
for parts in (25, 50, 100, 200):
zero = nest_affine_maps(replicated_map(shape, 1),
{0: zero_storage_affine_map(numel, list(range(100000, 100000 + parts)), -(-numel // parts))})
shard = contiguous_split_map(shape, [512] * 8, 0)
start = time.time()
union = union_affine_maps([zero, shard])
print(parts, sum(len(p) for p in union.pieces_by_rank.values()), f"{time.time() - start:.2f}s")2. Error types. A piece that isn't an axis-aligned block raises NotImplementedError, and mismatched inputs raise ValueError, while the planner reports the same kinds of problems as UnsupportedLayoutError / InvalidMapError (both AffineTransferError). A caller such as #9045 that falls back to its own planner on AffineTransferError would not catch these. Using the planner's error types would keep that one except. Minor; #9045 doesn't call this yet.
A rollout replica can keep a tensor-parallel shard that is still current while the trainer's ZeRO-3 partitions hold the whole parameter. union_affine_maps cuts each layout's pieces at the edges of the blocks the other layouts hold and lists every holder, so plan_transfer can read either copy. Move the planner's box subtraction into affine_geometry so both use one copy. Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
Replicas within one layout may store a region whole or in parts, so cutting a piece only at the other layout's blocks could place the same elements twice in one shard. Cut every piece into the disjoint refinement of all the blocks both layouts hold, each part listing every holder. A row-major run longer than what is left of its row is not a block, so refuse it instead of cutting it as one. Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
…'s errors A union compared every block with every part found so far, and cut every piece against every part, so its time grew with the square of the number of ZeRO-3 partitions: about 48 s at 1000 partitions of a 4096 x 4096 parameter. The blocks come in order of their first row, so a part that ends above a block is final, and each block now meets only the parts that reach into its rows. The parts refine every block, so a piece is cut against only the parts that start in its rows. The pieces are the same. Layouts that contradict each other now raise InvalidMapError, and a piece that is not a block UnsupportedLayoutError, so a caller that falls back on AffineTransferError also catches a union that cannot be formed. Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
d175096 to
4a6bd3d
Compare
|
Thanks @jinyouzhi, both points were right, and your script made the first one easy to reproduce. 1. Union cost. Fixed in 4a6bd3d.
The column split is the slowest case left, because each of its blocks spans every row. At 1000 partitions the union has 34k pieces. I compared the new code against the previous commit on 6,000 random unions of two or three layouts: ZeRO-3 partitions, row and column splits and replicas, with any subset of shards surviving. Both give identical pieces, the same holders for every element and the same shard at every location, and they refuse the same inputs. The timing is in the description now. 2. Error types. Agreed. Now:
Both subclass I also rebased onto master 97aa3c1, after #9039. The only conflicts were the geometry imports: |
Refs #9026 (open question 8), #9045.
Why
A rollout replica that changes placement can often keep a tensor-parallel shard that is still current, while the trainer's ZeRO-3 partitions hold the whole parameter.
plan_transfercan read a block from any of its holders only if the source map's blocks are equal or disjoint and each block lists every holder. A map that simply lists both layouts' pieces breaks both rules, since a partition can start or end inside a row of the shard. So today the planner refuses it, and #9045 falls back to its own planner for these weights.What
union_affine_maps(maps)indeepspeed/checkpoint/affine.pyreturns one map of a parameter that several layouts hold at once.TransferEndpointsalready lets several locations map to one process.InvalidMapError, and a piece that is not an axis-aligned block raisesUnsupportedLayoutError: the planner's types for the same problems, so a caller that falls back onAffineTransferErrorcatches these too.subtractmoves unchanged fromaffine_transfer._subtractintoaffine_geometry, the module [AffineIR] Move shared affine index and box geometry into affine_geometry #9010 added for shared box arithmetic. The planner imports it under its old name.affine-transfer.rst, after the replica-policy section from [checkpoint] Batch replica selection for affine transfer plans #8936.Scope
Cuts along rows are planned today. A ZeRO-3 block of several rows cut at a column edge keeps its row-major strides and its partition's offset, so it is not dense in its flat shard, and
plan_transferrefuses it. @0z5a's #9070 makes the planner accept that case (see #9026). The union already describes it exactly, and the tests check that at the map level.Cost
@jinyouzhi measured the first version growing with the square of the number of ZeRO-3 partitions.
union_affine_mapson a (4096, 4096) parameter's ZeRO-3 partitions plus another layout, macOS arm64 CPU (before = the previous version):A column split is the slowest case left, because each of its blocks spans every row.
Tests
test_affine_transfer_plan.py:extractexactly;test_affine_transfer_executor.py: on 2 ranks, one process holds a ZeRO-3 partition and a surviving row shard under two locations. The plan runs throughexecute_transfer, and every target shard equalsextract.Not merging the holders, dropping the part of a block that a later block doesn't overlap, or leaving the destination offset unmoved makes 7, 8 and 7 of these 9 tests fail. Setting a part aside one row too early, or ending the search for the parts under a piece one row short, makes 5 of them fail.
Validation
Rebased on master 97aa3c1, after #9039 renamed the geometry imports. The only conflicts were those imports and
affine_geometry.__all__;subtractnow sits with master's unaliased imports. macOS arm64 CPU, Python 3.9.6, torch 2.8.0, gloo (DS_ACCELERATOR=cpu LOCAL_SIZE=4):The runs below are from before this rebase and the cost change.
Earlier, rebased on master 892ed83, which includes #9011. The code and test changes are the same as before the rebase; the only conflicts were additions next to #9011's (
subtractnext toboxes_disjointinaffine_geometry.py, and the imports). macOS arm64 CPU, Python 3.9.6, torch 2.8.0, gloo (DS_ACCELERATOR=cpu LOCAL_SIZE=4):On master 988b2e3, before that rebase (with #9010 and #8936):
test_affine_replica_policy.py: 412 passed.plan_transferswithlocal_first_least_loadedand withfirst_holderinto both targets: all 16 plans reproduceextractexactly.Before the rebase, on f9fe4df with #9010 applied:
On GPU, also before the rebase: 4× A100-SXM4-40GB (NVLink), driver 595.71.05, torch 2.12.1+cu130, NCCL 2.29.7, Python 3.12.11, with
pytest --forkedover six of the affine test files (all buttest_autoep_affine_universal.py):cc @0z5a @LiRunGuo @jinyouzhi