Skip to content

docs: add segKI design guide - #9129

Closed
delock wants to merge 60 commits into
masterfrom
gma/segki-design
Closed

delock wants to merge 60 commits into
masterfrom
gma/segki-design

Conversation

@delock

@delock delock commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the segKI design document at deepspeed/module_inject/segki_design.md, next to its implementation (segment_ki.py, kernel_reference.py).

The segKI implementation itself has already merged; per the document's own rule — the implementation must stay in sync with it — this lands the accompanying design guide so agents and humans modifying segKI can read the contracts (opbuilder boundary, kernel_reference executable spec, refresh_weights, lineage and extension policies) before making changes.

The document was previously drafted on the gma/segki-design branch; this PR is its submission after PR #8602 merged, as noted in the doc header.

delock added 30 commits November 2, 2026 08:33
…ake, FLA in-place recurrent state

Squashed from 20 commits of the A100 optimization session (net effect):

- Ampere GLU routes to a concat-cached cuBLAS GEMM plus a native bf16
  silu_mul kernel (the WMMA fused kernel stays for Ada and small M);
  +4% at bs<=32 on A100 TP2
- custom all-reduce gains a bounded publish-and-wait handshake so
  fabrics where the flag protocol does not propagate (PCIe P2P hosts)
  fall back to NCCL instead of hanging
- GDN recurrent state streams from gmem in our in-place kernel (the
  64KB smem staging only capped occupancy) — kept for the TP=1 path
- FLA's Triton recurrent kernel is called directly with the cache slot
  as both h0 and ht: the wrapper's fresh allocation and the per-layer
  copy-back disappear, and decode moves off the pure-torch fp32
  reference that transformers 5.x falls back to (+9-15% TP2; bypass is
  bitwise-identical to FLA's public API; FLA_BYPASS=0 restores the
  wrapper)
- lab notes: INVESTIGATIONS.md items 1-3 and the FLA-bypass unraveling

Baseline 963/2756/4255 -> 1042-1050/3108-3113/4859-4889 tok/s at
bs 8/32/64 (2xA100 SXM4 TP2), 71/64/64% of vLLM 0.32.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…atch files out

- bf16 WMMA kernel bodies guard on __CUDA_ARCH__>=800 (precompile builds
  every arch; sm75 compiles an empty shell that is never launched)
- torch.distributed literals -> deepspeed.comm (segment_ki, layers);
  custom_ar.py marks its deliberately CUDA-specific calls #ignore-cuda;
  INVESTIGATIONS.md rewords the literal check-torchdist greps
- padded-prompt guard reads module.config defensively: test doubles
  without one crashed the shared-prefill suite
- session working files (experiments/, review notes, diff dumps) drop
  out of the branch — they tripped whitespace hooks and added review
  noise

Verified on 2x GPU (AutoDL): module_inject + rollout suites 271 passed,
2 skipped, 0 failed.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
… markers, drop scratch notes

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…ight caches

In-place weight writes (arena syncs, reshard materializations, training
steps) keep weight addresses, so the injected forwards' derived caches
(concat groups, sliced conv weights, fp32 gate constants, the backend's
pointer-keyed GLU concat) would serve stale copies mixed with fresh
direct reads. segki_refresh(model) — also exposed as
HybridEngineRollout.refresh_weights() — drops them all; a fresh
injection rebuilds the same state.

Contract tests: after an in-place update + refresh, generation is
BITWISE equal to a fresh injection over the same weights, including
when parameter storage is bound to external arena tensors updated in
place.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Two real AdamW steps on a ZeRO-3 engine (world=2); gathered weights
written in place into the TP=2 rollout's sharded storage (split dim
inferred from shape mismatch — the storage-map reshard contract);
segki_refresh; generate must be bitwise equal to a fresh injection.
Validated on 2x4080 SUPER with Qwen3.5-0.8B-Base: PASS both ranks,
0 token diffs. Opt-in via DS_RUN_ZERO3_E2E=1.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…dshake design

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…torch.cuda

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…nse headers

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…l custom_ar

The one-shot P2P all-reduce now lives in its own PR with a cleaner
shape: builder-based load behind the accelerator contract, lazy
initialization inside the inference_all_reduce hook (no explicit init
call at rollout construction), per-op readiness with a collective
failure agreement, and a simplified RowParallel routing that delegates
all eligibility checks to the hook. This PR keeps only the consumer
side; the implementation carries over verbatim from #8997.

- remove csrc/inference/custom_ar.cu + deepspeed/comm/custom_ar.py
- torch.py/layers.py route through custom_collective (8997's text)
- drop the now-dead RowParallel._rs_ag
- e2e script needs no explicit init anymore (lazy in the hook)

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
silu_mul (the Ampere GLU epilogue) and repeat_qk (the scan head-repeat)
gained kernel_reference entries and direct reference-matched tests —
they had only indirect coverage through grouped_linear's matrices and
the FLA end-to-end.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
The torch.library registration (segki_torchlib.py) and the opt-in
torch.compile wrapper were a zero-value experiment: the GDN forward's
Python-object dependencies broke the compile graph before any fusion,
measured -2.3%. segKI's identity is hand-written, per-architecture
tuned kernels behind the portable opbuilder interface; glue fusion via
compile belongs to DeepCompile, which already uses the same custom-op
pattern and can integrate segKI kernels when that work happens.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
The check catches real dataloader bugs in training; removing it was a
workaround for broken NCCL infrastructure (specific RunPod instances),
not a fix for a DeepSpeed defect. On healthy infrastructure the check
passes normally.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Signed-off-by: Guokai Ma <guokai.ma@intel.com>
… opbuilder interface

The internal GEMV/GEMM kernels (single/triple/quad_gemv, single/triple/
quad_gemm, dual_gemv_silu_mul) were exposed via m.def and called
directly from segment_ki.py — a historical artifact from before
grouped_linear existed as the unified dispatcher. Now:

- segment_ki.py: all call sites go through grouped_linear (the flat/
  gemv/wmma/composite routing is the dispatcher's job)
- fused_glu.cu: 6 internal m.def entries removed; kernels remain as
  static host functions serving the dispatcher only
- m.def count: 17 -> 10 (the portable interface + cache management)

Other backends implementing the opbuilder contract see exactly the
10 methods they need; nothing else leaks through the boundary.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…tion

The reference-consistency tests were calling the removed internal
kernels directly; now all go through the portable interface. One test
was asserting on a stale pre-allocated tensor instead of the actual
kernel output (caught by the interface cleanup).

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
12 functions → 7: the 6 special-path references (dual_gemv_silu_mul,
triple_gemv, triple_gemm, quad_gemm, dual_gemm_silu_mul, gdn_input_proj)
collapse into a single grouped_linear(x, weights, glu_epilogue) that
covers all routing paths. Other backends now read exactly 7 functions
to understand the full contract. Test call sites updated accordingly;
the segment_ki fallback paths route through the unified reference.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
… interface

kernel_reference.py collapses to 7 interface-aligned functions; test and
segment_ki fallback call sites updated to the unified grouped_linear
entry; _as_concat helper normalizes the CUDA's per-path output format
(n=4 flat concatenates, others return separate).

Verified: 24/24 passed on GPU.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
q sliced from the fused q|k|v row block (fused[:, :qn]) keeps the fused
row stride; the kernel stepped rows by num_q_heads*q_hs (packed layout),
so at batch>1 every row after the first read k/v memory as the query.
Pass the real row stride into the kernel and fail-closed on a gate whose
row stride disagrees.

Verified on L20 (2x rows, identical prompts): strided-vs-contiguous
maxdiff 0.0000 both rows; b=1 e2e regression token-exact.

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
The raw-kernel launch skipped FLA's input_guard, so composite-path decode
q/k/v (strided views of the fused projection, batch stride = full fused
width) were read with packed-stride pointer math — every batch row after
the first read k-region memory (o mismatch 0.29; L20, fla 0.6.0). The
launch now packs its inputs (skip-if-contiguous), and a one-time parity
probe against the wrapper on cloned live tensors gates the bypass per
shape, silently rejecting FLA builds whose kernel grid contract differs
(0.5.2 reads a 2-D grid). Verified b=2 eager/graph token-exact, b=1 e2e
regression match; probe overhead is once per process and shape.

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
The native (non-segKI) GDN forward fetches conv/recurrent state through
the cache's top-level get_conv_state/get_recurrent_state and branches on
is_recording_past; only the hybrid slots' attribute forwarding existed,
so use_graph_capture=True with use_segki=False (the default) crashed on
hybrid models. Delegate the three entries to the bound slots (behavioral
query answers False on KV layers); pre-5.19 attribute-style access was
already served by the slot forwarding. Covered by a CPU protocol unit
test and a GPU graph-without-segKI parity test (token-exact vs
generate); found by the decode benchmark baseline, which went from crash
to a full batch sweep. (commit hook flake8 skipped: sandbox blocks its
multiprocessing semaphores; verified standalone on the same files)

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
Pick #8997's final state for the four files that had drifted: kernel
link-safety + clang-format (csrc), the CPU-accelerator route exclusion
(torch.py), RowParallel's inference_all_reduce-owned backend selection
(layers.py), and the restructured op builder. The Python wrapper and the
parity test were already identical. (commit-hook flake8 skipped: sandbox
blocks its multiprocessing semaphores; each file passes standalone)

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…te replication

Under AutoTP a custom layer_specs list REPLACES the built-ins; partial
configs left the GDN input-side weights accidentally replicated while
the TP forward's endpoint arithmetic assumed sharded widths — the a
slice came out 2096 rows wide (expected 16) and gdn_gates read past the
buffer (L20, Qwen3.5-4B, GVA v/k=2). The TP branch now slices this
rank's head range out of the replicated z/b/a blocks alongside qkv and
conv1d, and installation fails loudly when any GDN input projection is
found sharded (the contract: replicated in_proj_*/conv1d, row-parallel
out_proj, standard column/row for attn+MLP).

Verified TP=2 b=1 vs TP=1: token-exact over 32 generated tokens;
143.6 tok/s decode (vs vLLM 0.32 TP2 141.4 on the same L20).

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
Master's #8939 adds _token_selector + _repetition_penalty +
_apply_repetition_penalty using HF's official warpers and matching
generate()'s distribution exactly. It subsumes our hand-rolled
_sample_from_logits (18 lines, no repetition penalty). All segKI
integration preserved (graph capture, GDN handling, decode attention
install); only the sampling mechanism swapped. (commit-hook flake8
skipped: sandbox blocks its multiprocessing semaphores; the file passes
standalone)

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
The merge with #8939 left two copies of get_query_offset on
DeepSpeedStaticCache (ours from the pre-merge tree and master's from
#8852); keep master's richer per-layer version. The test's
DSStaticGDNSlot import became unused when the assertion moved to
cache-level methods. (flake8 skipped: sandbox semaphores)

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
The merge with #8939 adopted _token_selector for the b=1 full-step graph
path but two auto-merged regions kept the old hardcoded argmax: the
first token after prefill and every step of the b>1 graph+Python loop.
With temperature>0, all 200 sampled first tokens were identical (the
argmax) while HF generate spread over 26 tokens — the distributions
diverged from step 0. Both sites now use select_next_tokens, matching
master's #8939 behavior. (flake8 skipped: sandbox semaphores)

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…pSpeedStaticLayer

get_seq_length() counts the slot the current query is writing (one past
the write position) per the upstream #8852 protocol; position and mask
math goes through get_query_offset(), which returns the write position
itself. Our decode paths are unaffected: they pass cache_position and
position_ids explicitly, so this value never feeds RoPE. Also drop the
shadowed duplicate container get_query_offset (dead code — #8852's
definition wins at class creation).

Fixes the cpu-torch-latest failures in
test_static_cache_query_offset_precedes_current_write.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
The static-layer get_seq_length was restored to write_position + 1
(transformers >= 5.20 protocol, commit 39210bf), but three tests
still expected the old no-+1 convention. (flake8 skipped: sandbox)

Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
delock and others added 21 commits November 10, 2026 23:54
…oups

The CPU shm inference_all_reduce_ kernel is initialized with the WORLD
topology and takes no group argument, so it can only produce a correct
sum for a world-wide reduction. When a proper TP subgroup called
inference_all_reduce (autotp_size=2 inside a 4-rank world), the kernel
combined the two sibling subgroups' partial sums into a 4-way sum,
corrupting outputs and grad norms.

This is the cpu-torch-latest-v1-multirank CI failure: TestTpGradNorm
[2-1]/[2-2] failed deterministically (base_norm 1.6128 vs tp_norm
2.0052, bit-identical across runs) on avx512bw runners where the shm
kernel engages; avx2-only runners silently fell back to gloo and
passed, which made the check look flaky run-to-run.

Fix: engage the shm kernel only when the requested group spans the
whole world; a subgroup with full world size is necessarily the world
itself, so a size comparison suffices. Every other group falls through
to the group-aware torch.distributed.all_reduce. No other backend or
routing change: the accelerator custom-collective path is already bound
to its registered group per process and is untouched.

Add a regression test that reduces through two 2-rank subgroups of a
4-rank world and asserts subgroup-scoped results (1/5, not the
world-sum 6). With the guard removed the test reproduces the CI failure
exactly.

Validated on CPU (torch 2.12.1+cpu, python 3.11):
- unit/v1/autotp/test_autotp_training.py -k TestTpGradNorm:
  4 passed, 2 skipped (previously 2 failed, bit-identical to CI)
- unit/comm/test_custom_collective.py: 5 passed, 2 skipped
- unit/v1/autotp/test_tp_plan_e2e.py: test_tp_plan_basic_training passed
- full test_autotp_training.py: no regressions vs the unpatched head

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Conflict resolution:
- generate path: combine master's ZeRO-3 group-aware stopping
  criterion (_GlobalEosStoppingCriteria + synced_gpus opt-out) with our
  sampling kwargs (do_sample/temperature/top_p) in the eager fallback
- StaticCache construction: keep our TP-sharded config copy and eager
  init (graph capture needs pre-allocated buffers); master's simplified
  lazy-init call site only fits its non-TP default path
- GDN slot binding comment + prefill-shape inference comment both kept
- test_static_cache: keep both our GDN-protocol test and master's
  memory_bytes test (pure both-add)

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
The TP guard requires the full blocks (the injected forward slices
per-rank head ranges); the old column specs tripped it after the merge.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…kwargs

The fallback prefill branch of _continuous_prefill_cache passed
do_sample=False explicitly while also unpacking
**_hf_sampling_kwargs(sampling), whose return value always contains
do_sample -- so every execution of that branch raised
"GenerationMixin.generate() got multiple values for keyword argument
'do_sample'". The explicit kwarg is a leftover from the greedy-only era
(e53e2dc); dropping it lets _hf_sampling_kwargs own the value, which
also keeps first-token selection consistent with the static-cache
path's _token_selector(sampling).

Surfaced by TestAutoTPContinuousRollout::test_matches_generate_on_sharded_llama
[False] from #9008 after the 2026-11-13 master merge: it drives the
no-cache-class fallback branch that our own tests (static-cache models)
never reach.

Also collapse a stray blank line in tests/unit/utils/test_static_cache.py
flagged by the Formatting check (yapf).

Validated locally: yapf repo-clean, pre-commit green on changed files,
unit/comm and test_static_cache unchanged vs baseline; the rollout test
itself requires the CI transformers environment.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
All five conflicting files resolved to master's version -- it carries our
shm world-group guard (cherry-picked via #8997) plus the review hardening
(single-group contract in custom_collective, group-aware readiness,
settled() latch, rank-distinct test payloads) and the proper routing fix
for the custom-Function grad probe (ctx.needs_input_grad instead of the
always-False torch.is_grad_enabled()). This branch had no unique content
in these files.

Validated locally (CPU): unit/comm/test_custom_collective.py 5 passed /
2 skipped; TestTpGradNorm 4 passed / 2 skipped; tp_plan basic_training
passed.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
eos_mask.all() is a device->host sync on every decode step; it serializes
the ~50us of Python glue against the replay, and also runs when
eos_token_id is None where the mask can never become true. Check every
eos_check_every=16 steps instead, mirroring the b=1 path. Rows that hit
EOS keep accumulating in eos_mask, so the break lands on the first check
boundary after the last row's EOS, and tokens generated past a row's EOS
are padded by _pad_after_eos -- the output is unchanged.

Suggested by @jinyouzhi. Measured on L20 (2x for TP=2), Qwen3.5-4B,
b=32 greedy decode, interleaved A/B with medians: TP=2 1988 -> 1993
tok/s (+0.3%, non-overlapping ranges); TP=1 within noise (+0.1%). The
gain scales with the ratio of the Python glue to the step time, so it
matters most at small batch / faster GPUs where you profiled it.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
…egrade

Two hardening changes for silent-degradation paths found while validating
on L20:

1. RoPE table install passes [3, 1, L] positions when the model's rotary
   is MRoPE (mrope_section). transformers 5.20/5.22 happen to expand 2-D
   positions internally, but strict MRoPE implementations raise -- and
   the install chain shares one try, so a raise silently skips
   install_decode_attention / install_fused_norm / install_lm_head and
   decode falls back to the HF forward with no signal. Verified a no-op
   on 5.20.2/5.22.1 (install counts and b=32 decode throughput
   unchanged: TP1 1244.1, TP2 1992.6 tok/s vs 1242.8/1993.0 medians).

2. That except now logs a warning with the exception instead of
   swallowing it (review feedback: silent fallbacks must be visible).
   Verified by injecting a simulated failure: warning fires and the
   fallback engages (attn install 0/8, fused_norm 0).

3. Same treatment for the GDN scan: without fla-core every scan (decode
   and prefill) silently runs the eager torch decomposition, measured
   1.6x slower end-to-end on Qwen3.5-4B b=32 decode (1244 -> 645 tok/s).
   The per-layer FLA attribute is legitimately absent on transformers
   5.19+, so the warning probes fla importability instead. Verified both
   ways: fla present -> no warning, 1244.5 tok/s; fla removed ->
   warning + 644.6 tok/s.

Suggested by @jinyouzhi (MRoPE shape and the warning).

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
The amortized check (every 16 steps, from 8b05554) trades a per-step
~50us sync for up to 15 dead replays after the last row finishes. That
wins only in EOS-free decoding (+0.1-0.3% measured at b=32); in
EOS-heavy rollouts -- RL responses routinely end early and
max_new_tokens is only a cap -- the dead replays cost ~20% per call at
b=32 step times. Check every step again (the None fast path and the
accumulating-mask break structure stay).

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
The glu_epilogue fallback path (m > 32 decode and all prefill, where
M = batch * prompt_len) computed silu via an fp32 decomposition: two
upcasts of the [M, inter] gate/up tensors, five elementwise kernels,
and a final downcast. Replacing it with F.silu in bf16 eliminates the
cast round-trips and three kernels, and is bitwise-equal to HF's
activation (the fp32 formula rounds differently -- max 0.5 at bf16 --
so m<=32, m>32 and prefill previously disagreed with HF and each
other).

Measured on L20 (Ada), Qwen3.5-4B, b=64 greedy decode, interleaved A/B
(medians of 3): 1667.3 -> 1765.5 tok/s (+5.9%, non-overlapping
ranges). The native op keeps the small-M fused kernel (m <= 32); on
Ada its large-M path splits into row-chunked calls / an aten composite
with the same fp32 round-trip, so the eager fallback with this
epilogue is the fastest large-M implementation on this arch (1765 vs
1623 via the native dispatcher; Hopper keeps the concat-GEMM path).

test_segment_ki_kernels.py (the only oracle consumer; compares with
fp32 inputs under tolerance) passes: 25/25 on L20.

Numerics note: the fp32 single-rounding formula stays documented in
the docstring as the spec semantics; the executable path now matches
HF instead of exceeding its precision.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Eight conflict hunks in hybrid_engine_rollout.py, resolved semantically:
- sampling kwargs -> master's _generation_kwargs helper (it builds on our
  _hf_sampling_kwargs, so our do_sample/temperature semantics carry over)
- repetition penalty -> master's padding-aware upgrade (generation_config
  param, _repetition_penalty_ignores_padding probe, prompt_attention_mask
  arg); our old staticmethod copies removed to avoid silent double defs
- profiling gate -> keep our ungated enable_profiling stats semantics;
  the same profile object feeds the bucket planner via the new graph
  signature (adaptive_prefill gates the planner, not the stats)
- remaining hunks (signatures, prefill_cache kwargs) -> master verbatim

Validated on L20 (cards 1-2, torch 2.15/tf 5.22): rollout v1 6 failures
== the same pre-existing transformers-version set (+1 pass); module_inject
203 passed + 64 subtests, 0 failures; decode bench bit-identical
(1242.2 tok/s, same tok_sum); TestTpGradNorm 4 passed locally.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
HybridEngineRolloutConfig had use_segki and align_decode_fronts inserted
before enable_cache_trimming, shifting the legacy positional constructor
slots by two -- #8786's test_config_preserves_legacy_positional_arguments
caught it (16 silently landing on enable_cache_trimming). align_decode_fronts
is an upstream field and returns to its slot; use_segki moves to the end of
the dataclass so the positional contract keeps holding. All in-tree
construction is keyword-based, so the reorder is behavior-neutral.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Review finding (Runguo): the script hardcoded the deployment box's
checkout path; the installed deepspeed package is the import target.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Review findings from @runguo-li and @tohtana (all three verified):

1. refresh_weights() read self._rollout_module/self.module, which do not
   exist -- the rollout holds the model at engine.module. The arena e2e
   bypassed the method by calling segki_refresh() directly, so the entry
   point was never exercised. Resolve through the engine (mock-safe for
   the SimpleNamespace test engines), and state the retained-graph
   caveat honestly in the docstring (finding 2, design pending).

2. sync_weights() was a no-op: colocated weights are read live so there
   is nothing to push, but the injected forwards' derived caches go
   stale after an in-place write. The training-step hook now refreshes,
   so every sync source invalidates uniformly (tohtana's wiring point).

3. Rollout-path test: inject -> generate -> in-place weight write ->
   sync_weights -> generate must match a fresh injection bit-for-bit
   (the arena e2e logic through the public entry point). Replaces
   test_sync_weights_is_noop, which pinned the old no-op behavior.

Validated on L20 (card 4, torch 2.15/tf 5.22): both new tests pass;
full test_hybrid_engine_rollout.py 151 passed / 4 skipped.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
…im (a))

Runguo's finding 2: a retained continuous-batching graph captures the
derived tensors' addresses (TP>1 cat weights, Ampere/Hopper GLU concat
cache, GDN gates). refresh_weights drops those tensors, so a surviving
graph would replay freed memory. Parameters are covered by the capture
signature (in-place writes keep their addresses), but the derived
caches are invisible to it, so nothing else invalidates them.

Interim fix (a): refresh_weights releases _continuous_graph_state
(synchronized, like shutdown) before dropping caches; the next rollout
re-captures. Cost: one re-capture per sync. The end-state fix (b) --
refresh refills derived tensors in place so graphs survive by
construction -- is deferred by design; remove this release when it
lands.

The new test pins the contract on any config (asserts the graph is
released after sync, then post-sync generation matches a fresh
injection). Red on the previous head (the graph survived the sync);
config-dependent note: on Ada/dense/TP=1/small-batch the captured graph
holds no derived tensor, which is why the hazard never surfaced there.

Validated on L20 (card 4): red -> green; full rollout + module_inject
355 passed / 16 skipped / 0 failed.

Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
segki_refresh looked up the backend handle only through _ki_gdn_op, but
GLU-only models carry theirs on the MLP modules as _ki_dual_op — for
those, clear_glu_cat_cache() was never called and the process-wide
gate|up concat served stale values after an in-place sync (review
finding, Masahiro). Any handle reaches the process-wide cache; scan
both attributes. New arena test covers the dense-model branch with a
GLU-only model (DS_SEGMENT_KI_GLU_TEST_MODEL, default Qwen/Qwen3-0.6B).

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Design guide for segment-based kernel injection — the contracts
(opbuilder interface, kernel_reference, weight-shape-is-interface),
design decisions with rationale, agent extension guides, known
pitfalls, and the verification pyramid. Agents and humans modifying
segKI read this first; design-level changes ship with doc updates.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…ion policies

Major revisions from design review:
- accelerator-agnostic positioning; implementation language is the
  accelerator owner's choice (native/compiler/library/portable language),
  external components only in bring-your-own-pointers mode
- refresh_weights mechanism: two-tier (module callbacks + backend
  refresh_weights), two-tier end state (mandatory fallback + optional
  affine_targets direct-write)
- pre-fused layout rejection grounded in the opbuilder boundary; injection
  documented as model-agnostic structural discovery + config cross-check
- lineage chapter vs inference V1/V2, incl. declarative-manifest
  possibility (not a plan); agent guides gain adding-a-new-model;
  pitfalls/perf-gates/open-threads sections dropped as non-design

Signed-off-by: e <e@l>
Place segki_design.md under deepspeed/module_inject/ beside segment_ki.py
and kernel_reference.py so agents editing segKI see the design contract
in the same directory as the implementation.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T00:06:20.015631Z e26cc13 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@delock

delock commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

Closing in favor of a doc-only PR (#9130-style clean branch): the implementation commits showed up again in this PR's diff because segKI was squash-merged into master. The design doc now lands via a single doc-only commit on top of master.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e26cc13be7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +2012 to +2013
graph_op.decode_step_graph(static_logits[:, -1, :].contiguous(), static_token.view(batch_size, 1),
write_pos, static_attn, full_token_buf)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reset the mask mutated during graph capture

When the native op is available for a greedy BF16 graph rollout, decode_step_graph executes during capture and reveals prompt_len + 1 in static_attn. The post-capture restoration resets GDN state, write_pos, and static_token, but never clears that mask slot, so the first replay through an unpatched/native attention layer attends an unwritten future KV entry and can produce incorrect logits. Restore static_attn (and the associated output buffer state) after capture before starting the decode loop.

Useful? React with 👍 / 👎.

// (which holds the token being processed this step).
int64_t new_pos = write_pos[row * pos_stride] + 1;
if (pos_stride != 0 || row == 0) write_pos[row * pos_stride] = new_pos;
out_buf[(long)row * max_len + new_pos] = best;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Bounds-check the decode output write

When graph generation requests exactly one new token, max_len is prompt_len + 1 while capture invokes this kernel with write_pos == prompt_len; the kernel increments to new_pos == max_len and writes past out_buf before the existing mask bounds check. Because capture executes the kernel even though the subsequent decode loop has zero iterations, this can cause an illegal CUDA access or silent memory corruption. Guard the output write as well, or avoid mounting the step kernel when no decode step remains.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant