Repository navigation
Conversation
…ake, FLA in-place recurrent state Squashed from 20 commits of the A100 optimization session (net effect): - Ampere GLU routes to a concat-cached cuBLAS GEMM plus a native bf16 silu_mul kernel (the WMMA fused kernel stays for Ada and small M); +4% at bs<=32 on A100 TP2 - custom all-reduce gains a bounded publish-and-wait handshake so fabrics where the flag protocol does not propagate (PCIe P2P hosts) fall back to NCCL instead of hanging - GDN recurrent state streams from gmem in our in-place kernel (the 64KB smem staging only capped occupancy) — kept for the TP=1 path - FLA's Triton recurrent kernel is called directly with the cache slot as both h0 and ht: the wrapper's fresh allocation and the per-layer copy-back disappear, and decode moves off the pure-torch fp32 reference that transformers 5.x falls back to (+9-15% TP2; bypass is bitwise-identical to FLA's public API; FLA_BYPASS=0 restores the wrapper) - lab notes: INVESTIGATIONS.md items 1-3 and the FLA-bypass unraveling Baseline 963/2756/4255 -> 1042-1050/3108-3113/4859-4889 tok/s at bs 8/32/64 (2xA100 SXM4 TP2), 71/64/64% of vLLM 0.32. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…atch files out - bf16 WMMA kernel bodies guard on __CUDA_ARCH__>=800 (precompile builds every arch; sm75 compiles an empty shell that is never launched) - torch.distributed literals -> deepspeed.comm (segment_ki, layers); custom_ar.py marks its deliberately CUDA-specific calls #ignore-cuda; INVESTIGATIONS.md rewords the literal check-torchdist greps - padded-prompt guard reads module.config defensively: test doubles without one crashed the shared-prefill suite - session working files (experiments/, review notes, diff dumps) drop out of the branch — they tripped whitespace hooks and added review noise Verified on 2x GPU (AutoDL): module_inject + rollout suites 271 passed, 2 skipped, 0 failed. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
… markers, drop scratch notes Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…ight caches In-place weight writes (arena syncs, reshard materializations, training steps) keep weight addresses, so the injected forwards' derived caches (concat groups, sliced conv weights, fp32 gate constants, the backend's pointer-keyed GLU concat) would serve stale copies mixed with fresh direct reads. segki_refresh(model) — also exposed as HybridEngineRollout.refresh_weights() — drops them all; a fresh injection rebuilds the same state. Contract tests: after an in-place update + refresh, generation is BITWISE equal to a fresh injection over the same weights, including when parameter storage is bound to external arena tensors updated in place. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Two real AdamW steps on a ZeRO-3 engine (world=2); gathered weights written in place into the TP=2 rollout's sharded storage (split dim inferred from shape mismatch — the storage-map reshard contract); segki_refresh; generate must be bitwise equal to a fresh injection. Validated on 2x4080 SUPER with Qwen3.5-0.8B-Base: PASS both ranks, 0 token diffs. Opt-in via DS_RUN_ZERO3_E2E=1. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…dshake design Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…torch.cuda Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…nse headers Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…l custom_ar The one-shot P2P all-reduce now lives in its own PR with a cleaner shape: builder-based load behind the accelerator contract, lazy initialization inside the inference_all_reduce hook (no explicit init call at rollout construction), per-op readiness with a collective failure agreement, and a simplified RowParallel routing that delegates all eligibility checks to the hook. This PR keeps only the consumer side; the implementation carries over verbatim from #8997. - remove csrc/inference/custom_ar.cu + deepspeed/comm/custom_ar.py - torch.py/layers.py route through custom_collective (8997's text) - drop the now-dead RowParallel._rs_ag - e2e script needs no explicit init anymore (lazy in the hook) Signed-off-by: Guokai Ma <guokai.ma@intel.com>
silu_mul (the Ampere GLU epilogue) and repeat_qk (the scan head-repeat) gained kernel_reference entries and direct reference-matched tests — they had only indirect coverage through grouped_linear's matrices and the FLA end-to-end. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
The torch.library registration (segki_torchlib.py) and the opt-in torch.compile wrapper were a zero-value experiment: the GDN forward's Python-object dependencies broke the compile graph before any fusion, measured -2.3%. segKI's identity is hand-written, per-architecture tuned kernels behind the portable opbuilder interface; glue fusion via compile belongs to DeepCompile, which already uses the same custom-op pattern and can integrate segKI kernels when that work happens. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
The check catches real dataloader bugs in training; removing it was a workaround for broken NCCL infrastructure (specific RunPod instances), not a fix for a DeepSpeed defect. On healthy infrastructure the check passes normally. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Signed-off-by: Guokai Ma <guokai.ma@intel.com>
… opbuilder interface The internal GEMV/GEMM kernels (single/triple/quad_gemv, single/triple/ quad_gemm, dual_gemv_silu_mul) were exposed via m.def and called directly from segment_ki.py — a historical artifact from before grouped_linear existed as the unified dispatcher. Now: - segment_ki.py: all call sites go through grouped_linear (the flat/ gemv/wmma/composite routing is the dispatcher's job) - fused_glu.cu: 6 internal m.def entries removed; kernels remain as static host functions serving the dispatcher only - m.def count: 17 -> 10 (the portable interface + cache management) Other backends implementing the opbuilder contract see exactly the 10 methods they need; nothing else leaks through the boundary. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…tion The reference-consistency tests were calling the removed internal kernels directly; now all go through the portable interface. One test was asserting on a stale pre-allocated tensor instead of the actual kernel output (caught by the interface cleanup). Signed-off-by: Guokai Ma <guokai.ma@intel.com>
12 functions → 7: the 6 special-path references (dual_gemv_silu_mul, triple_gemv, triple_gemm, quad_gemm, dual_gemm_silu_mul, gdn_input_proj) collapse into a single grouped_linear(x, weights, glu_epilogue) that covers all routing paths. Other backends now read exactly 7 functions to understand the full contract. Test call sites updated accordingly; the segment_ki fallback paths route through the unified reference. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
… interface kernel_reference.py collapses to 7 interface-aligned functions; test and segment_ki fallback call sites updated to the unified grouped_linear entry; _as_concat helper normalizes the CUDA's per-path output format (n=4 flat concatenates, others return separate). Verified: 24/24 passed on GPU. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
q sliced from the fused q|k|v row block (fused[:, :qn]) keeps the fused row stride; the kernel stepped rows by num_q_heads*q_hs (packed layout), so at batch>1 every row after the first read k/v memory as the query. Pass the real row stride into the kernel and fail-closed on a gate whose row stride disagrees. Verified on L20 (2x rows, identical prompts): strided-vs-contiguous maxdiff 0.0000 both rows; b=1 e2e regression token-exact. Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
The raw-kernel launch skipped FLA's input_guard, so composite-path decode q/k/v (strided views of the fused projection, batch stride = full fused width) were read with packed-stride pointer math — every batch row after the first read k-region memory (o mismatch 0.29; L20, fla 0.6.0). The launch now packs its inputs (skip-if-contiguous), and a one-time parity probe against the wrapper on cloned live tensors gates the bypass per shape, silently rejecting FLA builds whose kernel grid contract differs (0.5.2 reads a 2-D grid). Verified b=2 eager/graph token-exact, b=1 e2e regression match; probe overhead is once per process and shape. Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
The native (non-segKI) GDN forward fetches conv/recurrent state through the cache's top-level get_conv_state/get_recurrent_state and branches on is_recording_past; only the hybrid slots' attribute forwarding existed, so use_graph_capture=True with use_segki=False (the default) crashed on hybrid models. Delegate the three entries to the bound slots (behavioral query answers False on KV layers); pre-5.19 attribute-style access was already served by the slot forwarding. Covered by a CPU protocol unit test and a GPU graph-without-segKI parity test (token-exact vs generate); found by the decode benchmark baseline, which went from crash to a full batch sweep. (commit hook flake8 skipped: sandbox blocks its multiprocessing semaphores; verified standalone on the same files) Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
Pick #8997's final state for the four files that had drifted: kernel link-safety + clang-format (csrc), the CPU-accelerator route exclusion (torch.py), RowParallel's inference_all_reduce-owned backend selection (layers.py), and the restructured op builder. The Python wrapper and the parity test were already identical. (commit-hook flake8 skipped: sandbox blocks its multiprocessing semaphores; each file passes standalone) Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…te replication Under AutoTP a custom layer_specs list REPLACES the built-ins; partial configs left the GDN input-side weights accidentally replicated while the TP forward's endpoint arithmetic assumed sharded widths — the a slice came out 2096 rows wide (expected 16) and gdn_gates read past the buffer (L20, Qwen3.5-4B, GVA v/k=2). The TP branch now slices this rank's head range out of the replicated z/b/a blocks alongside qkv and conv1d, and installation fails loudly when any GDN input projection is found sharded (the contract: replicated in_proj_*/conv1d, row-parallel out_proj, standard column/row for attn+MLP). Verified TP=2 b=1 vs TP=1: token-exact over 32 generated tokens; 143.6 tok/s decode (vs vLLM 0.32 TP2 141.4 on the same L20). Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
Master's #8939 adds _token_selector + _repetition_penalty + _apply_repetition_penalty using HF's official warpers and matching generate()'s distribution exactly. It subsumes our hand-rolled _sample_from_logits (18 lines, no repetition penalty). All segKI integration preserved (graph capture, GDN handling, decode attention install); only the sampling mechanism swapped. (commit-hook flake8 skipped: sandbox blocks its multiprocessing semaphores; the file passes standalone) Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
The merge with #8939 left two copies of get_query_offset on DeepSpeedStaticCache (ours from the pre-merge tree and master's from #8852); keep master's richer per-layer version. The test's DSStaticGDNSlot import became unused when the assertion moved to cache-level methods. (flake8 skipped: sandbox semaphores) Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
The merge with #8939 adopted _token_selector for the b=1 full-step graph path but two auto-merged regions kept the old hardcoded argmax: the first token after prefill and every step of the b>1 graph+Python loop. With temperature>0, all 200 sampled first tokens were identical (the argmax) while HF generate spread over 26 tokens — the distributions diverged from step 0. Both sites now use select_next_tokens, matching master's #8939 behavior. (flake8 skipped: sandbox semaphores) Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…pSpeedStaticLayer get_seq_length() counts the slot the current query is writing (one past the write position) per the upstream #8852 protocol; position and mask math goes through get_query_offset(), which returns the write position itself. Our decode paths are unaffected: they pass cache_position and position_ids explicitly, so this value never feeds RoPE. Also drop the shadowed duplicate container get_query_offset (dead code — #8852's definition wins at class creation). Fixes the cpu-torch-latest failures in test_static_cache_query_offset_precedes_current_write. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
The static-layer get_seq_length was restored to write_position + 1 (transformers >= 5.20 protocol, commit 39210bf), but three tests still expected the old no-+1 convention. (flake8 skipped: sandbox) Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
…oups The CPU shm inference_all_reduce_ kernel is initialized with the WORLD topology and takes no group argument, so it can only produce a correct sum for a world-wide reduction. When a proper TP subgroup called inference_all_reduce (autotp_size=2 inside a 4-rank world), the kernel combined the two sibling subgroups' partial sums into a 4-way sum, corrupting outputs and grad norms. This is the cpu-torch-latest-v1-multirank CI failure: TestTpGradNorm [2-1]/[2-2] failed deterministically (base_norm 1.6128 vs tp_norm 2.0052, bit-identical across runs) on avx512bw runners where the shm kernel engages; avx2-only runners silently fell back to gloo and passed, which made the check look flaky run-to-run. Fix: engage the shm kernel only when the requested group spans the whole world; a subgroup with full world size is necessarily the world itself, so a size comparison suffices. Every other group falls through to the group-aware torch.distributed.all_reduce. No other backend or routing change: the accelerator custom-collective path is already bound to its registered group per process and is untouched. Add a regression test that reduces through two 2-rank subgroups of a 4-rank world and asserts subgroup-scoped results (1/5, not the world-sum 6). With the guard removed the test reproduces the CI failure exactly. Validated on CPU (torch 2.12.1+cpu, python 3.11): - unit/v1/autotp/test_autotp_training.py -k TestTpGradNorm: 4 passed, 2 skipped (previously 2 failed, bit-identical to CI) - unit/comm/test_custom_collective.py: 5 passed, 2 skipped - unit/v1/autotp/test_tp_plan_e2e.py: test_tp_plan_basic_training passed - full test_autotp_training.py: no regressions vs the unpatched head Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Conflict resolution: - generate path: combine master's ZeRO-3 group-aware stopping criterion (_GlobalEosStoppingCriteria + synced_gpus opt-out) with our sampling kwargs (do_sample/temperature/top_p) in the eager fallback - StaticCache construction: keep our TP-sharded config copy and eager init (graph capture needs pre-allocated buffers); master's simplified lazy-init call site only fits its non-TP default path - GDN slot binding comment + prefill-shape inference comment both kept - test_static_cache: keep both our GDN-protocol test and master's memory_bytes test (pure both-add) Signed-off-by: Guokai Ma <guokai.ma@intel.com>
The TP guard requires the full blocks (the injected forward slices per-rank head ranges); the old column specs tripped it after the merge. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…kwargs The fallback prefill branch of _continuous_prefill_cache passed do_sample=False explicitly while also unpacking **_hf_sampling_kwargs(sampling), whose return value always contains do_sample -- so every execution of that branch raised "GenerationMixin.generate() got multiple values for keyword argument 'do_sample'". The explicit kwarg is a leftover from the greedy-only era (e53e2dc); dropping it lets _hf_sampling_kwargs own the value, which also keeps first-token selection consistent with the static-cache path's _token_selector(sampling). Surfaced by TestAutoTPContinuousRollout::test_matches_generate_on_sharded_llama [False] from #9008 after the 2026-11-13 master merge: it drives the no-cache-class fallback branch that our own tests (static-cache models) never reach. Also collapse a stray blank line in tests/unit/utils/test_static_cache.py flagged by the Formatting check (yapf). Validated locally: yapf repo-clean, pre-commit green on changed files, unit/comm and test_static_cache unchanged vs baseline; the rollout test itself requires the CI transformers environment. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
All five conflicting files resolved to master's version -- it carries our shm world-group guard (cherry-picked via #8997) plus the review hardening (single-group contract in custom_collective, group-aware readiness, settled() latch, rank-distinct test payloads) and the proper routing fix for the custom-Function grad probe (ctx.needs_input_grad instead of the always-False torch.is_grad_enabled()). This branch had no unique content in these files. Validated locally (CPU): unit/comm/test_custom_collective.py 5 passed / 2 skipped; TestTpGradNorm 4 passed / 2 skipped; tp_plan basic_training passed. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
eos_mask.all() is a device->host sync on every decode step; it serializes the ~50us of Python glue against the replay, and also runs when eos_token_id is None where the mask can never become true. Check every eos_check_every=16 steps instead, mirroring the b=1 path. Rows that hit EOS keep accumulating in eos_mask, so the break lands on the first check boundary after the last row's EOS, and tokens generated past a row's EOS are padded by _pad_after_eos -- the output is unchanged. Suggested by @jinyouzhi. Measured on L20 (2x for TP=2), Qwen3.5-4B, b=32 greedy decode, interleaved A/B with medians: TP=2 1988 -> 1993 tok/s (+0.3%, non-overlapping ranges); TP=1 within noise (+0.1%). The gain scales with the ratio of the Python glue to the step time, so it matters most at small batch / faster GPUs where you profiled it. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
…egrade Two hardening changes for silent-degradation paths found while validating on L20: 1. RoPE table install passes [3, 1, L] positions when the model's rotary is MRoPE (mrope_section). transformers 5.20/5.22 happen to expand 2-D positions internally, but strict MRoPE implementations raise -- and the install chain shares one try, so a raise silently skips install_decode_attention / install_fused_norm / install_lm_head and decode falls back to the HF forward with no signal. Verified a no-op on 5.20.2/5.22.1 (install counts and b=32 decode throughput unchanged: TP1 1244.1, TP2 1992.6 tok/s vs 1242.8/1993.0 medians). 2. That except now logs a warning with the exception instead of swallowing it (review feedback: silent fallbacks must be visible). Verified by injecting a simulated failure: warning fires and the fallback engages (attn install 0/8, fused_norm 0). 3. Same treatment for the GDN scan: without fla-core every scan (decode and prefill) silently runs the eager torch decomposition, measured 1.6x slower end-to-end on Qwen3.5-4B b=32 decode (1244 -> 645 tok/s). The per-layer FLA attribute is legitimately absent on transformers 5.19+, so the warning probes fla importability instead. Verified both ways: fla present -> no warning, 1244.5 tok/s; fla removed -> warning + 644.6 tok/s. Suggested by @jinyouzhi (MRoPE shape and the warning). Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
The amortized check (every 16 steps, from 8b05554) trades a per-step ~50us sync for up to 15 dead replays after the last row finishes. That wins only in EOS-free decoding (+0.1-0.3% measured at b=32); in EOS-heavy rollouts -- RL responses routinely end early and max_new_tokens is only a cap -- the dead replays cost ~20% per call at b=32 step times. Check every step again (the None fast path and the accumulating-mask break structure stay). Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
The glu_epilogue fallback path (m > 32 decode and all prefill, where M = batch * prompt_len) computed silu via an fp32 decomposition: two upcasts of the [M, inter] gate/up tensors, five elementwise kernels, and a final downcast. Replacing it with F.silu in bf16 eliminates the cast round-trips and three kernels, and is bitwise-equal to HF's activation (the fp32 formula rounds differently -- max 0.5 at bf16 -- so m<=32, m>32 and prefill previously disagreed with HF and each other). Measured on L20 (Ada), Qwen3.5-4B, b=64 greedy decode, interleaved A/B (medians of 3): 1667.3 -> 1765.5 tok/s (+5.9%, non-overlapping ranges). The native op keeps the small-M fused kernel (m <= 32); on Ada its large-M path splits into row-chunked calls / an aten composite with the same fp32 round-trip, so the eager fallback with this epilogue is the fastest large-M implementation on this arch (1765 vs 1623 via the native dispatcher; Hopper keeps the concat-GEMM path). test_segment_ki_kernels.py (the only oracle consumer; compares with fp32 inputs under tolerance) passes: 25/25 on L20. Numerics note: the fp32 single-rounding formula stays documented in the docstring as the spec semantics; the executable path now matches HF instead of exceeding its precision. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Eight conflict hunks in hybrid_engine_rollout.py, resolved semantically: - sampling kwargs -> master's _generation_kwargs helper (it builds on our _hf_sampling_kwargs, so our do_sample/temperature semantics carry over) - repetition penalty -> master's padding-aware upgrade (generation_config param, _repetition_penalty_ignores_padding probe, prompt_attention_mask arg); our old staticmethod copies removed to avoid silent double defs - profiling gate -> keep our ungated enable_profiling stats semantics; the same profile object feeds the bucket planner via the new graph signature (adaptive_prefill gates the planner, not the stats) - remaining hunks (signatures, prefill_cache kwargs) -> master verbatim Validated on L20 (cards 1-2, torch 2.15/tf 5.22): rollout v1 6 failures == the same pre-existing transformers-version set (+1 pass); module_inject 203 passed + 64 subtests, 0 failures; decode bench bit-identical (1242.2 tok/s, same tok_sum); TestTpGradNorm 4 passed locally. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
HybridEngineRolloutConfig had use_segki and align_decode_fronts inserted before enable_cache_trimming, shifting the legacy positional constructor slots by two -- #8786's test_config_preserves_legacy_positional_arguments caught it (16 silently landing on enable_cache_trimming). align_decode_fronts is an upstream field and returns to its slot; use_segki moves to the end of the dataclass so the positional contract keeps holding. All in-tree construction is keyword-based, so the reorder is behavior-neutral. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Review finding (Runguo): the script hardcoded the deployment box's checkout path; the installed deepspeed package is the import target. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Review findings from @runguo-li and @tohtana (all three verified): 1. refresh_weights() read self._rollout_module/self.module, which do not exist -- the rollout holds the model at engine.module. The arena e2e bypassed the method by calling segki_refresh() directly, so the entry point was never exercised. Resolve through the engine (mock-safe for the SimpleNamespace test engines), and state the retained-graph caveat honestly in the docstring (finding 2, design pending). 2. sync_weights() was a no-op: colocated weights are read live so there is nothing to push, but the injected forwards' derived caches go stale after an in-place write. The training-step hook now refreshes, so every sync source invalidates uniformly (tohtana's wiring point). 3. Rollout-path test: inject -> generate -> in-place weight write -> sync_weights -> generate must match a fresh injection bit-for-bit (the arena e2e logic through the public entry point). Replaces test_sync_weights_is_noop, which pinned the old no-op behavior. Validated on L20 (card 4, torch 2.15/tf 5.22): both new tests pass; full test_hybrid_engine_rollout.py 151 passed / 4 skipped. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
…im (a)) Runguo's finding 2: a retained continuous-batching graph captures the derived tensors' addresses (TP>1 cat weights, Ampere/Hopper GLU concat cache, GDN gates). refresh_weights drops those tensors, so a surviving graph would replay freed memory. Parameters are covered by the capture signature (in-place writes keep their addresses), but the derived caches are invisible to it, so nothing else invalidates them. Interim fix (a): refresh_weights releases _continuous_graph_state (synchronized, like shutdown) before dropping caches; the next rollout re-captures. Cost: one re-capture per sync. The end-state fix (b) -- refresh refills derived tensors in place so graphs survive by construction -- is deferred by design; remove this release when it lands. The new test pins the contract on any config (asserts the graph is released after sync, then post-sync generation matches a fresh injection). Red on the previous head (the graph survived the sync); config-dependent note: on Ada/dense/TP=1/small-batch the captured graph holds no derived tensor, which is why the hazard never surfaced there. Validated on L20 (card 4): red -> green; full rollout + module_inject 355 passed / 16 skipped / 0 failed. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
segki_refresh looked up the backend handle only through _ki_gdn_op, but GLU-only models carry theirs on the MLP modules as _ki_dual_op — for those, clear_glu_cat_cache() was never called and the process-wide gate|up concat served stale values after an in-place sync (review finding, Masahiro). Any handle reaches the process-wide cache; scan both attributes. New arena test covers the dense-model branch with a GLU-only model (DS_SEGMENT_KI_GLU_TEST_MODEL, default Qwen/Qwen3-0.6B). Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Design guide for segment-based kernel injection — the contracts (opbuilder interface, kernel_reference, weight-shape-is-interface), design decisions with rationale, agent extension guides, known pitfalls, and the verification pyramid. Agents and humans modifying segKI read this first; design-level changes ship with doc updates. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
…ion policies Major revisions from design review: - accelerator-agnostic positioning; implementation language is the accelerator owner's choice (native/compiler/library/portable language), external components only in bring-your-own-pointers mode - refresh_weights mechanism: two-tier (module callbacks + backend refresh_weights), two-tier end state (mandatory fallback + optional affine_targets direct-write) - pre-fused layout rejection grounded in the opbuilder boundary; injection documented as model-agnostic structural discovery + config cross-check - lineage chapter vs inference V1/V2, incl. declarative-manifest possibility (not a plan); agent guides gain adding-a-new-model; pitfalls/perf-gates/open-threads sections dropped as non-design Signed-off-by: e <e@l>
Place segki_design.md under deepspeed/module_inject/ beside segment_ki.py and kernel_reference.py so agents editing segKI see the design contract in the same directory as the implementation. Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Closing in favor of a doc-only PR (#9130-style clean branch): the implementation commits showed up again in this PR's diff because segKI was squash-merged into master. The design doc now lands via a single doc-only commit on top of master. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e26cc13be7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| graph_op.decode_step_graph(static_logits[:, -1, :].contiguous(), static_token.view(batch_size, 1), | ||
| write_pos, static_attn, full_token_buf) |
There was a problem hiding this comment.
Reset the mask mutated during graph capture
When the native op is available for a greedy BF16 graph rollout, decode_step_graph executes during capture and reveals prompt_len + 1 in static_attn. The post-capture restoration resets GDN state, write_pos, and static_token, but never clears that mask slot, so the first replay through an unpatched/native attention layer attends an unwritten future KV entry and can produce incorrect logits. Restore static_attn (and the associated output buffer state) after capture before starting the decode loop.
Useful? React with 👍 / 👎.
| // (which holds the token being processed this step). | ||
| int64_t new_pos = write_pos[row * pos_stride] + 1; | ||
| if (pos_stride != 0 || row == 0) write_pos[row * pos_stride] = new_pos; | ||
| out_buf[(long)row * max_len + new_pos] = best; |
There was a problem hiding this comment.
Bounds-check the decode output write
When graph generation requests exactly one new token, max_len is prompt_len + 1 while capture invokes this kernel with write_pos == prompt_len; the kernel increments to new_pos == max_len and writes past out_buf before the existing mask bounds check. Because capture executes the kernel even though the subsequent decode loop has zero iterations, this can cause an illegal CUDA access or silent memory corruption. Guard the output write as well, or avoid mounting the step kernel when no decode step remains.
Useful? React with 👍 / 👎.
Summary
Adds the segKI design document at
deepspeed/module_inject/segki_design.md, next to its implementation (segment_ki.py,kernel_reference.py).The segKI implementation itself has already merged; per the document's own rule — the implementation must stay in sync with it — this lands the accompanying design guide so agents and humans modifying segKI can read the contracts (opbuilder boundary, kernel_reference executable spec, refresh_weights, lineage and extension policies) before making changes.
The document was previously drafted on the
gma/segki-designbranch; this PR is its submission after PR #8602 merged, as noted in the doc header.