Repository navigation
Conversation
…native Metal kernels or llama.cpp's DFLASH graph)
…ote in docs/MAC.md
hotocoo
marked this pull request as ready for review
October 21, 2026 01:00
…e drafter repo name
…cpp build (-DSTRATA_DFLASH=ON) -DSTRATA_DFLASH=ON builds --dflash with Strata's own Metal drafter only: no libllama.dylib, no second llama.cpp build. -DSTRATA_DFLASH_LLAMA keeps llama.cpp's DFLASH graph as a second backend (STRATA_DFLASH_NATIVE=0) and the parity tests. Neither set: --dflash is refused, as before. Without --dflash nothing changes. Same drafts: a 320-draft run (inject 3 + draft 3, 600 rows of context) hashes the lattice and drafts bitwise equal on the native-only build and the llama.cpp build; dflash_native_parity_test passes.
…er knows; refusals say how to make the draft The native drafter took only an IQ4_XS token_embd (the type the tested target's shard carries); tools/dflash_head.py copies the target's own type, so other targets were refused. It now reads one row of any type ref::dequant handles. A draft GGUF without the draft-vocabulary head (d2t), or with more than 40960 rows, is refused with the command that makes one (tools/dflash_head.py --draft-vocab). docs/MAC.md says the same. Checked: same lattice and drafts (bitwise, 320 drafts) as before on the tuned drafter; the stock incoai drafter loads and drafts; dflash_native_parity_test passes.
… inject's fc on 32,2,4,2 (draft 11% faster) The drafter put every matrix with n_in >= 6144 (ffn_down 7680 x 2560, fc 12800 x 2560) on gemv tile 32,1,8,2. Measured per shape on an idle M4 Max (a 300-cycle inject(3) + draft(3) loop, min/p10/p50, interleaved with the old build, three rounds), ffn_down is fastest on the 16,2,4,1 default and fc on 32,2,4,2: inject(3) + draft(3), p50 4.47-4.50 ms -> 3.98-4.00 ms draft(3), p50 3.97-3.99 ms -> 3.51-3.53 ms fc's new tile sums in the same order (lattice and drafts bitwise equal). ffn_down's sums in another order, so the drafts are checked against the f64 reference instead: dflash_native_parity_test passes with the same numbers as before (lattice scores max 0.0041, mean 0.000685; 300 / 300 chains equal llama.cpp's). The target's verify window decides every token, so the output does not change either way.
…same binary, same day Short-prompt decode 50.4-50.8 tok/s (DFlash2 v2, spec 4, min-p 0.75) against MTP's 46.8-46.9 on an M4 Max, two runs each; 29K cold 52.9-53.3 against 49.5-50.2. Greedy output identical to MTP on 6 of 6 prompts (reasoning and answer). No row ran beside another engine (the script checks at every request).
…ember line did not compile) The merge of main (11a9add) kept both main's member line (with `mixr`) and this branch's (without it) around the --dflash feature buffer, so window.cpp no longer compiled on Metal. One line with `mixr`, then `F`. Checked: strata builds with -DSTRATA_DFLASH=ON, with -DSTRATA_DFLASH_LLAMA and with neither; the native drafter's lattice and drafts are bitwise equal to ee5e11b's (320-draft hash, stock and fine-tuned drafters); dflash_native_parity_test passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and result
On a Mac the only drafter for verify windows is the MTP layer (plus the suffix drafter). The engine cannot use a DFlash2 draft model; #3322 adds DFlash2 to the CUDA engine, and the Metal engine has nothing.
With this change,
--dflash <draft.gguf>(opt-in, Metal--serve) drafts in place of--mtp: up to--spec - 1tokens a window from the outputs of target layers 5/15/25/35/45 (the mean of the four hyper-connection streams), and--spec-min-pstops a draft where the selector's confidence falls below it. Without--dflashthe engine runs the same dispatches as before: every new dispatch and buffer is behind a flag only--dflashsets.On an M4 Max (64 GB) with an abliterated IQ3_S target, greedy, at a 42 GB expert arena:
The stock
incoai/Qwen3.8-Flash-Next-DFlash2drafter was slower than MTP on this target (41.1-42.5 vs 42.1-43.7 tok/s short, arena 33). The gain comes from a drafter fine-tuned on the target's own features, not from DFlash2 alone. Greedy output is identical to MTP.The MTP layer here is not the target's: the abliterated GGUF has no MTP tensors, so (as setup does for it) the MTP arm uses the
mtp.*block of the officialQwen/Qwen3.8-Flash-NextBF16 checkpoint (revision de4b8e4d), packed asmtp-q2_0.gguf(experts Q2_0). It was trained with the official model, not this fine-tune, and its experts are Q2_0 against the drafter's Q8_0; both favor DFlash2. This compares what Strata runs for this GGUF today with the fine-tuned drafter, not two drafters made the same way.Update 2026-10-22, on this revision (native-only build, ee5e11b drafter tiles, main merged; same binary for both
arms, same day, M4 Max 64 GB, arena 42 GB, greedy; bench/results/2026-10-22-metal-dflash2-native):
Greedy output identical to MTP on 6 of 6 prompts (reasoning and answer). The drafter is the incoai checkpoint fine-tuned
on this target (v2, Q8_0); the stock incoai drafter is still not faster than MTP on this abliterated target.
Related discussion
src/program/generate.cpp, where everything is behind#if defined(STRATA_GPU_METAL) && defined(STRATA_DFLASH).Approach
metal/kernels/gr.metal:hc_mean, one dispatch per captured layer, writes the HC-stream mean ofRinto a feature buffer (5 x n_embd per row).window.cppandprompt_chain.incencode it after the hyper-connection read of layers 5/15/25/35/45 in the decode/verify window, the resident prompt chain and the streaming prompt chunks, only whendf_on_is set.src/spec/dflash_drafter.cpp,include/strata/spec/dflash_*.hpp: the drafter. Two backends: Strata's own Metal kernels (metal/kernels/dflash.metal,metal/src/engine/dflash.cpp), the default; and, only in a build withSTRATA_DFLASH_LLAMA, llama.cpp's DFLASH graph (libllamafrom the pinned llama.cpp 3cf03257, built shared with Metal;STRATA_DFLASH_NATIVE=0), the parity reference. The native drafter takes anytoken_embdtype the host dequantizer knows and needs the draft-vocabulary head (tools/dflash_head.py --draft-vocab, at most 40960 rows); otherwise it refuses with that command. Each prompt chunk and committed window injects its rows' features; a draft runs the block and the selector.src/program/generate.cpp:--dflashparsing and the serve loop's draft source; it refuses--dflashwith--mtpand with--spec< 2.STRATA_DFLASH_DUMP=<prefix>(opt-in, with--dflash): appends each committed row's position, token and features to<prefix>.idx/<prefix>.f16. That is how the training data for the fine-tuned drafter was made; the training script is not in this PR.tools/dflash_head.py: makes the draft GGUF: a converted DFlash2 draft plus the target'stoken_embdand output head (Q8_0), optionally only the rows of the token ids in the MTP folder'sdraft_vocab.bin(a token-id list,data/draft_vocab_en.binin these runs; no MTP weights) with ad2tmap.metal/src/ref_quant.cpplearns GGUF type 27 (I64) for that map.metal/CMakeLists.txt:-DSTRATA_DFLASH=ONbuilds--dflashwith the native drafter only (no llama.cpp build needed).-DSTRATA_DFLASH_LLAMA=<llama.cpp shared Metal build>adds the llama.cpp backend and the two test programs. Neither set: nothing new is built and--dflashis refused.docs/MAC.md: a short--dflashparagraph.bench/results/2026-10-20-metal-dflash2/: the runs.Validation
Revision:
metal-dflash2(main 9ae93f7 + this change). The engine runs were measured on main e88ee39 + this change before two edits made for review: a per-window debug trace (STRATA_DFLASH_TRACE, off unless set) was removed, and the prompt path's token copy for the drafter now runs only with--dflash. Neither changes what runs with--dflash; the engine runs were not repeated on the final revision. The change applied to 9ae93f7 without conflicts. Apple M4 Max, 64 GB, macOS 26.5. llama.cpp 3cf03257 (the pinned commit), metal-cpp from setup.Update (main 9de04cd merged; commits 2aa30f0 and 8377413: native drafter by default, no llama.cpp build needed):
-DSTRATA_DFLASH=ON(otool -L strata: nolibllama),-DSTRATA_DFLASH_LLAMA=…and neither. All link.DflashDrafter) gives the same FNV-1a hash over every lattice and draft on the native-only build and on the llama.cpp build: the native drafter is unchanged by the refactor.dflash_native_parity_test t3000-q8-dv.gguf: ok. The stockdflash2-q8-dv.ggufloads and drafts on the native drafter.--dflashgets by default changed, and the matrix's DFlash2 arms already ran with the native one (STRATA_DFLASH_NATIVE=1).Update 2 (commit ee5e11b: drafter gemv tiles, measured on an idle GPU): ffn_down (7680 x 2560) moves to the
16,2,4,1 default tile and the inject's fc (12800 x 2560) to 32,2,4,2. A 300-cycle inject(3) + draft(3) loop, interleaved
with the previous build, three rounds: draft(3) p50 3.97-3.99 ms -> 3.51-3.53 ms, inject(3) + draft(3) p50 4.47-4.50 ms
-> 3.98-4.00 ms (M4 Max). fc's tile keeps the sums' order (bitwise equal); ffn_down's does not, so it was checked against
the f64 reference:
dflash_native_parity_testpasses with the same lattice error as before (max 0.0041, mean 0.000685)and 300 / 300 chains equal llama.cpp's. Engine runs on this revision: not run yet (the GPU is busy with another job).
Update 3 (82a83d9): main merged into the branch on GitHub left a duplicated member line in
window.cpp(PromptBufs) and the Metal build failed; fixed. All three CMake configurations build; the drafter's output is bitwise
equal to ee5e11b's;
dflash_native_parity_testpasses. The 2026-10-22 runs above used ee5e11b before this merge. After it, the same build (82a83d9,-DSTRATA_DFLASH=ON) servedthree more fresh-server runs of the same workload (DFlash2 v2, spec 4, min-p 0.75, arena 42): 49.8, 49.3, 49.8 tok/s
short decode, 51.7, 53.0, 52.4 tok/s on the 29K cold prompt, 2.12 tokens per window, in line with ee5e11b's.
Passed (first revision):
cmake -S metal -B build-df -G Ninja -DCMAKE_BUILD_TYPE=Release -DMETAL_CPP_DIR=… -DSTRATA_GGML_DIR=<llama.cpp> -DSTRATA_DFLASH_LLAMA=<llama.cpp shared Metal build>andninja -C build-df strata dflash_drafter_test dflash_native_parity_test: links; warnings are existing ones plus a duplicate-rpathfrom the linker.-DSTRATA_DFLASH_LLAMA(ninja -C build-off strata): links.build-df/dflash_drafter_test t3000-q8-dv.gguf: ok (steady draft 3.55 ms for 1 token, 9.73 ms for 7; another model server was running, so these are upper bounds).build-df/dflash_native_parity_test t3000-q8-dv.gguf: ok (native Metal drafter against an f64 host reference, 2200 context rows, 300 trials).bench/results/…/scripts/run.py): MTP on the unpatched binary against the fine-tuned DFlash2 drafter on the patched one: 6 of 6 identical, reasoning and answer, character for character (greedy-mtp.json,greedy-tuned.json).scripts/matrix*.py,matrix.tsv, 59 rows): arenas 33/38/40/42/45/47 GB, spec 3/4/5, severalspec_min_p, suffix drafter on/off, stock and fine-tuned drafters. 102K engine lines inlong-102k-engine-lines.txt.Failed:
matrix.tsv).Not run:
--dflashagainst unpatched main, output by output. The MTP arms ran on an unpatched build. The no-change claim for the default path rests on reading the code: every new dispatch is behinddf_on_, set only by--dflash.src/program/generate.cpp, whose additions are all inside#if defined(STRATA_GPU_METAL) && defined(STRATA_DFLASH), apart from the--dflashoption string, its usage text and its parsing.strata launchpaths.Limits
serve/server.py; setup does not offer it.