[Engine] Metal: opt-in --dflash DFlash2 drafter in place of the MTP layer

bench/results/2026-10-20-metal-dflash2/README.md

@@ -0,0 +1,95 @@
+# Metal: DFlash2 drafter (--dflash) against the MTP layer, Apple M4 Max
+
+Measured on 2026-10-20 by hotocoo. One Mac, one target model (an abliterated IQ3_S fine-tune of Qwen3.8-Flash-Next),
+greedy decoding. Main limitation: most cells are one run each; the leaders were repeated once (`-r` rows).
+
+## Hardware and software
+
+- Apple M4 Max, 64 GB unified memory, internal SSD. macOS 26.5.
+- Strata source build: main e88ee39d plus this change, Metal engine (`cmake -S metal -G Ninja -DCMAKE_BUILD_TYPE=Release
+  -DMETAL_CPP_DIR=… -DSTRATA_GGML_DIR=<llama.cpp 3cf03257>`), with `-DSTRATA_DFLASH_LLAMA=<llama.cpp 3cf03257 Metal
+  shared build>` for the DFlash2 build.
+- MTP arms: `build-metal/strata` without this change (the binary holds no `dflash` symbol or string).
+  DFlash2 arms: `build-metal-dflash/strata`, built from the submitted source.
+
+## Model and configuration
+
+- Target: `DavidAU/Qwen3.8-Flash-Next-Abliterated-Uncensored-Turbo-Brilliance-50x-Reasoning-Instruct-GSQ-RCO-GGUF`,
+  revision b7f9593a, `Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_S-0000{1,2}-of-00002.gguf`, mmproj BF16 (vision on).
+- MTP layer (the baseline): not part of the target. DavidAU's GGUF has no MTP tensors (0 of 1224). It is the `mtp.*`
+  block (31 tensors) of the official `Qwen/Qwen3.8-Flash-Next` BF16 checkpoint, revision de4b8e4d, extracted by Strata's
+  setup and packed by `tools/mtp_pack.py` as `mtp-q2_0.gguf` (experts Q2_0, the rest BF16/F32;
+  sha256 a22206a4d74eafbf31063a84610292535fb3b49af9a339963817fd04aed81e4d). It was trained on the official model, not on
+  this abliterated fine-tune, and its experts are Q2_0 while the DFlash2 drafter is Q8_0: both favor DFlash2. The
+  comparison is what Strata runs for this GGUF today against this drafter, not two drafters made the same way.
+- Draft vocabulary (both arms): Strata's `data/draft_vocab_en.bin` (40525 token ids, English/code subset, sha256
+  369151522226a5edaa5f12cfd1e2ae7db8f4fbdbd222f3dcf327dced9597fb25). A list of token ids only; tokens outside it are never
+  drafted.
+- Drafters:
+  - stock: `incoai/Qwen3.8-Flash-Next-DFlash2` converted with llama.cpp's `convert_hf_to_gguf.py`, then
+    `tools/dflash_head.py --draft-vocab` (Q8_0, the target's own output-head rows for the draft-vocabulary token ids).
+    `dflash2-q8-dv.gguf`, sha256 32cd18db68465301d3281dd9a9e5a9e50f97ee070e5fd40683b1e5e822797312.
+  - tuned: the stock drafter fine-tuned for 3000 steps (batch 32) on this target's own captured features (layers
+    5/15/25/35/45, HC mean) and greedy continuations. `t3000-q8-dv.gguf`,
+    sha256 e5412a73583755eea8daa4c4c5793f5b827af1fbc0e76bcdd5cc181cdbf46eb1. Published as
+    `hotocooo/Qwen3.8-Flash-Next-Abliterated-Uncensored-Turbo-Brilliance-50x-Reasoning-Instruct-GSQ-RCO-DFlash`.
+    Held-out drafts accepted per 7-token block: 3.568 (stock) to 3.796 (tuned).
+- Both arms: context 262144, `--mmap-experts`, `STRATA_METAL_STREAM=1`, `STRATA_METAL_LEAVE_SHARE=0.3`, prompt
+  cache 12, prefill auto, `--suffix-draft 0` unless the name ends in `-sfx`, no reasoning limit, temperature 0.
+- `STRATA_METAL_ARENA_GB` (the expert arena) is the `aNN` in each name; `sN` is `--spec`.
+
+```text
+MTP:     strata --pack <pack> --native <IQ3_S 1 of 2> --ple-gguf <IQ3_S 2 of 2> --expert-profile data/expert-profile.bin
+         --expert-cache 64 --prefill auto --spec 3 --suffix-draft 0 --max-context 262144 --mmap-experts --vision
+         --prompt-cache 12 --prompt-cache-spread --mtp <mtp rt> --spec-min-p 0.7
+DFlash2: the same, with --spec 4 --dflash t3000-q8-dv.gguf --spec-min-p 0.75 in place of --mtp, and STRATA_DFLASH_NATIVE=1
+```
+
+## Method
+
+Scripts in `scripts/` (as run; local paths replaced by `<strata>` and `<data>`). `swap.sh` stops the server and starts
+the next config on a fresh process, so each config starts with a cold expert cache.
+
+- `matrix*.py`: for each config and each `spec_min_p` (sent per request as `strata_tune`): three short prompts
+  (600 output tokens each), then a ~29K-token code prompt (first 90,000 characters of
+  `src/core/mtp.cpp`) and one follow-up on the same prefix (prompt cache). Decode tok/s is generated tokens over the
+  engine's generation time from its `generated in` line (prompt reading excluded). `tok/window` is tokens per verify
+  window on the short prompts. The `swap` column is macOS swap used before and after the config.
+- `long120.py`: a 101,872-token prompt (`verify.cpp` + `mtp.cpp`), then a follow-up reusing 101,867 tokens. One run
+  per arm; engine lines in `long-102k-engine-lines.txt`.
+- `run.py`: six fixed prompts, greedy, 700 output tokens: the outputs of MTP and DFlash2 are compared as text.
+
+## Results
+
+The full matrix is `matrix.tsv` (every row, including the paging runs). Best settings at the 42 GB arena:
+
+| Configuration | Runs | Short decode tok/s | Tokens/window | 29K cold tok/s | 29K cached tok/s |
+| --- | ---: | --- | ---: | --- | --- |
+| MTP spec 3, min-p 0.7 | 2 | 48.1, 46.9 | 1.71 | 50.4, 50.0 | 50.4, 50.1 |
+| DFlash2 tuned spec 4, min-p 0.75 | 2 | 49.7, 49.7 | 2.11 | 51.0, 51.7 | 51.7, 51.8 |
+| DFlash2 tuned spec 4, min-p 0.6 | 2 | 50.4, 49.6 | 2.35 | 51.9, 50.8 | 51.4, 50.3 |
+
+101,872-token prompt, one run each:
+
+| Configuration | Prompt read, s (tok/s) | Decode cold tok/s | Cached follow-up decode tok/s |
+| --- | --- | --- | --- |
+| MTP spec 3, min-p 0.7 | 268.8 (379.1) | 46.8 | 44.4 |
+| DFlash2 tuned spec 4, min-p 0.75 | 276.9 (367.9) | 50.5 | 45.1 |
+
+Other observations in `matrix.tsv`:
+
+- The **stock** incoai drafter (`dfbase-s4-a33`) is slower than MTP on short prompts (41.1-42.5 vs 42.1-43.7 tok/s at
+  arena 33). The gain comes from the fine-tune on this target, not from DFlash2 alone.
+- At arena 33 the two are within noise (MTP 42.1-43.7, tuned DFlash2 41.9-45.4 short).
+- Arena 45 and 47 page on this 64 GB Mac: short decode falls to 8.8-41.1 tok/s for both drafters.
+- Draft cost: the native Metal drafter takes 3.55 ms (1 token) to 9.73 ms (7 tokens) per draft
+  (`dflash_drafter_test`, measured while another model server ran, so an upper bound).
+
+## Correctness and limitations
+
+- `run.py`, greedy: MTP (`greedy-mtp.json`) and DFlash2 tuned (`greedy-tuned.json`) are identical, reasoning and
+  answer, character for character, on 6 of 6 prompts.
+- `dflash_native_parity_test`: the native Metal drafter against an f64 host reference: passed.
+- Not measured: sampling (temperature > 0) acceptance and quality, time to first token, peak memory beyond swap,
+  other Macs, the official Qwen weights or other quantizations, concurrent requests.
+- Few repetitions: differences under about 2 tok/s on short prompts are within the spread seen between repeats.

bench/results/2026-10-20-metal-dflash2/greedy-mtp.json

@@ -0,0 +1,32 @@
+[
+ {
+  "role": "assistant",
+  "content": null,
+  "reasoning_content": "We need answer user: \"Write a Python function that parses an ISO 8601 duration string like P3DT4H12M into seconds, with tests.\" Need provide code and tests. Need think through implementation. Need likely include function parse_iso8601_duration_to_seconds(s: str) -> float/int. ISO 8601 duration format: P[n]Y[n]M[n]D[T[n]H[n]M[n]S]. Need parse into seconds. Ambiguities: years/months variable length. Could define 1 year = 365 days, 1 month = 30 days? Or use datetime? Need mention assumptions. Could support fractional seconds, negative durations? ISO 8601 allows negative sign before P? e.g. -P3D. Also weeks? ISO 8601 duration can include W: P3W. Need parse weeks too. Need tests. Need robust regex.\n\nNeed decide output type. Seconds maybe float if fractional seconds. Could return float. If no fractional, int? Python float okay. Could return float. Or int if exact? Simpler float.\n\nNeed parse: optional leading sign (+/-), P, date part, optional T, time part. Date components: Y, M, W, D. Time: H, M, S. M before T months, after T minutes. Need ensure at least one component. Need reject empty, invalid order, missing T if time components without T? ISO requires T before time. Could allow? Better strict: if H/M/S present, T required. Also no duplicate components. Need handle fractional seconds: e.g. PT1.5S, PT0.000001S. Also fractional on other components? ISO allows decimal fraction on smallest component? Usually only last component can have fraction? But can parse any numeric with decimal. Need convert years/months/weeks/days/hours/minutes/seconds. Assumptions: year=365 days, month=30 days, week=7 days, day=86400, hour=3600, minute=60. Could mention if calendar-aware needed use dateutil or relativedelta.\n\nNeed tests: use unittest or pytest. Provide function and tests. Could include examples: P3DT4H12M -> 3*86400 + 4*3600 + 12*60 = 259200+14400+720=274320. P1Y2M3W4DT5H6M7.5S -> (365+60+21+4)*86400 + 5*3600+6*60+7.5 = 450*86400? 365+60=425, +21=446, +4=450 => 38,880,000 + 18,000+360+7.5=38,898,367.5. Negative -PT1H -> -3600. +PT1H -> 3600. PT0S ->"
+ },
+ {
+  "role": "assistant",
+  "content": "Step 1: Time for the first part  \n\\[\n\\text{Time}=\\frac{\\text{Distance}}{\\text{Speed}}=\\frac{237}{79}=3\\text{ hours}\n\\]\n\nStep 2: Time for the second part  \n\\[\n\\text{Time}=\\frac{120}{60}=2\\text{ hours}\n\\]\n\nStep 3: Total travel time  \n\\[\n3+2=5\\text{ hours}\n\\]\n\nStep 4: Add to departure time  \n\\[\n9:40 + 5\\text{ hours} = 14:40\n\\]\n\nThe train arrives at **14:40**, or **2:40 PM**.",
+  "reasoning_content": "We need answer user: train leaves 9:40, travels 237 km at 79 km/h, then 120 km at 60 km/h. When arrives? Show steps. Need compute times. 237/79 = 3 hours exactly? 79*3=237 yes. 120/60=2 hours. Total 5 hours. 9:40 + 5h = 14:40 (2:40 PM). Need show steps. Ensure no mistakes. Could mention 24-hour and 12-hour. Final concise.\n"
+ },
+ {
+  "role": "assistant",
+  "content": null,
+  "reasoning_content": "We need answer user: \"Explain how a B-tree insert splits nodes, then give a short C++ implementation of insert for order 4.\"\n\nNeed produce final with explanation and C++ implementation. Need think carefully. B-tree order 4: max children 4, max keys 3, min children ceil(4/2)=2 except root, min keys 1. Insert splits full node (3 keys) into two nodes with 1 key each? For order 4, node full has 3 keys and 4 children. Splitting: median key (second key) moves to parent, left gets first key, right gets third key. If root splits, new root with median. Need implementation short. Could implement recursive insert returning split info. Need handle internal nodes with children. Need maybe use vector keys, children. Need order 4 constants. Need ensure correct for leaf/internal.\n\nNeed decide implementation style. Could define struct BTreeNode { vector<int> keys; vector<BTreeNode*> children; bool leaf; }. Insert function: bool insert(BTreeNode*& root, int key). If root null create leaf. Recursive insert returns pair<bool, int> maybe split occurred and median key, and right child pointer. Simpler: struct SplitResult { bool split; int median; BTreeNode* right; };\n\ninsert(BTreeNode* node, int key) -> SplitResult. If leaf: insert key sorted. If size <= maxKeys return no split. Else split leaf: median = keys[2]? For order 4 maxKeys=3. If keys size 4 after insertion. Need split into left keys[0], right keys[3], median keys[1]? Wait for B-tree order 4, full node before insert has 3 keys. Inserting into full leaf yields 4 keys. Split: median key (index 2? Let's check). For order m=4, max keys m-1=3. When overflow 4 keys, split into two nodes each with floor((m-1)/2)? Standard: median key moves up. For 4 keys [a,b,c,d], median could be b (index 1) or c (index 2) depending convention. For order 4, if split into left 1 key, right 1 key, median one key. Which median? Usually choose middle key: for even number of keys after overflow, choose lower median? Let's examine B-tree insertion algorithm: If node has 2t-1 keys (full), split into two nodes with t-1 keys each, median key moves up. For order 4, t=2, full has 3 keys. Insert into full gives 4 keys. Split into two nodes with t-1=1 key each, median key moves up. Which median? If keys [k1,k2,k3,k4], left gets k1, median k2, right gets k3? But then k4? Wait if full before insert has 3 keys, after insert 4. To split into two nodes with 1 key each and median up, total keys accounted: left 1 + median 1 + right 1 = 3, but we have 4 keys. Something off. Standard B-tree insertion splits a full node before descending (top-down) when node has 2t-1 keys. For order 4 (t=2), full node has"
+ },
+ {
+  "role": "assistant",
+  "content": null,
+  "reasoning_content": "We need to answer user: \"Summarize the causes and consequences of the 1997 Asian financial crisis in about 300 words.\" Need produce final ~300 words. Need likely concise. Need ensure about 300 words. Let's draft around 300. Need causes and consequences. Could mention fixed exchange rates, short-term foreign debt, current account deficits, asset bubbles, weak financial regulation, capital inflows, sudden reversal, contagion, IMF austerity, recessions, social unrest, reforms. Need word count maybe 280-320. Let's craft 300-ish.\n\nDraft:\n\"The 1997 Asian financial crisis began in Thailand and spread across East and Southeast Asia. Its causes were structural and cyclical. During the early 1990s, many Asian economies attracted large inflows of short-term foreign capital while maintaining fixed or heavily managed exchange rates pegged to the dollar. This encouraged borrowing in foreign currencies and fueled property and stock-market bubbles, especially in Thailand, Indonesia, South Korea, and Malaysia. Weak financial supervision, crony lending, and opaque corporate balance sheets meant banks and firms took excessive risks. Current-account deficits and falling export competitiveness made these pegs vulnerable. When investors lost confidence, currencies came under attack. Thailand\u2019s baht was devalued in July 1997, triggering contagion as investors fled similar economies. Currency collapses made foreign-currency debts unpayable, causing bank failures, corporate bankruptcies, and sharp capital outflows.\n\nThe consequences were severe. Output contracted, unemployment rose, and poverty increased, especially in Indonesia, Thailand, South Korea, and Malaysia. Stock markets and real estate collapsed, wiping out wealth. Governments faced fiscal strain as they rescued banks and businesses. The IMF provided emergency loans, but its conditions\u2014fiscal austerity, higher interest rates, and rapid financial liberalization\u2014deepened recessions and provoked political backlash. Indonesia suffered social unrest and the fall of Suharto. In response, affected countries reformed banking systems, strengthened corporate governance, moved toward more flexible exchange rates, and built foreign-exchange reserves. The crisis also prompted regional cooperation, including the Chiang Mai Initiative, and greater skepticism toward unfettered capital flows. Overall, it exposed the dangers of fixed exchange rates, short-term foreign borrowing, and weak financial regulation, while reshaping Asia\u2019s economic policies and institutions.\"\n\nCount words? Let's count roughly. The(1) 1997(2) Asian(3) financial(4) crisis(5) began(6) in(7) Thailand(8) and(9) spread(10) across(11) East(12) and(13) Southeast(14) Asia(15). Its(16) causes(17) were(18) structural(19) and(20) cyclical(21). During(22) the(23) early(24) 1990s(25), many(26) Asian(27) economies(28) attracted(29) large(30) inflows(31) of(32) short-term(33) foreign(34) capital(35) while(36) maintaining(37) fixed(38) or"
+ },
+ {
+  "role": "assistant",
+  "content": "```ts\ninterface Input {\n  id: string;\n  name: string;\n  on: boolean;\n}\n\ninterface Output {\n  id: string;\n  n: string;\n}\n\nfunction f(a: Input[]): Output[] {\n  return a\n    .filter((item) => item.on)\n    .map(({ id, name }) => ({\n      id,\n      n: name.trim(),\n    }));\n}\n```",
+  "reasoning_content": "We need answer user's request: \"Refactor this into idiomatic TypeScript with types: function f(a){var r=[];for(var i=0;i<a.length;i++){if(a[i].on){r.push({id:a[i].id,n:a[i].name.trim()})}}return r}\"\n\nNeed produce final answer likely code. Need think about idiomatic TS with types. Need maybe define interfaces. Original function takes array of objects with on, id, name. Returns array of {id, n}. Need types. Could use filter/map or reduce. Idiomatic TS:\n\ninterface Source { id: string; name: string; on: boolean; }\ninterface Result { id: string; n: string; }\n\nfunction f(a: Source[]): Result[] {\n  return a.filter(item => item.on).map(({ id, name }) => ({ id, n: name.trim() }));\n}\n\nCould also use type aliases. Need maybe preserve behavior: if name undefined? Original assumes name string. Could make robust: name: string. If optional, handle. But likely simple.\n\nNeed maybe mention if id type unknown use generic? Could be number/string. Use generic? Idiomatic with types:\n\ntype Input = { id: string; name: string; on: boolean };\ntype Output = { id: string; n: string };\n\nconst f = (a: Input[]): Output[] =>\n  a.filter(({ on }) => on).map(({ id, name }) => ({ id, n: name.trim() }));\n\nNeed maybe if want readonly? Could use readonly arrays. But not necessary.\n\nNeed final concise with explanation maybe. Ensure no analysis outside? Final answer.\n"
+ },
+ {
+  "role": "assistant",
+  "content": null,
+  "reasoning_content": "We need answer user: \"Write a bash script that finds the 10 largest files under a directory, skipping .git, and prints human-readable sizes.\" Need provide script. Need likely include usage, maybe robust. Need final only. Need think about best implementation.\n\nNeed bash script finds 10 largest files under a directory, skipping .git, prints human-readable sizes. Could use find with -path '*/.git' -prune -o -type f -print0 | xargs -0 du -h | sort -hr | head -n 10. But du -h prints sizes with human units, sort -hr works GNU sort. Need handle spaces. Maybe use find -printf '%s\\t%p\\0' then sort -z -nr and head, then du -h? But du -h on paths. Simpler:\n\n#!/usr/bin/env bash\nset -euo pipefail\ndir=\"${1:-.}\"\nfind \"$dir\" -name .git -prune -o -type f -print0 |\n  xargs -0 du -h |\n  sort -hr |\n  head -n 10\n\nBut issue: if no files, xargs du with no input? xargs -0 du -h with empty input may run du with no args? GNU xargs runs command once with no args if empty? Actually xargs runs command if input empty? By default yes? It may run du with no args, du .? Need use -r? GNU xargs -r don't run if empty. But not portable? Could use if. Also du -h output sort -hr: human-readable sort works with GNU sort. If du outputs \"1.0K\\tfile\" sort -hr sorts by size. Good.\n\nBut skipping .git: find \"$dir\" -name .git -prune -o -type f -print0. This prunes any directory named .git. If file named .git? -name .git -prune applies to any entry; if file .git, pruned? It will not print file .git. Fine.\n\nPotential issue: du -h on files may output sizes with tabs and paths. If path contains newline? print0 handles. du -h output line per file; if path contains newline, sort/head may break. Could use find -printf '%s\\t%p\\0' | sort -z -nr | head -z -n 10 | while IFS= read -r -d '' line; size=${line%%$'\\t'*}; path=${line#*$'\\t'}; du -h \"$path\"; done. But du -h again. Or use numfmt --to=iec-i --suffix=B? Need human-readable. Could use find -printf '%s\\t%p\\0' | sort -z -nr | head -z -n 10 | while ...; do printf '%s\\t%s\\n' \"$(du -h \"$path\" | cut -f1)\" \"$path\"; done. But if path newline, du output line? Could use find -printf '%s\\t%p\\0' and then use awk? Need human-readable sizes. Could use du -h --files0-from? GNU du supports --files0-from=-? It can read NUL-separated file names from stdin and output human sizes. Then sort? Need"
+ }
+]