Skip to content

gc-plan runtime: data-only packs run by the engine (embedding and decoder models) - #7

Open
peti12352 wants to merge 19 commits into
devfrom
gc2-plan
Open

peti12352 wants to merge 19 commits into
devfrom
gc2-plan

Conversation

@peti12352

@peti12352 peti12352 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

A new pack runtime kind, gc-plan v1, in the existing pack 4 container. A bucket is data only: plan.json (device buffers, the model tensors bound into them by name, the step's kernel launches, memsets and copies, inputs, outputs) plus the cubins its launches name. No library, no cudart, no cuBLAS.

goethe_engine/gc_plan.py runs it through the CUDA driver API:

  • weights read from the GGUF at their data offsets (new structure.gguf_offsets); weights and caches shared by every bucket, scratch per bucket
  • one CUDA graph per bucket (PDL and cluster attributes, 2D copies)
  • inputs generated from the step: token ids, positions, causal and encoder masks (padded key cells, symmetric sliding window), KV cell indices, recurrent-state slots, mean and cls pooling, per-row constants the compiler measured

Backends (registry dispatches gc-plan packs by model type):

  • PlanEmbed: embedding models from encode buckets of a chunk-length ladder, each item padded into the smallest bucket that holds it.
  • PlanBackend: decoders as a page backend (paged=False, one engine block = one sequence). A bucket's graph fixes its streams, so live sequences stay packed in streams [0, n) with a spare stream: prefills run in stream 0 with its occupant parked, a step that decodes every live sequence runs the decode bucket of exactly that width. Prompts split exactly into the pack's chunk lengths (padding would advance recurrent state). Buckets come in KV widths and each launch takes the width llama.cpp reads (cells padded to 256).

Checked on an RTX PRO 6000 (sm_120a) against llama.cpp's own runs:

  • embeddinggemma-2 Q8_0: 128 tokens and 100 tokens padded into the 128 bucket, bitwise equal.
  • Llama-3.2-1B Q4_K_M and a hybrid recurrent model (OrcaCyber-Nano): 300-token prompts, 48 greedy tokens through the Engine loop, every token equal to llama.cpp run with the same chunk split; 4 concurrent requests batch through the 4-sequence decode bucket.

Tests: tests/test_gc_plan.py (CPU: GGUF offsets, masks, pooling, fills, refusal of unknown inputs); test_pack's pinned runtime list now includes gc-plan.

Not in this PR: speculative decoding and prefix cache for gc-plan decoders, the pack index (index.py accepts abi_major 3 only), signing of gc-plan packs.

…river API (embedding first)

A pack 4 whose runtime kind is gc-plan v1 carries no library: each bucket
is plan.json (device buffers, the model tensors bound into them by name,
the step's kernel launches with argument bytes and pointer slots, memsets
and device copies, inputs, outputs) plus the cubins its launches name.
goethe_engine/gc_plan.py runs it: one context, weights read from the GGUF
at their data offsets (structure.gguf_offsets), weights and caches shared
by every bucket, scratch per bucket, a CUDA graph per bucket captured on
first use (PDL and cluster attributes, 2D copies), inputs generated from
the step (token ids, positions, encoder masks with padded key cells and
the symmetric sliding window, mean and cls pooling, per-row fills the
compiler measured). PlanEmbed serves embedding models from encode buckets
of a chunk-length ladder, padding an item to the smallest that holds it.
registry dispatches gc-plan embedding packs to it; RUNTIMES lists
gc-plan 1.

Checked on an RTX PRO 6000 (sm_120a): embeddinggemma-2 Q8_0, 128 tokens
and 100 tokens padded into the 128 bucket, both bitwise equal to
llama.cpp's own output. CPU tests: test_gc_plan.py.
A decoder pack has prefill buckets (T tokens at any position whose cells
fit the bucket) and a one-sequence decode bucket, all of one KV width: a
bucket reads n_kv cells per stream and masks the cells past the position,
so one bucket serves every position below n_kv. A token run splits
exactly into the prefill chunk lengths, largest first, the last tokens
one at a time through the decode bucket (padding would advance recurrent
state). PlanBackend is a page backend (paged=False: an engine block is
one sequence's stream and recurrent slot), max_seqs 1 so far; release
zeroes the slot's recurrent state; copy_blocks copies slots. registry
dispatches gc-plan decoders to it.

Checked on an RTX PRO 6000: Llama-3.2-1B Q4_K_M, a 300-token prompt, 48
greedy tokens through the engine's Engine loop, all equal to llama.cpp run
with the same chunk split (llama.cpp's one-ubatch prefill itself departs
from its chunked one at token 6).
…fore the last

PlanBackend keeps live sequences packed in streams [0, n) with the top
stream spare (a bucket's graph fixes its streams): a prefill runs in
stream 0 with its occupant parked in the spare, a step that decodes every
live sequence runs the decode bucket of exactly that width, other decodes
run one sequence at a time; release moves the last sequence into the
hole. Buckets come in KV widths (llama.cpp's cells per stream, padded to
256): a launch takes pad(last position + 1), the width llama.cpp reads, so
short contexts read few cells and the numerics are llama.cpp's own. A
prompt chunk before its last (want 0) returns no row.

Checked on an RTX PRO 6000, Llama-3.2-1B Q4_K_M, 43 buckets (widths 256 to
1024, prefill 512/64/8, decode 1 to 8 sequences), 4 requests of 300-token
prompts and 48 greedy tokens through the Engine: one at a time, every
token equal to llama.cpp with the same chunk split; 4 at once (batched
decode, 882 tok/s), equal while the batch kernels agree with B=1.
@peti12352 peti12352 changed the title gc-plan runtime: data-only packs run by the engine (embedding first) gc-plan runtime: data-only packs run by the engine (embedding and decoder models) Nov 17, 2026
The loader reads both pack majors (the container's magic fixes which), so
an index entry may name either; match_dir keeps the runtime filter and
ranks gc-plan after gc-c when both serve a model. A major the loader does
not read is still refused. test_index: a v4 gc-plan entry validates and
matches after the v3 gc-c one; abi_major 5 is refused.
A launch action may carry TMA descriptors (argument, byte offset, the
recorded encode parameters, the tensor's buffer and offset): the runtime
encodes each with this driver at the real address. A plan's init actions
(e.g. a weight relaid into another kernel's layout) run once, after the
weights load. PlanBackend picks for prefill and decode separately the
width llama.cpp would read (cells padded to 256), else the narrowest wider
one (cells past the position masked).

Checked on an RTX PRO 6000 with Qwen3.8-27B NVFP4 (prefill GEMMs on
CUTLASS sm_120 block-scaled kernels with TMA, weights relaid at load):
64 documents, 512-token prompts and 32 teacher-forced steps at 64
sequences through PlanBackend; KL to llama.cpp's one-sequence run no
larger than llama.cpp's own batched run's.
A plan input that is a view of another (llama.cpp's Mamba-2 rs_s_copy
views) is generated as its base and sliced at the element offset the
compiler measured (io.inputs.views, keyed by name and size); base()
maps a view to its base's kind. Checked through the accuracy gate on
a Nemotron-H transcriber (prefill and 64-sequence decode).
A bucket may carry a second plan (affine: lib1, n_kv0, n_kv1): at open
the engine expands it into a plan per KV width between them in steps
of 256, by lerp_plan: every differing quantity affine in the width
(device pointers as 64-bit words, ggml fastdiv triples recomputed from
their divisor, host addresses kept, CUTLASS parameter blocks' unread
bytes kept), exact in integers or refused. The widths of one bucket
share the widest width's scratch buffers. Checked by the compiler at
pack time at every width, and through PlanBackend on a Llama-1B pack
(greedy tokens equal to llama.cpp).
…bucketed backends; device argmax in gc-plan

A backend whose decode bucket runs every live sequence
(batches_all_live: gc-plan) gets steps of prefill only while prompts
remain (decodes wait a step, then run batched; a step is never left
empty), and prompt pieces short of the prompt's end in whole multiples
of its largest prefill chunk (prefill_align): ragged leftovers had
become runs of 64-, 8- and 1-token buckets (15874 steps for 64 prompts
of 4096 tokens, now 544). PlanBackend returns tokens when every wanted
row is greedy: the pack's gc_argmax_rows reduces the decode's logits
on the device (4 bytes a row read back instead of the vocabulary).

Llama-1B scoreboard (64 x 4096 prompts, 128 tokens): decode B64 266
-> 6545 tok/s; batched prompt throughput 11.8k -> 58k tok/s.
…np_kq_mask_swa)

A sliding-window layer's cache is a ring of swa_cells cells per stream
(llama.cpp: pad256(min(cells, n_swa + n_ubatch))): a token at position p
goes to cell stream * S + p % S, and a query at p sees ring cell j iff it
holds the newest written position p_j = j (mod S) with p - n_swa < p_j
<= p. Before, both caches' inputs (llama.cpp names them alike) were
generated as the full cache's, so a sliding-window layer attended over
the whole context. The compiler names the kinds and checks every
input against llama.cpp's recorded bytes; the test checks the
generators against a simulated ring.
…uploads, argmax and readback captured)

Inputs were copied from pageable Python buffers (staged synchronously by
the driver, one call each) and the greedy argmax and readback ran
outside the graph. Now each (bucket, argmax rows) graph captures the
uploads from a pinned staging buffer, the actions, the device argmax and
the copies into pinned outputs: a step generates its inputs into the
staging buffer, launches, synchronizes. PlanBackend.warm captures the
greedy forms of the decode buckets too. Llama-3.2-1B Q4_K_M on GPU 3:
B1 790 -> 803 tok/s, B64 6533 -> 6634, pp8192 49720 -> 53447.
…xt bucket up

A decode of n live sequences with no n-row bucket ran one sequence at
a time (a batch of 63 after one of 64 finished: 780 tok/s on Llama
3.2 1B). It now runs in the smallest decode bucket of at least n rows,
the rows past the live ones in free streams at position 0 (the live
sequences are packed in [0, n)), their outputs dropped; a free
stream's state is zeroed when a sequence takes it. 63 live: 11488
tok/s. Rows are independent in a bucket: 5 live sequences padded into
B8 give bitwise the tokens of the same five among 8 live, on Llama
3.2 1B and on LFM2.5 (recurrent state).
…he first step)

warm() only captured; the first real step of each bucket paid CUDA's
lazy module loading (gemma-4 pp512 on a fresh pack: 10510 tok/s; with
the graphs run once at warm: 12683). Staging buffers start zeroed, so
the warm run reads token 0 at position 0; the garbage it leaves in the
caches is zeroed when a sequence takes its stream.
… now takes any block size)

Qwen3-1.7B decode: the greedy pick was 0.21 ms of a 3.3 ms step; with
float4 loads and 1024 threads it is 11 us (kern/test_argmax.py). B1
291 -> 310 tok/s, B2 454 -> 476, tg128 319 -> 340.
…ed in-graph, the fastest aliased, cached per GPU UUID, pack and driver

jit(): round robin over the candidates (7 launches each, median), the pick in
GOETHE_HOME/tune/<uuid>-<pack sha>-<driver>.json; later starts read it (GOETHE_JIT=0 skips, =force
re-times); the timing steps' cache writes zeroed. Qwen3-1.7B batch pack: 3 buckets, 8 plans, 1.33 s.
… the smallest prefill chunk that holds it

- JIT: an affine bucket's alternative plan pairs interpolated at every width; the widest timed,
  every width takes the pick (Qwen3-1.7B batch: 6 buckets in 2.0 s).
- Padding (not for recurrent models; GOETHE_PAD=0: the exact split): a tail with no exact chunk ran
  token by token through the decode bucket. Qwen3-1.7B, Max-Q: 37 / 100 / 300-token prompts 10 / 27 /
  10 ms vs 105 / 304 / 870 ms; last logits cosine >= 0.9999993, 32 greedy tokens identical.
_split: a DP over exact and padded chunks minimizing the launches' time on this device (each chunk
length timed at its narrowest and widest KV widths with the JIT, affine between; cached with it),
then the launch count; launched tokens when untimed. The JIT's prefill timing step starts where
the bucket's last position fits (it ran past the width).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant