Repository navigation
Conversation
…river API (embedding first) A pack 4 whose runtime kind is gc-plan v1 carries no library: each bucket is plan.json (device buffers, the model tensors bound into them by name, the step's kernel launches with argument bytes and pointer slots, memsets and device copies, inputs, outputs) plus the cubins its launches name. goethe_engine/gc_plan.py runs it: one context, weights read from the GGUF at their data offsets (structure.gguf_offsets), weights and caches shared by every bucket, scratch per bucket, a CUDA graph per bucket captured on first use (PDL and cluster attributes, 2D copies), inputs generated from the step (token ids, positions, encoder masks with padded key cells and the symmetric sliding window, mean and cls pooling, per-row fills the compiler measured). PlanEmbed serves embedding models from encode buckets of a chunk-length ladder, padding an item to the smallest that holds it. registry dispatches gc-plan embedding packs to it; RUNTIMES lists gc-plan 1. Checked on an RTX PRO 6000 (sm_120a): embeddinggemma-2 Q8_0, 128 tokens and 100 tokens padded into the 128 bucket, both bitwise equal to llama.cpp's own output. CPU tests: test_gc_plan.py.
A decoder pack has prefill buckets (T tokens at any position whose cells fit the bucket) and a one-sequence decode bucket, all of one KV width: a bucket reads n_kv cells per stream and masks the cells past the position, so one bucket serves every position below n_kv. A token run splits exactly into the prefill chunk lengths, largest first, the last tokens one at a time through the decode bucket (padding would advance recurrent state). PlanBackend is a page backend (paged=False: an engine block is one sequence's stream and recurrent slot), max_seqs 1 so far; release zeroes the slot's recurrent state; copy_blocks copies slots. registry dispatches gc-plan decoders to it. Checked on an RTX PRO 6000: Llama-3.2-1B Q4_K_M, a 300-token prompt, 48 greedy tokens through the engine's Engine loop, all equal to llama.cpp run with the same chunk split (llama.cpp's one-ubatch prefill itself departs from its chunked one at token 6).
…fore the last PlanBackend keeps live sequences packed in streams [0, n) with the top stream spare (a bucket's graph fixes its streams): a prefill runs in stream 0 with its occupant parked in the spare, a step that decodes every live sequence runs the decode bucket of exactly that width, other decodes run one sequence at a time; release moves the last sequence into the hole. Buckets come in KV widths (llama.cpp's cells per stream, padded to 256): a launch takes pad(last position + 1), the width llama.cpp reads, so short contexts read few cells and the numerics are llama.cpp's own. A prompt chunk before its last (want 0) returns no row. Checked on an RTX PRO 6000, Llama-3.2-1B Q4_K_M, 43 buckets (widths 256 to 1024, prefill 512/64/8, decode 1 to 8 sequences), 4 requests of 300-token prompts and 48 greedy tokens through the Engine: one at a time, every token equal to llama.cpp with the same chunk split; 4 at once (batched decode, 882 tok/s), equal while the batch kernels agree with B=1.
The loader reads both pack majors (the container's magic fixes which), so an index entry may name either; match_dir keeps the runtime filter and ranks gc-plan after gc-c when both serve a model. A major the loader does not read is still refused. test_index: a v4 gc-plan entry validates and matches after the v3 gc-c one; abi_major 5 is refused.
A launch action may carry TMA descriptors (argument, byte offset, the recorded encode parameters, the tensor's buffer and offset): the runtime encodes each with this driver at the real address. A plan's init actions (e.g. a weight relaid into another kernel's layout) run once, after the weights load. PlanBackend picks for prefill and decode separately the width llama.cpp would read (cells padded to 256), else the narrowest wider one (cells past the position masked). Checked on an RTX PRO 6000 with Qwen3.8-27B NVFP4 (prefill GEMMs on CUTLASS sm_120 block-scaled kernels with TMA, weights relaid at load): 64 documents, 512-token prompts and 32 teacher-forced steps at 64 sequences through PlanBackend; KL to llama.cpp's one-sequence run no larger than llama.cpp's own batched run's.
A plan input that is a view of another (llama.cpp's Mamba-2 rs_s_copy views) is generated as its base and sliced at the element offset the compiler measured (io.inputs.views, keyed by name and size); base() maps a view to its base's kind. Checked through the accuracy gate on a Nemotron-H transcriber (prefill and 64-sequence decode).
A bucket may carry a second plan (affine: lib1, n_kv0, n_kv1): at open the engine expands it into a plan per KV width between them in steps of 256, by lerp_plan: every differing quantity affine in the width (device pointers as 64-bit words, ggml fastdiv triples recomputed from their divisor, host addresses kept, CUTLASS parameter blocks' unread bytes kept), exact in integers or refused. The widths of one bucket share the widest width's scratch buffers. Checked by the compiler at pack time at every width, and through PlanBackend on a Llama-1B pack (greedy tokens equal to llama.cpp).
…bucketed backends; device argmax in gc-plan A backend whose decode bucket runs every live sequence (batches_all_live: gc-plan) gets steps of prefill only while prompts remain (decodes wait a step, then run batched; a step is never left empty), and prompt pieces short of the prompt's end in whole multiples of its largest prefill chunk (prefill_align): ragged leftovers had become runs of 64-, 8- and 1-token buckets (15874 steps for 64 prompts of 4096 tokens, now 544). PlanBackend returns tokens when every wanted row is greedy: the pack's gc_argmax_rows reduces the decode's logits on the device (4 bytes a row read back instead of the vocabulary). Llama-1B scoreboard (64 x 4096 prompts, 128 tokens): decode B64 266 -> 6545 tok/s; batched prompt throughput 11.8k -> 58k tok/s.
…np_kq_mask_swa) A sliding-window layer's cache is a ring of swa_cells cells per stream (llama.cpp: pad256(min(cells, n_swa + n_ubatch))): a token at position p goes to cell stream * S + p % S, and a query at p sees ring cell j iff it holds the newest written position p_j = j (mod S) with p - n_swa < p_j <= p. Before, both caches' inputs (llama.cpp names them alike) were generated as the full cache's, so a sliding-window layer attended over the whole context. The compiler names the kinds and checks every input against llama.cpp's recorded bytes; the test checks the generators against a simulated ring.
…uploads, argmax and readback captured) Inputs were copied from pageable Python buffers (staged synchronously by the driver, one call each) and the greedy argmax and readback ran outside the graph. Now each (bucket, argmax rows) graph captures the uploads from a pinned staging buffer, the actions, the device argmax and the copies into pinned outputs: a step generates its inputs into the staging buffer, launches, synchronizes. PlanBackend.warm captures the greedy forms of the decode buckets too. Llama-3.2-1B Q4_K_M on GPU 3: B1 790 -> 803 tok/s, B64 6533 -> 6634, pp8192 49720 -> 53447.
…xt bucket up A decode of n live sequences with no n-row bucket ran one sequence at a time (a batch of 63 after one of 64 finished: 780 tok/s on Llama 3.2 1B). It now runs in the smallest decode bucket of at least n rows, the rows past the live ones in free streams at position 0 (the live sequences are packed in [0, n)), their outputs dropped; a free stream's state is zeroed when a sequence takes it. 63 live: 11488 tok/s. Rows are independent in a bucket: 5 live sequences padded into B8 give bitwise the tokens of the same five among 8 live, on Llama 3.2 1B and on LFM2.5 (recurrent state).
…he first step) warm() only captured; the first real step of each bucket paid CUDA's lazy module loading (gemma-4 pp512 on a fresh pack: 10510 tok/s; with the graphs run once at warm: 12683). Staging buffers start zeroed, so the warm run reads token 0 at position 0; the garbage it leaves in the caches is zeroed when a sequence takes its stream.
… now takes any block size) Qwen3-1.7B decode: the greedy pick was 0.21 ms of a 3.3 ms step; with float4 loads and 1024 threads it is 11 us (kern/test_argmax.py). B1 291 -> 310 tok/s, B2 454 -> 476, tg128 319 -> 340.
…ed in-graph, the fastest aliased, cached per GPU UUID, pack and driver jit(): round robin over the candidates (7 launches each, median), the pick in GOETHE_HOME/tune/<uuid>-<pack sha>-<driver>.json; later starts read it (GOETHE_JIT=0 skips, =force re-times); the timing steps' cache writes zeroed. Qwen3-1.7B batch pack: 3 buckets, 8 plans, 1.33 s.
… the smallest prefill chunk that holds it - JIT: an affine bucket's alternative plan pairs interpolated at every width; the widest timed, every width takes the pick (Qwen3-1.7B batch: 6 buckets in 2.0 s). - Padding (not for recurrent models; GOETHE_PAD=0: the exact split): a tail with no exact chunk ran token by token through the decode bucket. Qwen3-1.7B, Max-Q: 37 / 100 / 300-token prompts 10 / 27 / 10 ms vs 105 / 304 / 870 ms; last logits cosine >= 0.9999993, 32 greedy tokens identical.
_split: a DP over exact and padded chunks minimizing the launches' time on this device (each chunk length timed at its narrowest and widest KV widths with the JIT, affine between; cached with it), then the launch count; launched tokens when untimed. The JIT's prefill timing step starts where the bucket's last position fits (it ran past the width).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A new pack runtime kind, gc-plan v1, in the existing pack 4 container. A bucket is data only: plan.json (device buffers, the model tensors bound into them by name, the step's kernel launches, memsets and copies, inputs, outputs) plus the cubins its launches name. No library, no cudart, no cuBLAS.
goethe_engine/gc_plan.py runs it through the CUDA driver API:
Backends (registry dispatches gc-plan packs by model type):
Checked on an RTX PRO 6000 (sm_120a) against llama.cpp's own runs:
Tests: tests/test_gc_plan.py (CPU: GGUF offsets, masks, pooling, fills, refusal of unknown inputs); test_pack's pinned runtime list now includes gc-plan.
Not in this PR: speculative decoding and prefix cache for gc-plan decoders, the pack index (index.py accepts abi_major 3 only), signing of gc-plan packs.