Skip to content

Dev - #188

Merged
JanosMozer merged 49 commits into
mainfrom
dev
Nov 19, 2026
Merged

Dev#188
JanosMozer merged 49 commits into
mainfrom
dev

Conversation

@JanosMozer

Copy link
Copy Markdown
Collaborator

No description provided.

JanosMozer and others added 30 commits November 18, 2026 20:49
…; a pinned GPU count shrinks the context to fit

Every layer was counted at full context, so gemma-4-E2B planned 8.8 GB of KV for one request instead of 0.8 GB and did not fit a 16 GB Mac. Sliding-window layers now hold their window (plus a batch) per request, layers that share an earlier layer's KV hold none, and the header is read at every plan so models installed before get it too. A llama.cpp card pins tp 1, which skipped the context halving the automatic plan does; a pinned tp now shrinks context, then concurrency, like a pinned GPU set.
…pp on the model's GGUF build, else refused; ENGINE in goethe models

A model SGLang does not serve now goes to llama.cpp on its own GGUF files, or on the GGUF build Hugging Face lists for it (the publisher's own <id>-GGUF first, else the most downloaded for the model or the model it derives from), checked against the architectures the pinned llama.cpp serves (llama-cpp/ARCHS); the name typed becomes an alias. The Mac takes the same GGUF build instead of a name search. After a download, the pack structure check runs (and installs goethe-engine) only when a pack of the weights' architecture exists for these GPUs, and a failing check or unusable pack gives way to the plain runtime by default (GOETHE_ALLOW_FALLBACK=0 makes it an error). goethe models shows each model's engine.
Light engines stop paying SGLang's 1.8 GB overhead; retired pkg host migrates; pack SM unknown, verification origin, public share allowlist, and cmdline secrets are tightened. Share --allow is the client's IP (or goethe api allow); nginx always deny all.
Drop the separate ternary Engine and per_gpu_budget_gb wrapper. Card runtime spelling stays ternary for learned rows; launch and overhead go through sglang. Equivalence tests cover argv, env gating, reserved flags, and digests.
Examples and the --allow row matched the old open-to-anyone behavior; they now match fail-closed share.
Top-level examples and the --allow clap line no longer teach opening without who may connect.
Reject everyone-CIDRs and the URL host as --allow; do not inherit goethe api allow for the public window; leaf cert is CA:FALSE; menu prints who may connect.
0.0.0.0/0 still needs that flag; rate limits stay on by default. Docs restore the URL/key/cert example comment.
…s warn.

Reopening without --key no longer reprints the secret. Card LD_*/DYLD_* drops are named on stderr. models and models recommend both show per-GPU need with ctx and concurrency.
…nstall and gateway setup

Load/Unload sits under Quick setup (grayed out until a model is installed): installed models beside it, a load with goethed started when down and its bar next to the model until it says loaded, several loaded models unloaded at once. API keys and gateway gets API schema (OpenAI, Anthropic, LiteLLM with placeholders on the gateway's address), and Quick setup ends on it. q quits from any screen, the left arrow goes back. Install drops its summary lines and verifies with one line; gateway setup ends with Gateway updated and live. Embedding models warm up with an embedding request and refuse chat with a plain 400; SGLang's decode CUDA graphs are capped at the model's concurrency.
…ory plans, menu Load/Unload and API schema

- Engine order, decided before any download: a goethe kernel pack on a logged-in node, else SGLang, else llama.cpp on the repo's GGUF or its Hugging Face GGUF build (name kept as an alias), else refused with each reason. Embedding models such as embeddinggemma install this way; a pack that cannot run falls back by default.
- GGUF memory counts sliding-window and shared-KV layers (gemma-4-E2B: 0.8 GB KV instead of 8.8 GB), and a fixed GPU count shrinks the context to fit.
- goethe models shows each model's engine.
- Menu: Load/Unload with in-place loading and multi-unload, API schema (OpenAI, Anthropic, LiteLLM), q quits everywhere, left arrow goes back.
- Quieter install and gateway setup output; loads only say Loading into VRAM.
- Embedding models get an embedding warm-up and a plain 400 for chat; SGLang decode CUDA graphs are capped at the model's concurrency.
Keep both: their GGUF window KV, cuda-graph caps, ENGINE column, menu Load/Unload;
our engine overhead_gb, fit ctx/concurrency labels, SGLang boot api-key, ternary fold.
Bare `expose public --yes` and raw `expose 0.0.0.0` refuse; cert SANs include interface IPs; models scan resolves folder names under GOETHE_HOME/models. isolated-home E2E traces and stage5 cover the regressions.
…l run packs.

Also teach fake uv the wheel install path used after a plain-runtime fallback.
Fail-closed fit, share, login, and engine key handling
… private gateway.

Defaults stay 10 req/s and 16 connections; set before open to tune.
Share: hidden rate/conn env knobs
A one-crate edit runs test+clippy, not fresh-user/user-paths or ninfer.
Push/schedule/dispatch still run everything.
…hen it changes; share needs its clients in the menu too

The boot script ran SGLang at module level, so each spawned scheduler re-imported it and started a second server: every SGLang load died during init. It was also written with write_private, which refuses the 0755/0775 lib/ a normal umask makes, and only when missing, so a fixed script never replaced a broken one. The menu's Allowed IPs is required like --allow (everyone opens it on purpose, as --dangerously-allow-all-inbound does), and setup cleans the retired package host after making the home private.
…ts clients in the menu too

- SGLang boot script guarded with __main__ (spawned schedulers re-imported it and started a second server, so every SGLang load died), written in any lib/ mode and rewritten when it changes.
- Menu public share: Allowed IPs is required; everyone opens it on purpose, like --dangerously-allow-all-inbound.
- setup cleans the retired package host after making the home private.
… builds.

Goethe enables HTTP/2 on the gateway and share listeners; 1.30.6 is in the
affected range for CVE-2026-123007. Package rebuild still required to ship the binary.
… the installed wheel.

Prove-and-fix: re-exec used invalid --engine (clap wants positional engine);
plain update skipped bundled nginx; packs gated on compile-time PIN so a new
binary with an old py-env could accept packs it cannot run. Docs match the CLI.
peti12352 and others added 19 commits November 18, 2026 14:57
Fold clippy into engine-test and merge FreshUser+Matrix e2e; unify nginx limit/quote helpers; simplify pack broker auth; use std ARCH.
Harden update + nginx 1.30.7 (HTTP/2 CVE)
Marlin dual-copy was counted as steady VRAM and refused 27B NVFP4 on a
32 GB card that runs it after auto-shrink. Registry --quantization
modelopt_fp4 also aborted compressed-tensors checkpoints at startup.
Keep Marlin in the weight budget (measured ~30 GB resident) but raise
mem_fraction to 0.99 and skip the 1.5+1.8 padding when weights already
fill the card and KV has been shrunk away — matching SGLang's own floor.
Marlin's second weight copy left no KV/mamba room on full cards; plan
without it (GOETHE_MARLIN=0) when that yields a better or only placement,
and allow mem_fraction up to 1.0. Remove the near-full padding bypass that
falsely allowed oversubscribed loads.
Torch OOM lines report MiB; treating them as GiB inflated learned weights
into hundreds of GB. Prefill graph capture also OOMs after a near-full
static pool, so disable it automatically when mem_fraction is high.
Prefill capture lives outside the static pool and OOMs on tight cards — decode
graphs are enough. Also skip the bare-weight replan when the dual-copy plan
already meets the asked ctx/concurrency.
Tighten calc shortfall asserts under mem_fraction=1.0 and exercise planner
shapes (roomy/tight/foreign/empty/partial) plus drop_quantization across
common quant_method values without needing a live GPU.
With mem_fraction capped at 1.0 the static-pool shortfall path never bound;
margin already covers it. Lint-clean the CPU fit tests.
Fit stays scan+card only; leave engine_overhead learning untouched for a
separate decision.
One helper converts the allocate/free figures to GB so MiB lines stop
inflating learned static VRAM; same DB learn path as before.
Same changes/gate shape as Linux ci.yml: crates only pay for unit+clippy;
packaging/installer/e2e script pay for the Mac package smoke. rust-cache on
both Mac jobs; cancel-in-progress only on pull requests.
Fit near-full cards: shrink, drop dual-copy, fix quant/OOM units
CI: split macos into unit vs package e2e
Product lives at goethe/; the Python helpers are lab (import lab). Path rewrites cover CI, docs, and packaging; stale mutant patches are regenerated so all 12 apply and compile.
Rename goethe/→lab/ and engine/→goethe/
@JanosMozer
JanosMozer merged commit 22671ed into main Nov 19, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants