Repository navigation
Dev - #188
Merged
Merged
Dev#188
Conversation
…; a pinned GPU count shrinks the context to fit Every layer was counted at full context, so gemma-4-E2B planned 8.8 GB of KV for one request instead of 0.8 GB and did not fit a 16 GB Mac. Sliding-window layers now hold their window (plus a batch) per request, layers that share an earlier layer's KV hold none, and the header is read at every plan so models installed before get it too. A llama.cpp card pins tp 1, which skipped the context halving the automatic plan does; a pinned tp now shrinks context, then concurrency, like a pinned GPU set.
…pp on the model's GGUF build, else refused; ENGINE in goethe models A model SGLang does not serve now goes to llama.cpp on its own GGUF files, or on the GGUF build Hugging Face lists for it (the publisher's own <id>-GGUF first, else the most downloaded for the model or the model it derives from), checked against the architectures the pinned llama.cpp serves (llama-cpp/ARCHS); the name typed becomes an alias. The Mac takes the same GGUF build instead of a name search. After a download, the pack structure check runs (and installs goethe-engine) only when a pack of the weights' architecture exists for these GPUs, and a failing check or unusable pack gives way to the plain runtime by default (GOETHE_ALLOW_FALLBACK=0 makes it an error). goethe models shows each model's engine.
Light engines stop paying SGLang's 1.8 GB overhead; retired pkg host migrates; pack SM unknown, verification origin, public share allowlist, and cmdline secrets are tightened. Share --allow is the client's IP (or goethe api allow); nginx always deny all.
Drop the separate ternary Engine and per_gpu_budget_gb wrapper. Card runtime spelling stays ternary for learned rows; launch and overhead go through sglang. Equivalence tests cover argv, env gating, reserved flags, and digests.
Examples and the --allow row matched the old open-to-anyone behavior; they now match fail-closed share.
Top-level examples and the --allow clap line no longer teach opening without who may connect.
Reject everyone-CIDRs and the URL host as --allow; do not inherit goethe api allow for the public window; leaf cert is CA:FALSE; menu prints who may connect.
0.0.0.0/0 still needs that flag; rate limits stay on by default. Docs restore the URL/key/cert example comment.
…s warn. Reopening without --key no longer reprints the secret. Card LD_*/DYLD_* drops are named on stderr. models and models recommend both show per-GPU need with ctx and concurrency.
…nstall and gateway setup Load/Unload sits under Quick setup (grayed out until a model is installed): installed models beside it, a load with goethed started when down and its bar next to the model until it says loaded, several loaded models unloaded at once. API keys and gateway gets API schema (OpenAI, Anthropic, LiteLLM with placeholders on the gateway's address), and Quick setup ends on it. q quits from any screen, the left arrow goes back. Install drops its summary lines and verifies with one line; gateway setup ends with Gateway updated and live. Embedding models warm up with an embedding request and refuse chat with a plain 400; SGLang's decode CUDA graphs are capped at the model's concurrency.
…ory plans, menu Load/Unload and API schema - Engine order, decided before any download: a goethe kernel pack on a logged-in node, else SGLang, else llama.cpp on the repo's GGUF or its Hugging Face GGUF build (name kept as an alias), else refused with each reason. Embedding models such as embeddinggemma install this way; a pack that cannot run falls back by default. - GGUF memory counts sliding-window and shared-KV layers (gemma-4-E2B: 0.8 GB KV instead of 8.8 GB), and a fixed GPU count shrinks the context to fit. - goethe models shows each model's engine. - Menu: Load/Unload with in-place loading and multi-unload, API schema (OpenAI, Anthropic, LiteLLM), q quits everywhere, left arrow goes back. - Quieter install and gateway setup output; loads only say Loading into VRAM. - Embedding models get an embedding warm-up and a plain 400 for chat; SGLang decode CUDA graphs are capped at the model's concurrency.
Keep both: their GGUF window KV, cuda-graph caps, ENGINE column, menu Load/Unload; our engine overhead_gb, fit ctx/concurrency labels, SGLang boot api-key, ternary fold.
… that is now the default
Bare `expose public --yes` and raw `expose 0.0.0.0` refuse; cert SANs include interface IPs; models scan resolves folder names under GOETHE_HOME/models. isolated-home E2E traces and stage5 cover the regressions.
…l run packs. Also teach fake uv the wheel install path used after a plain-runtime fallback.
Fail-closed fit, share, login, and engine key handling
… private gateway. Defaults stay 10 req/s and 16 connections; set before open to tune.
Share: hidden rate/conn env knobs
A one-crate edit runs test+clippy, not fresh-user/user-paths or ninfer. Push/schedule/dispatch still run everything.
CI: path-scoped PR jobs
CI: rename ninfer flag to patches
…hen it changes; share needs its clients in the menu too The boot script ran SGLang at module level, so each spawned scheduler re-imported it and started a second server: every SGLang load died during init. It was also written with write_private, which refuses the 0755/0775 lib/ a normal umask makes, and only when missing, so a fixed script never replaced a broken one. The menu's Allowed IPs is required like --allow (everyone opens it on purpose, as --dangerously-allow-all-inbound does), and setup cleans the retired package host after making the home private.
…ts clients in the menu too - SGLang boot script guarded with __main__ (spawned schedulers re-imported it and started a second server, so every SGLang load died), written in any lib/ mode and rewritten when it changes. - Menu public share: Allowed IPs is required; everyone opens it on purpose, like --dangerously-allow-all-inbound. - setup cleans the retired package host after making the home private.
… builds. Goethe enables HTTP/2 on the gateway and share listeners; 1.30.6 is in the affected range for CVE-2026-123007. Package rebuild still required to ship the binary.
… the installed wheel. Prove-and-fix: re-exec used invalid --engine (clap wants positional engine); plain update skipped bundled nginx; packs gated on compile-time PIN so a new binary with an old py-env could accept packs it cannot run. Docs match the CLI.
Fold clippy into engine-test and merge FreshUser+Matrix e2e; unify nginx limit/quote helpers; simplify pack broker auth; use std ARCH.
Harden update + nginx 1.30.7 (HTTP/2 CVE)
Marlin dual-copy was counted as steady VRAM and refused 27B NVFP4 on a 32 GB card that runs it after auto-shrink. Registry --quantization modelopt_fp4 also aborted compressed-tensors checkpoints at startup.
Keep Marlin in the weight budget (measured ~30 GB resident) but raise mem_fraction to 0.99 and skip the 1.5+1.8 padding when weights already fill the card and KV has been shrunk away — matching SGLang's own floor.
Marlin's second weight copy left no KV/mamba room on full cards; plan without it (GOETHE_MARLIN=0) when that yields a better or only placement, and allow mem_fraction up to 1.0. Remove the near-full padding bypass that falsely allowed oversubscribed loads.
Torch OOM lines report MiB; treating them as GiB inflated learned weights into hundreds of GB. Prefill graph capture also OOMs after a near-full static pool, so disable it automatically when mem_fraction is high.
Prefill capture lives outside the static pool and OOMs on tight cards — decode graphs are enough. Also skip the bare-weight replan when the dual-copy plan already meets the asked ctx/concurrency.
Tighten calc shortfall asserts under mem_fraction=1.0 and exercise planner shapes (roomy/tight/foreign/empty/partial) plus drop_quantization across common quant_method values without needing a live GPU.
With mem_fraction capped at 1.0 the static-pool shortfall path never bound; margin already covers it. Lint-clean the CPU fit tests.
Fit stays scan+card only; leave engine_overhead learning untouched for a separate decision.
One helper converts the allocate/free figures to GB so MiB lines stop inflating learned static VRAM; same DB learn path as before.
Same changes/gate shape as Linux ci.yml: crates only pay for unit+clippy; packaging/installer/e2e script pay for the Mac package smoke. rust-cache on both Mac jobs; cancel-in-progress only on pull requests.
Fit near-full cards: shrink, drop dual-copy, fix quant/OOM units
CI: split macos into unit vs package e2e
Product lives at goethe/; the Python helpers are lab (import lab). Path rewrites cover CI, docs, and packaging; stale mutant patches are regenerated so all 12 apply and compile.
Rename goethe/→lab/ and engine/→goethe/
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.