Skip to content

Fail-closed fit, share, login, and engine key handling - #180

Merged
peti12352 merged 14 commits into
devfrom
harden-fit-and-fail-closed
Nov 18, 2026
Merged

peti12352 merged 14 commits into
devfrom
harden-fit-and-fail-closed

Conversation

@peti12352

@peti12352 peti12352 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Fit / overhead

Light engines (llama.cpp, goethe-engine, ninfer) were paying SGLang's 1.8 GB base overhead, so fit refused or looked too big.

  • Each Engine owns overhead_gb (sglang / ternary 1.8, light 0.5); need() uses that
  • Formula unchanged: Need = weights/tp + KV + window + state + overhead + load_peak
  • KV_tokens = max(ctx, ceil(max_conc·ctx/4))
  • Fit: safety 1.5, pool ≤ 0.95×(free−1.4)
  • Mac GGUF pick unchanged (file_mb + file_mb/7 + 512)

Model → engine

Unchanged install path; ternary launch is folded into SGLang:

  • Install writes card.runtime — kernel pack → goethe-engine; GGUF → llama.cpp; else registry / default sglang (or ternary / ninfer)
  • Load: engines::of_card → by_id
  • by_id("ternary") launches as sglang; card.runtime stays "ternary" for learned rows
  • TERNARY_* env applies only when the card says ternary and the kernels are installed

Fail-closed

  • verification_uri_complete must share the platform host (www twin ok)
  • Pack SM / smem unknown refuses when the pack asks for any
  • Card env drops LD_* / DYLD_* and names them on stderr (no silent strip)
  • Engine key: GOETHE_ENGINE_API_KEY / LLAMA_API_KEY in environ; SGLang via goethe_sglang_boot (not on /proc/cmdline)
  • Residual: goethe-engine and ninfer still pass --api-key until those binaries grow env/file support
  • setup / update still drop retired pkg.meshapplied.com from goethe.env (window still open)

Public share

Naive user path: --allow is who may connect (their phone/laptop public IP — ask them, or ifconfig.me on their device). This machine's IP is only the URL goethe prints.

Opening with --key still prints the URL, the key, and the certificate to send. Reopening without --key keeps the same keys/addresses but does not reprint the bearer (shows name + id; goethe api shows the secret).

Behavior Detail
Empty allow Refuses to open; nginx always ends with deny all
No VPN inherit Does not use goethe api allow / Tailscale CIDRs for the public window
Reopen Same keys keep addresses from their last share; bearer not reprinted
This machine as --allow Rejected (that IP is the URL host, not the client)
0.0.0.0/0 / ::/0 Rejected unless --dangerously-allow-all-inbound (not with --allow); CLI warns
Rate limits (default, no knobs) 10 req/s per client (burst 20), 16 connections — stricter than private gateway (100/s)
Certificate Leaf is CA:FALSE
Menu Success line prints who may connect (Only: …)

Expose / TLS / scan (isolated-home E2E)

  • Expose public: --yes alone no longer binds every interface on a leftover allowlist; needs --allow <their-ip>/32 --yes (or interactive y). Raw expose 0.0.0.0 refused. Hidden --dangerously-allow-all-inbound for world-open clients.
  • TLS: gateway cert SAN includes interface IPs so curl --cacert to the bind address works; cert refreshes when SANs change.
  • Scan: models scan <foldername> resolves under GOETHE_HOME/models, not CWD.
Sev Finding Status
P0 expose public --yes → bind every interface without client allow Fixed
P0 TLS SAN missing bind IP (curl --cacert fails) Fixed
P1 models scan folder name from CWD Fixed
P1 Symlink under models/ refused by scan Intentional; message points at install --name
P1 Non-TTY load needs --yes next to a display manager Intentional; E2E uses --yes
P2 Second GOETHE_HOME → alt port Warned; harness must read status URL
P2 goethe run has no -p Positional INPUT; documented

Sanitized summary: docs/e2e-findings-pr180.md. Session logs with host paths and keys stay off-repo.

Fit display (models / models recommend)

Both show the same shape so numbers are comparable:

  • goethe models: fits: X.X GB per GPU (ctx N, M concurrent)
  • goethe models recommend: X.X GB/GPU · ctx N · M conc (per-GPU, not total×tp)

Merged from engine-routing

Kept both: their GGUF window KV, cuda-graph caps, ENGINE column, menu Load/Unload; our engine overhead_gb, fit ctx/concurrency labels, SGLang boot api-key, ternary fold.

Cleanup

  • One RESERVED_FLAGS table for SGLang extras
  • per_gpu_budget_gb removed
  • hf / packs / ninfer hash through update::sha256_file

Tests

No public bind, no GPU. Covers real failure modes:

  • Ternary argv ≡ plain sglang (env only differs); plain cards ignore top-level TERNARY_*; missing kernels fail
  • Reserved flags cover former OWNED plus pack / weights / tokenizer; sha256 wrappers match
  • Share: refuses everyone / this-machine IP; --dangerously-allow-all-inbound → 0.0.0.0/0; flag + --allow errors; reopen strips bearer; empty allowlist never admits the world; rate/conn embedded in nginx block
  • Card LD_* / DYLD_* listed in card_env_parts drops; still absent from the process env
  • Fit label names ctx and concurrency; recommend preview uses the same per-GPU + ctx wording
  • Verification host; retired pkg env; light vs sglang overhead; llama key in env not argv; clap parses --dangerously-allow-all-inbound
  • stage5_configure_api.py: bare public --yes, raw 0.0.0.0, and 0.0.0.0/0 refuse; named client allow succeeds
  • Unit: scan folder names under models/; cert SAN includes interface IPs
  • E2E harness: fake CUDA reports SM/smem so fail-closed pack checks can still exercise packs; fake uv handles the wheel install path
  • clippy -D warnings clean

Test plan

  • cargo test configure / scan / nginx SAN / sglang / chooser / share
  • python3 tests/stage5_configure_api.py
  • CI: lint, engine-test, fresh-user, user-paths, gate
  • CI: vm, macos
  • Optional: re-smoke expose public --yes refuse + cert SAN includes interface IPs after gateway write

@peti12352
peti12352 force-pushed the harden-fit-and-fail-closed branch from 1c229f7 to 7a913eb Compare November 18, 2026 20:24
Light engines stop paying SGLang's 1.8 GB overhead; retired pkg host migrates; pack SM unknown, verification origin, public share allowlist, and cmdline secrets are tightened. Share --allow is the client's IP (or goethe api allow); nginx always deny all.
@peti12352
peti12352 force-pushed the harden-fit-and-fail-closed branch from 7a913eb to 9f3dcb1 Compare November 18, 2026 20:24
Drop the separate ternary Engine and per_gpu_budget_gb wrapper. Card runtime spelling stays ternary for learned rows; launch and overhead go through sglang. Equivalence tests cover argv, env gating, reserved flags, and digests.
@peti12352 peti12352 changed the title Fail-closed fit, share, login, and engine key handling Fail-closed fit, share, login; ternary folded into SGLang Nov 18, 2026
Examples and the --allow row matched the old open-to-anyone behavior; they now match fail-closed share.
Top-level examples and the --allow clap line no longer teach opening without who may connect.
Reject everyone-CIDRs and the URL host as --allow; do not inherit goethe api allow for the public window; leaf cert is CA:FALSE; menu prints who may connect.
0.0.0.0/0 still needs that flag; rate limits stay on by default. Docs restore the URL/key/cert example comment.
@peti12352 peti12352 changed the title Fail-closed fit, share, login; ternary folded into SGLang Fail-closed fit, share, login, and engine key handling Nov 18, 2026
…s warn.

Reopening without --key no longer reprints the secret. Card LD_*/DYLD_* drops are named on stderr. models and models recommend both show per-GPU need with ctx and concurrency.
Keep both: their GGUF window KV, cuda-graph caps, ENGINE column, menu Load/Unload;
our engine overhead_gb, fit ctx/concurrency labels, SGLang boot api-key, ternary fold.
Bare `expose public --yes` and raw `expose 0.0.0.0` refuse; cert SANs include interface IPs; models scan resolves folder names under GOETHE_HOME/models. isolated-home E2E traces and stage5 cover the regressions.
…l run packs.

Also teach fake uv the wheel install path used after a plain-runtime fallback.
@peti12352
peti12352 force-pushed the harden-fit-and-fail-closed branch from a44b87e to 57b6b02 Compare November 18, 2026 22:08
@peti12352
peti12352 merged commit 6fcc77d into dev Nov 18, 2026
14 checks passed
@peti12352
peti12352 deleted the harden-fit-and-fail-closed branch November 19, 2026 04:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant