diff --git a/CHANGELOG.md b/CHANGELOG.md index fdac8d8..223484d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ ## Unreleased +- `goethe api expose public --yes` without `--allow` no longer binds every interface on a leftover allowlist; name the clients and confirm (`--yes` or `y`). Raw `goethe api expose 0.0.0.0` is refused (use `expose public`). World-open clients need the hidden `--dangerously-allow-all-inbound --yes`. +- The gateway TLS certificate SAN includes this host's interface IPs (tailnet, LAN), so `curl --cacert` to the bind address verifies; the cert refreshes when that list changes. +- `goethe models scan ` resolves the name under `GOETHE_HOME/models`, not the current working directory. - goethe picks the engine before downloading: a goethe kernel pack on a logged-in node, else SGLang when it supports the model, else llama.cpp on the repo's GGUF files or on the GGUF build Hugging Face lists for the model (the name typed becomes an alias), else a refusal with each engine's reason. Embedding models like embeddinggemma install this way. On a Mac, llama.cpp with the same GGUF lookup instead of a name search. - A kernel pack that cannot run gives way to the plain runtime by default, saying why (`GOETHE_ALLOW_FALLBACK=0` restores the error). The pack structure check after a download, and the goethe-engine install it needs, run only when a pack of the model's architecture exists for the GPU. - `goethe models` shows each model's engine. diff --git a/docs/cli.md b/docs/cli.md index 47817d1..b90ff70 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -271,24 +271,24 @@ Who can reach the API. Without an argument it shows where the API listens and ho | `goethe api expose` | Show where the API listens and who may connect (`--json` for scripts) | | `goethe api expose local` | This machine only | | `goethe api expose vpn` | A VPN or other private network (recommended) | -| `goethe api expose public` | A public address | -| `goethe api expose
` | Listen on `127.0.0.1` (this machine only), `0.0.0.0` (all addresses), one IP of this machine, or an interface such as `wg0`, whose address is followed | +| `goethe api expose public` | A public address (needs `--allow /32` and `--yes`, or `y` on a terminal) | +| `goethe api expose
` | Listen on `127.0.0.1`, one IP of this machine, or an interface such as `wg0` (followed). Not `0.0.0.0`: use `expose public` | For a VPN it lists the networks this machine is on, with the ranges each one reaches, and works with any VPN that has a network interface. It then asks whether to restrict access to specific IP addresses. The gateway follows the chosen interface: when its address changes, it moves within a few seconds, and while the interface has no address only this machine can connect. +`expose public` never binds every interface on a leftover allowlist. Name the clients with `--allow`, confirm with `--yes` (or `y`), and only then does it listen on `0.0.0.0`. Opening every client address needs the hidden `--dangerously-allow-all-inbound --yes`. + | Option | Description | |---|---| -| `--interface ` | With `vpn` or `public`: the network interface to serve, such as `wg0`; needed without a terminal when there are several | -| `--ip ` | With `vpn` or `public`: a fixed address of this machine instead of an interface | -| `--allow ` | With `vpn` or `public`: a client address or range to allow; repeatable | -| `--yes` | Apply without questions (`public` still refuses `0.0.0.0/0`) | +| `--interface ` | With `vpn`: the network interface to serve, such as `wg0`; needed without a terminal when there are several | +| `--allow ` | With `vpn` or `public`: a client address or range to allow; **required for public**; repeatable | +| `--yes` | Apply without questions. Public still needs `--allow` (or `--dangerously-allow-all-inbound`) | | `--json` | With no argument: the status as JSON | ```bash goethe api expose local goethe api expose vpn --interface wg0 -goethe api expose public --allow 203.0.113.7/32 -goethe api expose 0.0.0.0 +goethe api expose public --allow 203.0.113.7/32 --yes ``` ### `goethe api allow` @@ -318,11 +318,12 @@ The DNS record that serves the API under your own domain name, with this machine Opens the API on a public address for a limited time, for API keys you choose, so you can give someone outside your network a URL and a key. The normal API address, local or VPN, stays as it is. ``` -goethe api share 5h --key my-app # share my-app for 5 hours -goethe api share 1d --key ci --allow 203.0.113.7 # also share ci, for a day, from one address -goethe api share # status: URL, each key with its time left and addresses -goethe api share stop --key ci # stop sharing one key -goethe api share stop # close the public gateway +goethe api share 5h --key my-app --allow 203.0.113.7/32 # prints the URL, the key and the certificate to send +goethe api share 5h --key my-app --dangerously-allow-all-inbound # every address (rate limits still apply) +goethe api share 5h --key my-app # reopen: keeps addresses from the last share of those keys +goethe api share # status: URL, each key with its time left and addresses +goethe api share stop --key ci # stop sharing one key +goethe api share stop # close the public gateway ``` Every shared key uses the same public gateway, each until its own time and from its own addresses. The gateway stays open until the last key's time ends. A key whose time has ended is refused while the others keep working, and the addresses only it allowed are dropped from the gateway. Sharing a key again starts its time over (`5h` means 5 hours from now) and replaces its addresses; the other keys are not touched. Without `--key`, the keys of the last opening are shared again. Opening prints the URL, each key with the models it may use, the certificate to send along, and a `curl` command to try it. The keys stay the same, so a recipient can keep using the key you sent. Connections in progress are not interrupted. The menu's Public address page does the same with a form: the key, how long, and optional addresses. `goethe status` shows the gateway while it is open, and a node linked to the platform shows the public URL and when it closes on its page. @@ -331,7 +332,8 @@ Every shared key uses the same public gateway, each until its own time and from |---|---| | `` | How long to share the keys: minutes, hours or days, such as `30m`, `5h` or `1d`, at most `7d`. `stop` stops sharing; nothing shows the status | | `--key ` | API keys to share, each until the duration; repeatable or comma-separated. With `stop`, the keys to stop sharing | -| `--allow ` | Only these client addresses or ranges may use the keys given, such as `203.0.113.7`; repeatable. Without it any address may, with the key | +| `--allow ` | Only these client addresses or ranges may use the keys given, such as `203.0.113.7`; repeatable. Without it, addresses from the last share of those keys are kept; with none, Goethe refuses to open | +| `--dangerously-allow-all-inbound` | Allow every address (`0.0.0.0/0`). Rate limits still apply (10 requests/s per client, burst 20, 16 connections). Not with `--allow` | | `--port ` | The public port (default `8444`, or the last one used) | The recipient verifies the gateway with the certificate goethe prints, `GOETHE_HOME/nginx/certs/public.crt`. It holds no secret: `curl --cacert public.crt https://
:8444/v1/models -H "Authorization: Bearer "`. Most OpenAI-compatible clients take it through `SSL_CERT_FILE` or their CA-bundle setting. If nobody can connect, forward the port to this machine on your router or allow it in your cloud firewall; goethe does not open ports on routers. The public listener uses IPv4. @@ -347,7 +349,7 @@ The recipient verifies the gateway with the certificate goethe prints, `GOETHE_H - **API routes only:** health, readiness, admin and metrics endpoints are not served publicly. - **HTTPS only:** TLS 1.2 or 1.3, with its own certificate for the public listener. Plain HTTP connections are dropped. - **Stricter limits than the normal address:** 10 requests per second per client address (bursts of 20) and 16 connections per address, plus the same body size limit and header and body timeouts. -- **Optional address limit per key:** a key shared with `--allow` works only from those addresses, checked on every request; nginx also admits only the addresses of the shared keys when every key has a limit. +- **Client address required:** every shared key has an allowlist (`--allow`, or addresses kept from the last share of those keys). Nginx admits only that union and always ends with `deny all`. Every address (`0.0.0.0/0`) needs `--dangerously-allow-all-inbound`; this machine's own public IP is refused as `--allow`. Goethe also checks the client on every public request. - **Checks before opening:** nothing changes unless every check passes: the gateway and goethed are running, the keys are valid, and the port is free and is not the normal gateway's port. If nginx does not accept the new listener, the previous state is restored. ## Learning and traces diff --git a/docs/e2e-findings-pr180.md b/docs/e2e-findings-pr180.md new file mode 100644 index 0000000..d6f298a --- /dev/null +++ b/docs/e2e-findings-pr180.md @@ -0,0 +1,20 @@ +# E2E findings (PR180) + +Isolated-home session against the PR binary. Session logs with host paths and keys stay off-repo. + +## Status + +| Sev | Finding | Status | +|-----|---------|--------| +| P0 | `expose public --yes` binds every interface without a client `--allow` | Fixed | +| P0 | Gateway cert SAN missing interface IPs (`curl --cacert` to the bind address fails) | Fixed | +| P1 | `models scan ` resolved from the current directory | Fixed | +| P1 | Symlink under `models/` refused by scan | Intentional; use `goethe install --name` | +| P1 | Non-TTY load next to a display manager needs `--yes` | Intentional | +| P2 | A second `GOETHE_HOME` may pick another gateway port | Warned; read status URL | +| P2 | `goethe run` has no `-p`; input is positional | Documented | + +## CI coverage + +- Unit: public `--yes` without `--allow` refuses; scan folder names under `models/`; cert SAN includes interface IPs +- `engine/tests/stage5_configure_api.py`: bare public `--yes`, raw `0.0.0.0`, and `0.0.0.0/0` refuse; named client allow succeeds diff --git a/docs/user-guide.md b/docs/user-guide.md index fe417bd..fa90050 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -214,7 +214,7 @@ Models that were loaded when Goethe stopped come back when it starts again, incl | `goethe trace` | Record requests and replies as training traces (`on`, `off`, `--key`); see [traces.md](traces.md) | | `goethe learn` | Let a session keep what compaction would drop (`on`, `off`, `--model`); see [learn.md](learn.md) | | `goethe api domain` | DNS record for your own hostname (`check `) | -| `goethe api share 5h --key ` | Open the API on a public address for a while for chosen keys (`stop` closes it) | +| `goethe api share 5h --key --allow /32` | Open the API on a public address for a while for chosen keys from their IP (`stop` closes it) | ### GPUs and system @@ -307,7 +307,7 @@ goethe api expose vpn # or local, or public; asks who may connect - **VPN or private network (recommended):** connect your VPN first, then pick it from the list. Any VPN with a network interface works, and so does a cloud VM's private network. Goethe lists tunnels and interfaces on private addresses, leaves out container bridges and public network cards, and shows the range each one reaches, taken from the system's routes. When a VPN routes each device separately, the private block its address is in (such as `100.64.0.0/10` or `10.0.0.0/8`) is used, so devices added later can connect too; you can narrow it to specific addresses. Default routes are never allowed. The gateway follows the interface: a new address after a reconnect or a reboot is picked up within a few seconds, and while the interface is down only this machine can connect (`goethe status` says so). The allowlist itself only changes when you run `goethe api expose` again; `goethe doctor` warns if it no longer lets the network's clients in. A VPN without an interface (userspace mode) can't be listened on directly: use local mode and the VPN's own port forwarding. - **Public network:** any address that can reach this machine. You open the port in your firewall yourself. -It then asks whether to restrict access to specific IP addresses, and offers to create the first key. In scripts: `goethe api expose vpn --yes`, or `goethe api expose public --allow 203.0.113.7 --yes`. +It then asks whether to restrict access to specific IP addresses, and offers to create the first key. In scripts: `goethe api expose vpn --yes`, or `goethe api expose public --allow 203.0.113.7/32 --yes`. Public always needs `--allow` and a confirm (`--yes` or `y`); it will not open every interface on a leftover allowlist alone. ### Keys @@ -339,13 +339,13 @@ goethe api allow remove 192.168.1.0/24 To give someone outside your network access for a limited time, without changing how the API is normally reached: ```bash -goethe api share 5h --key my-app # prints the URL, the key and the certificate to send -goethe api share 1d --key ci # another key on the same address, for a day -goethe api share stop --key ci # stop sharing one key -goethe api share stop # close early +goethe api share 5h --key my-app --allow 203.0.113.7/32 # prints the URL, the key and the certificate to send +goethe api share 1d --key ci --allow 203.0.113.7/32 # another key on the same address, for a day +goethe api share stop --key ci # stop sharing one key +goethe api share stop # close early ``` -The public address exists only while some key is shared, accepts only the keys you chose (never admin keys), each until its own time, and closes by itself when the last one ends. Keys stay the same every time. The menu's Public address page does the same with a form. Details and the full list of security precautions: [`goethe api share`](cli.md#goethe-api-share-duration). +The public address exists only while some key is shared, accepts only the keys you chose (never admin keys), each until its own time and from the client addresses you named with `--allow` (or kept from the last share of those keys), and closes by itself when the last one ends. Keys stay the same every time. The menu's Public address page does the same with a form. Details and the full list of security precautions: [`goethe api share`](cli.md#goethe-api-share-duration). ### Address, aliases and your own domain diff --git a/engine/crates/core/src/calc.rs b/engine/crates/core/src/calc.rs index e8b46f1..76f0060 100644 --- a/engine/crates/core/src/calc.rs +++ b/engine/crates/core/src/calc.rs @@ -6,10 +6,6 @@ const CUDA_CONTEXT_GB: f64 = crate::engine_overhead::DEFAULT_CONTEXT_GB; pub const BASE_OVERHEAD_GB: f64 = 1.8; pub const METAL_OVERHEAD_GB: f64 = 0.5; -pub fn base_overhead_gb(cc: &str) -> f64 { - // llama.cpp on Metal has no CUDA context or graph pools, only its compute buffers. - if crate::gpu_arch::is_apple(cc) { METAL_OVERHEAD_GB } else { BASE_OVERHEAD_GB } -} const LEARN_SLACK_GB: f64 = 0.25; #[derive(Debug, Clone, Copy, PartialEq, Default)] diff --git a/engine/crates/core/src/chooser.rs b/engine/crates/core/src/chooser.rs index d3cb95a..50a588d 100644 --- a/engine/crates/core/src/chooser.rs +++ b/engine/crates/core/src/chooser.rs @@ -1,6 +1,6 @@ // Tensor parallel size and GPU set selection for goethe load. -use crate::calc::{base_overhead_gb, missing_gb, shortfall, Need}; +use crate::calc::{missing_gb, shortfall, Need, METAL_OVERHEAD_GB}; use crate::memory_spec::MemorySpec; const GIB: f64 = 1024.0 * 1024.0 * 1024.0; @@ -23,6 +23,7 @@ pub struct ChooserInput { pub supported_sm: Vec, pub memory: Option, pub window_gb_per_request: f64, + pub overhead_gb: f64, } pub struct CardFacts { @@ -50,6 +51,7 @@ impl ChooserInput { marlin_extra_gb: crate::install::card_marlin_extra_gb(card), supported_sm: crate::install::card_supported_sm(card), memory, + overhead_gb: crate::engines::of_card(card).map_or(crate::calc::BASE_OVERHEAD_GB, |e| e.overhead_gb), } } } @@ -131,13 +133,14 @@ pub fn need(input: &ChooserInput, tp: usize, cc: &str) -> Need { let tokens = kv_tokens(input.ctx, input.max_conc) as f64; let conc = input.max_conc.max(1); let request_table = ((conc + 1) * (input.ctx + 4) * 4) as f64 / GIB; + let oh = if gpu_arch::is_apple(cc) { METAL_OVERHEAD_GB } else { input.overhead_gb } + request_table; let Some(m) = &input.memory else { let shard = kv_shard(tp, input.num_kv_heads, &input.attention); return Need { weights: (input.static_vram_gb + marlin) / tp as f64, kv: tokens * input.kv_bytes_per_token as f64 / GIB / shard, window: conc as f64 * input.window_gb_per_request / shard, - overhead: base_overhead_gb(cc) + request_table, + overhead: oh, ..Need::default() }; }; @@ -152,7 +155,7 @@ pub fn need(input: &ChooserInput, tp: usize, cc: &str) -> Need { / tp as f64 + ((conc + 1) * m.request_bytes + m.fixed_bytes) as f64) / GIB, - overhead: base_overhead_gb(cc) + request_table, + overhead: oh, load_peak: m.load_peak_bytes as f64 / tp as f64 / GIB, } } @@ -168,10 +171,6 @@ pub fn cap_default_ctx(input: &ChooserInput, tp: usize, cc: &str, free_gb: f64) trial.ctx } -pub fn per_gpu_budget_gb(input: &ChooserInput, tp: usize, cc: &str) -> f64 { - need(input, tp, cc).total() -} - pub fn with_learned(base: Need, learned: Learned) -> Need { // Fold what a model taught us on a GPU type and tp into the budget parts: static extra and non-static overhead. Need { @@ -464,13 +463,16 @@ mod tests { marlin_extra_gb: 0.0, supported_sm: Vec::new(), memory: None, + overhead_gb: crate::calc::BASE_OVERHEAD_GB, } } #[test] fn metal_pays_no_cuda_overhead() { let (cuda, metal) = (need(&inp(), 1, "12.0"), need(&inp(), 1, "metal")); - assert!((cuda.overhead - metal.overhead - (crate::calc::BASE_OVERHEAD_GB - crate::calc::METAL_OVERHEAD_GB)).abs() < 1e-9); + assert!((cuda.overhead - metal.overhead - (crate::calc::BASE_OVERHEAD_GB - METAL_OVERHEAD_GB)).abs() < 1e-9); + let llama = need(&ChooserInput { overhead_gb: METAL_OVERHEAD_GB, ..inp() }, 1, "12.0"); + assert!((llama.overhead - metal.overhead).abs() < 1e-9); } fn gpu(idx: u32, free_gb: f64, foreign: bool) -> GpuPlacementInput { diff --git a/engine/crates/core/src/configure.rs b/engine/crates/core/src/configure.rs index a0ab324..30f3476 100644 --- a/engine/crates/core/src/configure.rs +++ b/engine/crates/core/src/configure.rs @@ -31,6 +31,7 @@ pub struct ApiOpts { pub interface: Option, pub allow: Vec, pub yes: bool, + pub dangerously_allow_all_inbound: bool, } #[derive(Clone, Debug, PartialEq, Eq)] @@ -349,8 +350,50 @@ fn configure_vpn(opts: &ApiOpts, nets: &[Network]) -> Result { Ok(target.addr().to_string()) } +fn resolve_public_allows(opts: &ApiOpts) -> Result, String> { + // Named clients required; --yes alone must not bind every interface on a leftover allowlist. + if opts.dangerously_allow_all_inbound { + if !opts.allow.is_empty() { + return Err("--dangerously-allow-all-inbound does not mix with --allow".to_string()); + } + return Ok(vec!["0.0.0.0/0".to_string()]); + } + let mut ranges = opts.allow.clone(); + if ranges.is_empty() { + if !interactive() || opts.yes { + return Err( + "public needs who may connect: goethe api expose public --allow /32 --yes\n\ + (to allow every address on purpose: --dangerously-allow-all-inbound --yes)" + .to_string(), + ); + } + if !confirm(RESTRICT_QUESTION, true, false)? { + return Err( + "public needs client addresses (--allow /32), or --dangerously-allow-all-inbound --yes" + .to_string(), + ); + } + let a = ask("Client IPs or ranges to allow, comma separated (e.g. 203.0.113.7/32):", "")?; + ranges = a.split(',').map(str::trim).filter(|r| !r.is_empty()).map(String::from).collect(); + } + if ranges.is_empty() { + return Err( + "public needs who may connect: goethe api expose public --allow /32 --yes".to_string(), + ); + } + for r in &ranges { + security::canonical_cidr(r)?; + if is_everyone(r) { + return Err(format!( + "{r} opens every address; use --dangerously-allow-all-inbound --yes" + )); + } + } + Ok(ranges) +} + fn configure_public(opts: &ApiOpts) -> Result { - // Warn, show the public IP and its API URL, confirm, bind every interface and allow only the named clients. + // Warn, require named clients (or an explicit world-open), confirm, bind every interface. println!("{}: {PUBLIC_WARNING}", crate::term::paint("WARNING", crate::term::Color::Yellow)); let port = nginx::public_port_u16(&goethe_home()); let public = lookup_public_ip(); @@ -358,38 +401,28 @@ fn configure_public(opts: &ApiOpts) -> Result { for line in public_ip_lines(public, port, &nginx::interface_addrs().unwrap_or_default()) { println!("{line}"); } - if !confirm("Expose the API on every network interface anyway?", false, opts.yes)? { - return Err("nothing changed (goethe api expose local or goethe api expose vpn to pick one)".to_string()); + let ranges = resolve_public_allows(opts)?; + if !opts.yes && !interactive() { + return Err( + "public open needs a terminal to confirm (y), or pass --yes with --allow /32".to_string(), + ); } - let mut ranges = opts.allow.clone(); - let mut everyone_chosen = false; - if ranges.is_empty() && interactive() && !opts.yes { - if confirm(RESTRICT_QUESTION, false, false)? { - let a = ask("Client IPs or ranges to allow, comma separated (e.g. 203.0.113.7/32):", "")?; - ranges = a.split(',').map(str::trim).filter(|r| !r.is_empty()).map(String::from).collect(); - } else { - ranges = vec!["0.0.0.0/0".to_string()]; - everyone_chosen = true; - println!("Every address that can reach this machine may connect; each request still needs an API key."); - } + let confirm_q = if opts.dangerously_allow_all_inbound { + "Bind every network interface and allow EVERY client address?" + } else { + "Expose the API on every network interface (0.0.0.0) for the allowlisted clients?" + }; + if !confirm(confirm_q, false, opts.yes)? { + return Err("nothing changed (goethe api expose local or goethe api expose vpn to pick one)".to_string()); } - for r in &ranges { - security::canonical_cidr(r)?; - if is_everyone(r) && !everyone_chosen { - println!("{r} lets every address on the internet try your API keys."); - if !interactive() || !confirm("Allow every address anyway?", false, false)? { - return Err(format!("refusing to allow {r}; name the client addresses instead")); - } - } + if opts.dangerously_allow_all_inbound { + println!("Every address that can reach this machine may connect; each request still needs an API key. Rate limits still apply."); } bind("0.0.0.0")?; allow("127.0.0.1/32", "this machine", true)?; for r in &ranges { allow(r, "public client", true)?; } - if ranges.is_empty() { - println!("No client addresses were added: only addresses already on the allowlist can connect (goethe api allow add /32)."); - } Ok(public_host.unwrap_or_else(|| "0.0.0.0".to_string())) } @@ -470,4 +503,43 @@ mod tests { assert!(is_everyone("0.0.0.0/0") && is_everyone("::/0") && !is_everyone("203.0.113.7/32")); assert_eq!(Net::new("10.1.2.3".parse().unwrap(), 16).to_string(), "10.1.0.0/16"); } + + fn opts(allow: &[&str], yes: bool, dangerously: bool) -> ApiOpts { + ApiOpts { + mode: Some(Mode::Public), + vpn_ip: None, + interface: None, + allow: allow.iter().map(|s| (*s).to_string()).collect(), + yes, + dangerously_allow_all_inbound: dangerously, + } + } + + #[test] + fn public_yes_without_allow_is_refused() { + let err = resolve_public_allows(&opts(&[], true, false)).unwrap_err(); + assert!(err.contains("--allow"), "{err}"); + } + + #[test] + fn public_allow_is_accepted() { + let got = resolve_public_allows(&opts(&["203.0.113.7/32"], true, false)).unwrap(); + assert_eq!(got, vec!["203.0.113.7/32".to_string()]); + } + + #[test] + fn public_everyone_cidr_needs_dangerously_flag() { + let err = resolve_public_allows(&opts(&["0.0.0.0/0"], true, false)).unwrap_err(); + assert!(err.contains("dangerously-allow-all-inbound"), "{err}"); + } + + #[test] + fn public_dangerously_opens_everyone_and_rejects_mixed_allow() { + assert_eq!( + resolve_public_allows(&opts(&[], true, true)).unwrap(), + vec!["0.0.0.0/0".to_string()] + ); + let err = resolve_public_allows(&opts(&["203.0.113.7/32"], true, true)).unwrap_err(); + assert!(err.contains("does not mix"), "{err}"); + } } diff --git a/engine/crates/core/src/engine_overhead.rs b/engine/crates/core/src/engine_overhead.rs index 0f3fe05..23884d9 100644 --- a/engine/crates/core/src/engine_overhead.rs +++ b/engine/crates/core/src/engine_overhead.rs @@ -99,7 +99,8 @@ pub fn model_runtime(home: &Path, model_id: &str) -> String { } pub fn card_runtime(card: &toml::Value) -> String { - crate::engines::of_card(card).map_or_else(|_| "unknown".to_string(), |e| e.id.to_string()) + // Spelling on the card (ternary ≠ launch id sglang; learned rows key on the spelling). + card.get("runtime").and_then(|v| v.as_str()).unwrap_or("sglang").to_string() } pub fn parse_pre_load_gb(log_text: &str) -> Option { @@ -258,6 +259,20 @@ mod tests { assert!(parse_shortfall_gb(log).unwrap() > 0.0); } + #[test] + fn ternary_card_spelling_stays_ternary_for_learned_rows() { + let tern: toml::Value = toml::from_str("runtime = \"ternary\"").unwrap(); + let plain: toml::Value = toml::from_str("runtime = \"sglang\"").unwrap(); + assert_eq!(card_runtime(&tern), "ternary"); + assert_eq!(card_runtime(&plain), "sglang"); + assert_eq!(crate::engines::of_card(&tern).unwrap().id, "sglang"); + let td = tempfile::tempdir().unwrap(); + let con = crate::db::open(td.path()).unwrap(); + store_learned(&con, "m", "GPU", 1, Kind::Overhead, 2.5, "ternary").unwrap(); + assert_eq!(read_learned(&con, "m", "GPU", 1, "ternary").overhead_gb, 2.5); + assert_eq!(read_learned(&con, "m", "GPU", 1, "sglang").overhead_gb, 0.0, "sglang rows stay separate"); + } + #[test] fn a_figure_learned_under_a_plain_runtime_never_blocks_the_kernel_pack() { let dir = tempfile::tempdir().unwrap(); diff --git a/engine/crates/core/src/engines/goethe_engine.rs b/engine/crates/core/src/engines/goethe_engine.rs index 86ecd60..18ba833 100644 --- a/engine/crates/core/src/engines/goethe_engine.rs +++ b/engine/crates/core/src/engines/goethe_engine.rs @@ -1,11 +1,12 @@ // goethe-engine: a signed kernel pack plus the weights its verification lists, served by python -m goethe_engine. -use super::{card_env, kept_env, placement_env, resolve_python, EngineSpec}; +use super::{card_env, engine_key_env, kept_env, placement_env, resolve_python, EngineSpec}; use std::path::PathBuf; use std::process::Command; pub const ENGINE: super::Engine = super::Engine { id: "goethe-engine", + overhead_gb: crate::calc::METAL_OVERHEAD_GB, layer: Some(crate::pyenv::Layer::Engine), params: None, command: goethe_engine_command, @@ -54,6 +55,7 @@ fn goethe_engine_command(spec: &EngineSpec) -> Result { .envs(card_env(spec)) .envs(placement_env(spec)) .env_remove("GOETHE_ENGINE_ALLOW_UNSIGNED"); + engine_key_env(&mut cmd, spec.engine_key); Ok(cmd) } diff --git a/engine/crates/core/src/engines/llama_cpp.rs b/engine/crates/core/src/engines/llama_cpp.rs index 75fd235..a30fbaa 100644 --- a/engine/crates/core/src/engines/llama_cpp.rs +++ b/engine/crates/core/src/engines/llama_cpp.rs @@ -1,6 +1,6 @@ // llama.cpp: llama-server on one GGUF, one GPU, OpenAI-compatible; the universal engine for GGUF models. -use super::{card_env, kept_env, placement_env, EngineSpec}; +use super::{card_env, engine_key_env, kept_env, placement_env, EngineSpec}; use std::collections::HashMap; use std::io::Read; use std::path::{Path, PathBuf}; @@ -24,6 +24,7 @@ type Get<'a> = &'a dyn Fn(&str) -> Result, String>; pub const ENGINE: super::Engine = super::Engine { id: "llama.cpp", + overhead_gb: crate::calc::METAL_OVERHEAD_GB, layer: None, params: Some(&[ "temperature", "top_p", "top_k", "min_p", "max_tokens", "max_completion_tokens", "n", "stop", "seed", @@ -382,8 +383,6 @@ fn llama_command(spec: &EngineSpec) -> Result { .arg(gguf_in(spec.model_path, spec.card, None)?) .args(["--host", "127.0.0.1", "--port"]) .arg(spec.port.to_string()) - .arg("--api-key") - .arg(spec.engine_key) .arg("--alias") .arg(spec.served_name) .arg("-c") @@ -404,6 +403,7 @@ fn llama_command(spec: &EngineSpec) -> Result { .envs(kept_env()) .envs(card_env(spec)) .envs(placement_env(spec)); + engine_key_env(&mut cmd, spec.engine_key); Ok(cmd) } @@ -503,8 +503,11 @@ mod tests { assert_eq!((at("--host").as_str(), at("--alias").as_str(), at("-np").as_str()), ("127.0.0.1", "org/m", "4")); assert_eq!((at("-c"), at("--kv-unified-per-slot")), (crate::chooser::kv_tokens(4096, 4).to_string(), "4096".to_string())); assert!(!a.iter().any(|x| x == "--embeddings")); + assert!(!a.iter().any(|x| x == "--api-key"), "key stays off argv: {a:?}"); let env = |k: &str| c.get_envs().find(|(n, _)| *n == k).and_then(|(_, v)| v).map(|v| v.to_string_lossy().into_owned()); assert_eq!(env("CUDA_VISIBLE_DEVICES").as_deref(), Some("2")); + assert_eq!(env("LLAMA_API_KEY").as_deref().map(|s| s.len()), Some(32)); + assert_eq!(env(super::super::ENGINE_KEY_ENV).as_deref().map(|s| s.len()), Some(32)); let emb: toml::Value = toml::from_str("runtime = \"llama.cpp\"\ntask = \"embed\"").unwrap(); assert!(command(&test_spec(d.path(), &emb, &[2])).unwrap().get_args().any(|x| x == "--embeddings")); assert!(command(&test_spec(d.path(), &card, &[2, 3])).unwrap_err().contains("one GPU")); diff --git a/engine/crates/core/src/engines/mod.rs b/engine/crates/core/src/engines/mod.rs index 591d49c..b26584b 100644 --- a/engine/crates/core/src/engines/mod.rs +++ b/engine/crates/core/src/engines/mod.rs @@ -11,22 +11,23 @@ use std::process::Command; pub struct Engine { pub id: &'static str, + pub overhead_gb: f64, pub layer: Option, pub params: Option<&'static [&'static str]>, pub command: fn(&EngineSpec) -> Result, } -pub const ENGINES: &[&Engine] = &[&goethe_engine::ENGINE, &ninfer::ENGINE, &sglang::SGLANG, &sglang::TERNARY, &llama_cpp::ENGINE]; +pub const ENGINES: &[&Engine] = &[&goethe_engine::ENGINE, &ninfer::ENGINE, &sglang::SGLANG, &llama_cpp::ENGINE]; const ENVELOPE: &[&str] = &["model", "messages", "prompt", "input", "stream", "stream_options", "user"]; pub fn by_id(id: &str) -> Result<&'static Engine, String> { + let id = if id == "ternary" { sglang::SGLANG.id } else { id }; let ids: Vec<&str> = ENGINES.iter().map(|e| e.id).collect(); ENGINES.iter().copied().find(|e| e.id == id).ok_or_else(|| format!("engine {id} is not one goethe runs ({})", ids.join(", "))) } pub fn of_card(card: &toml::Value) -> Result<&'static Engine, String> { - // The card's runtime; a card without one is SGLang's. by_id(card.get("runtime").and_then(|v| v.as_str()).unwrap_or(sglang::SGLANG.id)) } @@ -164,12 +165,35 @@ fn compiler_env() -> Vec<(String, String)> { crate::host_compiler::engine_env(crate::host_compiler::find().as_ref(), inherited.as_deref()) } +pub const ENGINE_KEY_ENV: &str = "GOETHE_ENGINE_API_KEY"; + +fn engine_key_env(cmd: &mut Command, key: &str) { + // Owner-only in /proc//environ; never put the secret on argv (world-readable cmdline). + cmd.env(ENGINE_KEY_ENV, key).env("LLAMA_API_KEY", key); +} + fn card_env(spec: &EngineSpec) -> Vec<(String, String)> { - // Card env table and card pythonpath. + // Card env table and card pythonpath. Loader hooks from a card never enter the process. + let (out, dropped) = card_env_parts(spec); + if !dropped.is_empty() { + eprintln!( + "goethe: card [env] dropped {}: loader hooks never enter the engine", + dropped.join(", ") + ); + } + out +} + +fn card_env_parts(spec: &EngineSpec) -> (Vec<(String, String)>, Vec) { let home = spec.home; let mut out = Vec::new(); + let mut dropped = Vec::new(); if let Some(table) = spec.card.get("env").and_then(|v| v.as_table()) { for (k, v) in table { + if k.starts_with("LD_") || k.starts_with("DYLD_") { + dropped.push(k.clone()); + continue; + } if let Some(s) = v.as_str() { out.push((k.clone(), expand_home(s, home))); } @@ -186,7 +210,7 @@ fn card_env(spec: &EngineSpec) -> Vec<(String, String)> { if !paths.is_empty() { out.push(("PYTHONPATH".to_string(), paths.join(":"))); } - out + (out, dropped) } #[cfg(test)] @@ -235,7 +259,6 @@ mod tests { assert_eq!(a[n + 1], "org/m"); let q = a.iter().position(|x| x == "--uvicorn-access-log-exclude-prefixes").unwrap(); assert_eq!(&a[q + 1..q + 4], ["/health", "/v1/models", "/get_model_info"]); - assert!(a[q + 4].starts_with("--")); assert!(!a.iter().any(|x| x == "--disable-radix-cache")); } @@ -262,13 +285,25 @@ mod tests { assert_eq!(e.get("CUDA_VISIBLE_DEVICES"), Some(&Some("3,2".to_string()))); assert_eq!(e.get(INSTANCE_ENV), Some(&Some("inst-x".to_string()))); assert_eq!(e.get("SPT_NOENV"), Some(&Some("1".to_string()))); - assert_eq!(e.get("PYTHONPATH"), Some(&Some("/h/x".to_string()))); + assert_eq!(e.get("PYTHONPATH"), Some(&Some("/h/lib:/h/x".to_string()))); assert_eq!(e.get("A"), Some(&Some("1".to_string()))); assert!(!e.contains_key("TERNARY_PACK_DIR")); assert!(!e.contains_key("CARGO_MANIFEST_DIR")); let a = args(&plain); - let k = a.iter().position(|x| x == "--api-key").unwrap(); - assert_eq!(a[k + 1].len(), 32); + assert!(!a.iter().any(|x| x == "--api-key"), "{a:?}"); + assert_eq!(e.get(ENGINE_KEY_ENV).and_then(|v| v.as_ref().map(|s| s.len())), Some(32)); + let card: toml::Value = toml::from_str("[env]\nLD_PRELOAD = \"/x.so\"\nDYLD_INSERT_LIBRARIES = \"/y.dylib\"\nOK = \"1\"").unwrap(); + let (kept, dropped) = card_env_parts(&EngineSpec { + model_path: Path::new("/h/models/m"), + tp: 1, + ..test_spec(Path::new("/h"), &card, &[3, 2]) + }); + assert!(kept.iter().any(|(k, v)| k == "OK" && v == "1")); + assert!(!kept.iter().any(|(k, _)| k.starts_with("LD_") || k.starts_with("DYLD_"))); + assert!(dropped.contains(&"LD_PRELOAD".to_string()) && dropped.contains(&"DYLD_INSERT_LIBRARIES".to_string()), "{dropped:?}"); + let blocked = envs(&spec_cmd("[env]\nLD_PRELOAD = \"/x.so\"\nDYLD_INSERT_LIBRARIES = \"/y.dylib\"\nOK = \"1\"", 1)); + assert!(!blocked.contains_key("LD_PRELOAD") && !blocked.contains_key("DYLD_INSERT_LIBRARIES")); + assert_eq!(blocked.get("OK"), Some(&Some("1".into()))); } #[test] @@ -303,8 +338,9 @@ mod tests { assert!(err.contains("not allowed"), "{err}"); assert!(check_extra_args(&strs(&["--tokenizer-worker-num=2"])).is_err()); let single = args(&spec_cmd("extra_sglang_args = [\"--tokenizer-worker-num\", \"1\"]", 1)); - assert!(single.iter().any(|a| a == "--api-key")); + assert!(!single.iter().any(|a| a == "--api-key")); assert!(single.windows(2).any(|w| w[0] == "--host" && w[1] == "127.0.0.1")); + assert!(single.iter().any(|a| a == "goethe_sglang_boot")); } #[test] @@ -316,6 +352,16 @@ mod tests { assert_ne!(a, b); } + #[test] + fn light_engines_pay_half_a_gig_on_cuda_sglang_pays_base() { + assert!((by_id("llama.cpp").unwrap().overhead_gb - crate::calc::METAL_OVERHEAD_GB).abs() < 1e-9); + assert!((by_id("goethe-engine").unwrap().overhead_gb - crate::calc::METAL_OVERHEAD_GB).abs() < 1e-9); + assert!((by_id("ninfer").unwrap().overhead_gb - crate::calc::METAL_OVERHEAD_GB).abs() < 1e-9); + assert!((by_id("sglang").unwrap().overhead_gb - crate::calc::BASE_OVERHEAD_GB).abs() < 1e-9); + assert!((by_id("ternary").unwrap().overhead_gb - crate::calc::BASE_OVERHEAD_GB).abs() < 1e-9); + assert_eq!(by_id("ternary").unwrap().id, "sglang", "ternary launches as sglang"); + } + #[test] fn hybrid_models_get_the_state_slots_goethe_budgeted() { let dir = tempfile::tempdir().unwrap(); @@ -335,10 +381,13 @@ mod tests { #[test] fn every_runtime_a_card_was_ever_written_with_resolves_and_an_unknown_one_is_refused() { assert_eq!(of_card(&toml::from_str("").unwrap()).unwrap().id, "sglang"); - for id in ["sglang", "ternary", "ninfer", "goethe-engine", "llama.cpp"] { + for id in ["sglang", "ninfer", "goethe-engine", "llama.cpp"] { assert_eq!(of_card(&toml::from_str(&format!("runtime = \"{id}\"")).unwrap()).unwrap().id, id); } - assert!(args(&spec_cmd("", 1)).iter().any(|a| a == "sglang.launch_server")); + let ternary: toml::Value = toml::from_str("runtime = \"ternary\"").unwrap(); + assert_eq!(of_card(&ternary).unwrap().id, "sglang"); + assert_eq!(crate::engine_overhead::card_runtime(&ternary), "ternary"); + assert!(args(&spec_cmd("", 1)).iter().any(|a| a == "goethe_sglang_boot")); assert!(of_card(&toml::from_str("runtime = \"vllm\"").unwrap()).map(|e| e.id).unwrap_err().contains("not one goethe runs")); } diff --git a/engine/crates/core/src/engines/ninfer.rs b/engine/crates/core/src/engines/ninfer.rs index c7bd1f3..cf2dd53 100644 --- a/engine/crates/core/src/engines/ninfer.rs +++ b/engine/crates/core/src/engines/ninfer.rs @@ -1,11 +1,12 @@ // NInfer: the pinned .ninfer artifact (fetch, verify, head patch) and its ninfer-serve command line. -use super::{kept_env, placement_env, EngineSpec}; +use super::{engine_key_env, kept_env, placement_env, EngineSpec}; use std::path::{Path, PathBuf}; use std::process::Command; pub const ENGINE: super::Engine = super::Engine { id: "ninfer", + overhead_gb: crate::calc::METAL_OVERHEAD_GB, layer: None, params: None, command: ninfer_command, @@ -55,19 +56,7 @@ pub fn accept_ninfer(len: u64, digest: &str) -> Result<(), String> { } pub fn sha256_file(path: &Path) -> Result { - use sha2::{Digest, Sha256}; - use std::io::Read; - let mut f = std::fs::File::open(path).map_err(|e| format!("{}: {e}", path.display()))?; - let mut h = Sha256::new(); - let mut buf = vec![0u8; 1 << 20]; - loop { - let n = f.read(&mut buf).map_err(|e| format!("{}: {e}", path.display()))?; - if n == 0 { - break; - } - h.update(&buf[..n]); - } - Ok(format!("{:x}", h.finalize())) + crate::update::sha256_file(path).map_err(|e| format!("{}: {e}", path.display())) } pub fn ninfer_in(dir: &Path) -> Option { @@ -303,6 +292,7 @@ fn ninfer_command(spec: &EngineSpec) -> Result { cmd.env_clear() .envs(kept_env()) .envs(placement_env(spec)); + engine_key_env(&mut cmd, spec.engine_key); Ok(cmd) } diff --git a/engine/crates/core/src/engines/sglang.rs b/engine/crates/core/src/engines/sglang.rs index ee8e6c9..e041f10 100644 --- a/engine/crates/core/src/engines/sglang.rs +++ b/engine/crates/core/src/engines/sglang.rs @@ -1,37 +1,27 @@ -// SGLang: python -m sglang.launch_server with Goethe-owned flags, card and load extras; ternary is SGLang plus its kernel env. +// SGLang launch: Goethe-owned flags, card extras; ternary cards are the same command plus TERNARY_* env. -use super::{arch_env, card_env, compiler_env, default_cuda_home, engine_path, expand_home, kept_env, placement_env, resolve_python, EngineSpec}; +use super::{ + arch_env, card_env, compiler_env, default_cuda_home, engine_key_env, engine_path, expand_home, kept_env, placement_env, + resolve_python, EngineSpec, +}; +use crate::sglang_infer::RESERVED_FLAGS; use std::process::Command; pub const SGLANG: super::Engine = super::Engine { id: "sglang", + overhead_gb: crate::calc::BASE_OVERHEAD_GB, layer: Some(crate::pyenv::Layer::Sglang), params: None, - command: |spec| Ok(sglang_command(spec)), + command: |spec| { + let mut cmd = sglang_command(spec); + apply_ternary(spec, &mut cmd)?; + Ok(cmd) + }, }; -pub const TERNARY: super::Engine = super::Engine { id: "ternary", command: ternary_command, ..SGLANG }; - -const OWNED_FLAGS: &[&str] = &[ - "--host", - "--port", - "--tp-size", - "--tensor-parallel-size", - "--mem-fraction-static", - "--context-length", - "--max-running-requests", - "--max-mamba-cache-size", - "--swa-full-tokens-ratio", - "--api-key", - "--served-model-name", - "--pack", - "--weights", - "--tokenizer", -]; const QUIET_PATHS: &[&str] = &["/health", "/v1/models", "/get_model_info"]; pub fn check_extra_args(args: &[String]) -> Result<(), String> { - // Refuse Goethe owned flags, and tokenizer workers above 1 because that mode cannot take an API key. if multi_tokenizer(args) { return Err("extra SGLang arg --tokenizer-worker-num greater than 1 is not allowed: that mode cannot take an API key, so Goethe will not listen on TCP".into()); } @@ -40,10 +30,8 @@ pub fn check_extra_args(args: &[String]) -> Result<(), String> { if !name.starts_with("--") || name.len() < 3 { continue; } - if let Some(owned) = OWNED_FLAGS.iter().find(|o| o.starts_with(name)) { - return Err(format!( - "extra SGLang arg {arg} is not allowed: Goethe sets {owned} itself (use goethe load options instead)" - )); + if let Some((owned, _)) = RESERVED_FLAGS.iter().find(|(f, _)| *f == name || f.starts_with(name)) { + return Err(format!("extra SGLang arg {arg} is not allowed: Goethe sets {owned} itself (use goethe load options instead)")); } } Ok(()) @@ -57,7 +45,6 @@ pub fn card_extra_args(card: &toml::Value) -> Vec { } fn multi_tokenizer(args: &[String]) -> bool { - // SGLang refuses --api-key with --tokenizer-worker-num > 1, so such engines run keyless on loopback. args.iter().enumerate().any(|(i, a)| { let value = match a.strip_prefix("--tokenizer-worker-num") { Some("") => args.get(i + 1).map(String::as_str), @@ -69,13 +56,14 @@ fn multi_tokenizer(args: &[String]) -> bool { } fn memory_args(spec: &EngineSpec, extras: &[String]) -> Vec { - // The window ratio, state slots and chunk size the budget assumed; a chunk size the user set wins. let Some(m) = crate::memory_spec::installed(spec.model_path, spec.card, spec.load_extra) else { return Vec::new() }; let args = crate::chooser::engine_args(&m, spec.ctx, spec.max_conc); let user_chunk = extras.iter().any(|a| a.starts_with("--chunked-prefill-size")); args.chunks(2).filter(|p| !(user_chunk && p[0] == "--chunked-prefill-size")).flatten().cloned().collect() } +const SGLANG_BOOT: &str = "import os,runpy,sys\nsys.argv=['sglang.launch_server',*sys.argv[1:],'--api-key',os.environ['GOETHE_ENGINE_API_KEY']]\nrunpy.run_module('sglang.launch_server',run_name='__main__')\n"; + fn graph_args(max_conc: u64, extras: &[String]) -> Vec { // Decode CUDA graphs captured only up to the batch the model runs (its concurrency), not SGLang's default range: a shorter load and less VRAM; graph flags the user set win. let theirs = extras.iter().any(|a| a.starts_with("--cuda-graph") || a == "--disable-cuda-graph"); @@ -86,10 +74,15 @@ fn graph_args(max_conc: u64, extras: &[String]) -> Vec { } fn sglang_command(spec: &EngineSpec) -> Command { - // Goethe flags, then checked card and load extras, in a cleared env; probe paths stay out of the access log. + let lib = spec.home.join("lib"); + let boot = lib.join("goethe_sglang_boot.py"); + if !boot.is_file() { + let _ = std::fs::create_dir_all(&lib); + let _ = crate::platform::write_private(&boot, SGLANG_BOOT.as_bytes()); + } let mut cmd = Command::new(resolve_python(spec.home)); cmd.arg("-m") - .arg("sglang.launch_server") + .arg("goethe_sglang_boot") .arg("--model-path") .arg(spec.model_path) .arg("--host") @@ -111,7 +104,6 @@ fn sglang_command(spec: &EngineSpec) -> Command { let extras: Vec = card_extra_args(spec.card).into_iter().chain(spec.load_extra.iter().cloned()).collect(); cmd.args(memory_args(spec, &extras)); cmd.args(graph_args(spec.max_conc, &extras)); - cmd.arg("--api-key").arg(spec.engine_key); if let Some(dtype) = spec .card .get("kv_cache_dtype") @@ -126,6 +118,10 @@ fn sglang_command(spec: &EngineSpec) -> Command { } cmd.args(&extras); let card_vars = card_env(spec); + let mut py_path = lib.display().to_string(); + if let Some((_, v)) = card_vars.iter().find(|(k, _)| k == "PYTHONPATH") { + py_path = format!("{py_path}:{v}"); + } cmd.env_clear() .envs(kept_env()) .env("PATH", engine_path(spec.home)) @@ -135,27 +131,35 @@ fn sglang_command(spec: &EngineSpec) -> Command { .envs(compiler_env()) .envs(card_vars) .envs(placement_env(spec)) + .env("PYTHONPATH", py_path) .env("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True") .env("SPT_NOENV", "1"); + engine_key_env(&mut cmd, spec.engine_key); cmd } -fn ternary_command(spec: &EngineSpec) -> Result { - // SGLang with the card's top-level TERNARY_* keys in its env, once the ternary kernels are installed. +fn apply_ternary(spec: &EngineSpec, cmd: &mut Command) -> Result<(), String> { + if spec.card.get("runtime").and_then(|v| v.as_str()) != Some("ternary") { + return Ok(()); + } crate::runtime::require_ternary_runtime(spec.home)?; - let mut cmd = sglang_command(spec); for (k, v) in spec.card.as_table().into_iter().flatten() { if let (true, Some(s)) = (k.starts_with("TERNARY_"), v.as_str()) { cmd.env(k, expand_home(s, spec.home)); } } - Ok(cmd) + Ok(()) } #[cfg(test)] mod tests { use super::super::test_spec; - use super::graph_args; + use super::{check_extra_args, graph_args}; + use std::ffi::OsStr; + + fn arg_vec(cmd: &std::process::Command) -> Vec { + cmd.get_args().map(|a| a.to_string_lossy().into_owned()).collect() + } #[test] fn graphs_are_captured_up_to_the_concurrency_unless_the_user_says_otherwise() { @@ -176,5 +180,55 @@ mod tests { let cmd = super::super::command(&spec).unwrap(); let want = format!("{}/p", home.path().display()); assert!(cmd.get_envs().any(|(k, v)| k == "TERNARY_PACK_DIR" && v.is_some_and(|v| v.to_string_lossy() == want))); + assert_eq!(super::super::of_card(&card).unwrap().id, "sglang"); + assert_eq!(crate::engine_overhead::card_runtime(&card), "ternary"); + } + + #[test] + fn plain_sglang_does_not_require_ternary_kernels() { + let home = tempfile::tempdir().unwrap(); + let card: toml::Value = toml::from_str("runtime = \"sglang\"\nTERNARY_PACK_DIR = \"/should-not-apply\"").unwrap(); + let cmd = super::super::command(&test_spec(home.path(), &card, &[0])).unwrap(); + assert!(!cmd.get_envs().any(|(k, _)| k == OsStr::new("TERNARY_PACK_DIR"))); + } + + #[test] + fn ternary_argv_matches_plain_sglang_only_env_differs() { + let home = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(crate::runtime::ternary_kernels_dir(home.path())).unwrap(); + let plain: toml::Value = toml::from_str("runtime = \"sglang\"").unwrap(); + let tern: toml::Value = + toml::from_str("runtime = \"ternary\"\nTERNARY_PACK_DIR = \"$GOETHE_HOME/p\"\nTERNARY_MODEL_DIR = \"$GOETHE_HOME/b\"").unwrap(); + let a = super::super::command(&test_spec(home.path(), &plain, &[0])).unwrap(); + let b = super::super::command(&test_spec(home.path(), &tern, &[0])).unwrap(); + assert_eq!(arg_vec(&a), arg_vec(&b), "fold must not change the SGLang argv"); + assert_eq!(a.get_program(), b.get_program()); + let mut tern_envs: Vec<_> = b + .get_envs() + .filter(|(k, _)| k.to_string_lossy().starts_with("TERNARY_")) + .map(|(k, v)| (k.to_string_lossy().into_owned(), v.map(|v| v.to_string_lossy().into_owned()))) + .collect(); + tern_envs.sort(); + assert_eq!( + tern_envs, + [ + ("TERNARY_MODEL_DIR".into(), Some(format!("{}/b", home.path().display()))), + ("TERNARY_PACK_DIR".into(), Some(format!("{}/p", home.path().display()))), + ] + ); + assert!(!a.get_envs().any(|(k, _)| k.to_string_lossy().starts_with("TERNARY_"))); + } + + #[test] + fn reserved_table_rejects_former_owned_and_engine_pack_flags() { + for bad in [ + "--host", "--port", "--tp-size", "--tensor-parallel-size", "--mem-fraction-static", "--context-length", + "--max-running-requests", "--max-mamba-cache-size", "--swa-full-tokens-ratio", "--api-key", "--served-model-name", + "--pack", "--weights", "--tokenizer", "--model-path", "--model", + ] { + let err = check_extra_args(&[bad.to_string(), "1".into()]).unwrap_err(); + assert!(err.contains("not allowed"), "{bad}: {err}"); + } + check_extra_args(&["--chunked-prefill-size".into(), "4096".into(), "--trust-remote-code".into()]).unwrap(); } } diff --git a/engine/crates/core/src/hf.rs b/engine/crates/core/src/hf.rs index a6125f9..cc60af2 100644 --- a/engine/crates/core/src/hf.rs +++ b/engine/crates/core/src/hf.rs @@ -1,7 +1,6 @@ // Hugging Face download and verify for goethe install; goethe never logs in or sends credentials. use serde::{Deserialize, Serialize}; -use sha2::{Digest, Sha256}; use std::collections::{BTreeSet, HashMap}; use std::fs; use std::io::{Read, Write}; @@ -835,18 +834,10 @@ fn verify_one_file(path: &Path, expect_sha: &str) -> Result Result { - let mut f = fs::File::open(path).map_err(|e| e.to_string())?; - let mut h = Sha256::new(); - let mut buf = vec![0u8; 1024 * 1024]; - loop { - let n = f.read(&mut buf).map_err(|e| e.to_string())?; - if n == 0 { - break; - } - h.update(&buf[..n]); - HASHED.fetch_add(n as u64, Ordering::Relaxed); - } - Ok(format!("{:x}", h.finalize())) + let meta = fs::metadata(path).map_err(|e| e.to_string())?; + let digest = crate::update::sha256_file(path)?; + HASHED.fetch_add(meta.len(), Ordering::Relaxed); + Ok(digest) } fn file_identity(path: &Path) -> Option<(u64, i64)> { diff --git a/engine/crates/core/src/hw.rs b/engine/crates/core/src/hw.rs index 5b54782..6212251 100644 --- a/engine/crates/core/src/hw.rs +++ b/engine/crates/core/src/hw.rs @@ -143,6 +143,7 @@ pub fn run_pick_gpus(tp: Option, model: Option<&str>) -> Result<(), String> marlin_extra_gb: 0.0, supported_sm: Vec::new(), memory: None, + overhead_gb: crate::calc::BASE_OVERHEAD_GB, }; let fit = view.fit("", &empty, Some(t)); (String::new(), fit.map(|f| (format!("tp={} on GPU {}", f.tp, crate::placement::format_gpu_list(&f.gpus)), f.gpus))) diff --git a/engine/crates/core/src/install.rs b/engine/crates/core/src/install.rs index 843616f..10d7cc8 100644 --- a/engine/crates/core/src/install.rs +++ b/engine/crates/core/src/install.rs @@ -2519,7 +2519,21 @@ mod tests { crate::engine_overhead::store_learned(&con, "nvidia/Big-NVFP4", "NVIDIA GeForce RTX 5090", 1, crate::engine_overhead::Kind::Overhead, 4.8, "llama.cpp").unwrap(); assert!(spare(card(tight)).is_ok(), "a llama.cpp row never reaches the pack budget"); let view = view_of(home.path(), card(tight)); - let input = crate::chooser::ChooserInput { window_gb_per_request: 0.0, static_vram_gb: 20.0, kv_bytes_per_token: 131072, num_attention_heads: 32, num_kv_heads: 8, attention: "gqa".into(), max_ctx: 8192, ctx: 4096, max_conc: 8, marlin_extra_gb: 0.0, supported_sm: vec!["12.0".into()], memory: None }; + let input = crate::chooser::ChooserInput { + window_gb_per_request: 0.0, + static_vram_gb: 20.0, + kv_bytes_per_token: 131072, + num_attention_heads: 32, + num_kv_heads: 8, + attention: "gqa".into(), + max_ctx: 8192, + ctx: 4096, + max_conc: 8, + marlin_extra_gb: 0.0, + supported_sm: vec!["12.0".into()], + memory: None, + overhead_gb: crate::calc::METAL_OVERHEAD_GB, + }; assert!(view.fit_as("nvidia/Big-NVFP4", &input, None, "llama.cpp").is_err(), "the same row does bind the llama.cpp runtime"); } diff --git a/engine/crates/core/src/menu.rs b/engine/crates/core/src/menu.rs index 6582d9e..6b52143 100644 --- a/engine/crates/core/src/menu.rs +++ b/engine/crates/core/src/menu.rs @@ -830,7 +830,14 @@ fn access_page(tty: &mut Tty) -> Result<(), String> { tty.resume()?; } Some(1) => tty.inside(|| { - crate::configure::run_api(crate::configure::ApiOpts { mode: None, vpn_ip: None, interface: None, allow: Vec::new(), yes: false }) + crate::configure::run_api(crate::configure::ApiOpts { + mode: None, + vpn_ip: None, + interface: None, + allow: Vec::new(), + yes: false, + dangerously_allow_all_inbound: false, + }) })?, Some(_) => { let mut page = SharePage::new(); @@ -944,13 +951,34 @@ impl SharePage { }; let allow: Vec = self.allow.split([',', ' ']).map(str::trim).filter(|c| !c.is_empty()).map(str::to_string).collect(); let home = goethe_home(); - match share::open(&home, share::OpenArgs { secs, keys: std::slice::from_ref(&name), allow: &allow, port: None }) { + match share::open( + &home, + share::OpenArgs { + secs, + keys: std::slice::from_ref(&name), + allow: &allow, + port: None, + dangerously_allow_all_inbound: false, + }, + ) { Ok(o) => { self.reload(); self.key.clear(); self.allow.clear(); let warn = o.warnings.first().map(|w| format!(" Note: {w}")).unwrap_or_default(); - self.say(format!("Shared {name} until {}; send {} with the key.{warn}", share::local_time(share::now() + secs), o.cert.display()), false); + let clients = if o.share.allow.is_empty() { + String::new() + } else { + format!(" Only: {}.", o.share.allow.join(", ")) + }; + self.say( + format!( + "Shared {name} until {}; send {} with the key.{clients}{warn}", + share::local_time(share::now() + secs), + o.cert.display() + ), + false, + ); } Err(e) => self.say(e, true), } diff --git a/engine/crates/core/src/models.rs b/engine/crates/core/src/models.rs index 4395784..145b018 100644 --- a/engine/crates/core/src/models.rs +++ b/engine/crates/core/src/models.rs @@ -36,7 +36,7 @@ struct ModelRow { #[derive(Debug, Clone, Copy, PartialEq)] enum Verdict { Unknown, - Fits { per_gpu: f64, total: f64 }, + Fits { per_gpu: f64, total: f64, ctx: u64, max_conc: u64 }, TooLarge, Unsupported, } @@ -45,7 +45,9 @@ impl Verdict { fn label(&self) -> String { match self { Verdict::Unknown => "unknown (run goethe doctor --hardware)".to_string(), - Verdict::Fits { per_gpu, .. } => format!("fits: {per_gpu:.1} GB per GPU"), + Verdict::Fits { per_gpu, ctx, max_conc, .. } => { + format!("fits: {per_gpu:.1} GB per GPU (ctx {ctx}, {max_conc} concurrent)") + } Verdict::TooLarge => "does not fit: not enough GPU memory".to_string(), Verdict::Unsupported => "does not fit: unsupported architecture".to_string(), } @@ -135,11 +137,22 @@ pub fn run_list(json: bool, installed_only: bool, available_only: bool, all: boo id: row.model_id, status: "available".to_string(), gb: row.vram_per_gpu_gb * row.tp as f64, - fit: Verdict::Fits { per_gpu: row.vram_per_gpu_gb, total: row.vram_per_gpu_gb * row.tp as f64 }.label(), + fit: Verdict::Fits { + per_gpu: row.vram_per_gpu_gb, + total: row.vram_per_gpu_gb * row.tp as f64, + ctx: row.ctx, + max_conc: row.max_conc, + } + .label(), custom: None, optimised: None, runtime: Some(crate::engines::ninfer::ENGINE.id.to_string()), - verdict: Verdict::Fits { per_gpu: row.vram_per_gpu_gb, total: row.vram_per_gpu_gb * row.tp as f64 }, + verdict: Verdict::Fits { + per_gpu: row.vram_per_gpu_gb, + total: row.vram_per_gpu_gb * row.tp as f64, + ctx: row.ctx, + max_conc: row.max_conc, + }, }); } } @@ -247,7 +260,12 @@ fn verdict(view: Option<&View>, id: &str, input: &ChooserInput, pinned: Option Verdict::Fits { per_gpu: f.need.total(), total: f.need.total() * f.tp as f64 }, + Ok(f) => Verdict::Fits { + per_gpu: f.need.total(), + total: f.need.total() * f.tp as f64, + ctx: f.ctx, + max_conc: f.max_conc, + }, Err(_) => Verdict::TooLarge, } } @@ -461,6 +479,7 @@ fn installed_rows(home: &Path, con: &Connection) -> Result, Strin marlin_extra_gb: install::card_marlin_extra_gb(&card), supported_sm: install::card_supported_sm(&card), memory: crate::memory_spec::installed(&home.join(r.get::<_, String>(11)?), &card, &[]), + overhead_gb: crate::engines::of_card(&card).map_or(crate::calc::BASE_OVERHEAD_GB, |e| e.overhead_gb), }, &home.join(r.get::<_, String>(11)?), &card), }) }) @@ -873,9 +892,25 @@ mod tests { assert_eq!(cap_screen("a\nb\n", 40), "a\nb\n"); } + #[test] + fn fit_label_names_ctx_and_concurrency() { + let v = Verdict::Fits { per_gpu: 23.3, total: 23.3, ctx: 8192, max_conc: 4 }; + let label = v.label(); + assert!(label.contains("23.3 GB per GPU") && label.contains("ctx 8192") && label.contains("4 concurrent"), "{label}"); + } + #[test] fn a_gated_catalog_model_is_marked_and_explained() { - let row = |id: &str, fit: String| ModelRow { id: id.into(), status: "available".into(), gb: 5.0, fit, custom: None, optimised: None, runtime: None, verdict: Verdict::Fits { per_gpu: 5.0, total: 5.0 } }; + let row = |id: &str, fit: String| ModelRow { + id: id.into(), + status: "available".into(), + gb: 5.0, + fit, + custom: None, + optimised: None, + runtime: None, + verdict: Verdict::Fits { per_gpu: 5.0, total: 5.0, ctx: 8192, max_conc: 4 }, + }; let text = list_text(&[row("org/open", "fits: 5.0 GB per GPU".into()), row("org/closed", format!("fits: 5.0 GB per GPU{GATED_FIT}"))], None); assert!(text.contains("org/closed") && text.contains("gated: needs your Hugging Face token or access"), "{text}"); assert!(text.contains(&format!("gated models: {}", crate::hf::GATED_WAYS)), "{text}"); diff --git a/engine/crates/core/src/nginx.rs b/engine/crates/core/src/nginx.rs index 6341c40..2c6b045 100644 --- a/engine/crates/core/src/nginx.rs +++ b/engine/crates/core/src/nginx.rs @@ -527,7 +527,7 @@ pub fn write_gateway_configs(home: &Path) -> Result<(), String> { } fn tls_names() -> Vec { - // Loopback, this host's name, and anything in GOETHE_TLS_NAMES. A pod's public IP changes every start, so a name is what a client pins. + // Loopback, hostname, every interface IP, and GOETHE_TLS_NAMES (so --cacert to the bind address verifies). let mut names: Vec = vec!["localhost".into(), "127.0.0.1".into(), "::1".into()]; if let Ok(h) = fs::read_to_string("/etc/hostname") { let h = h.trim(); @@ -535,6 +535,15 @@ fn tls_names() -> Vec { names.push(h.to_string()); } } + if let Some(addrs) = interface_addrs() { + for a in addrs { + match a.ip { + IpAddr::V4(v4) if v4.is_link_local() => {} + IpAddr::V6(v6) if v6.is_unicast_link_local() => {} + ip => names.push(ip.to_string()), + } + } + } names.extend(std::env::var("GOETHE_TLS_NAMES").unwrap_or_default().split(',').map(|n| n.trim().to_string())); names.retain(|n| !n.is_empty()); names.dedup(); @@ -590,8 +599,14 @@ fn worker_user_line(root: bool, group: Option<&str>) -> String { } pub fn ensure_certs(home: &Path) -> Result<(), String> { - // The gateway's self-signed certificate for this host's names, made at install so it exists before the first goethe up. - ensure_tls(home, &san_extension(&tls_names()), false) + // The gateway's self-signed certificate for this host's names; refreshed when the SAN list changes (new iface IP). + let san = san_extension(&tls_names()); + let san_file = nginx_prefix(home).join("certs/sans.txt"); + let stale = fs::read_to_string(&san_file).map(|s| s.trim() != san).unwrap_or(true); + ensure_tls(home, &san, stale)?; + private_dir(&nginx_prefix(home).join("certs"), false)?; + write_private(&san_file, format!("{san}\n").as_bytes())?; + Ok(()) } fn ensure_tls(home: &Path, san: &str, stale: bool) -> Result<(), String> { @@ -1214,6 +1229,15 @@ mod tests { let names = tls_names(); assert!(names.contains(&"localhost".to_string()) && names.contains(&"127.0.0.1".to_string())); assert!(names.iter().all(|n| !n.trim().is_empty())); + if let Some(addrs) = interface_addrs() { + for a in addrs { + match a.ip { + IpAddr::V4(v4) if v4.is_link_local() => {} + IpAddr::V6(v6) if v6.is_unicast_link_local() => {} + ip => assert!(names.contains(&ip.to_string()), "missing iface IP {ip} in {names:?}"), + } + } + } std::env::set_var("GOETHE_TLS_NAMES", "pod-8443.proxy.runpod.net, ,"); assert!(tls_names().contains(&"pod-8443.proxy.runpod.net".to_string()), "blank entries dropped"); std::env::remove_var("GOETHE_TLS_NAMES"); diff --git a/engine/crates/core/src/onboard.rs b/engine/crates/core/src/onboard.rs index 1e7fd4f..c22ebf8 100644 --- a/engine/crates/core/src/onboard.rs +++ b/engine/crates/core/src/onboard.rs @@ -278,7 +278,14 @@ fn step_serve(model: &str) -> Result<(), String> { fn step_access(model: &str) -> Result<(), String> { // Who can reach the API, then a key for the first client, then the ready-to-use call with that key filled in. heading(3, "Configure API access", Some("Choose who can reach the API:")); - let api = configure::apply_access(&ApiOpts { mode: None, vpn_ip: None, interface: None, allow: Vec::new(), yes: false })?; + let api = configure::apply_access(&ApiOpts { + mode: None, + vpn_ip: None, + interface: None, + allow: Vec::new(), + yes: false, + dangerously_allow_all_inbound: false, + })?; let token = choose_key(model)?; READY.lock().unwrap_or_else(|e| e.into_inner()).replace((api.clone(), model.to_string(), token.clone())); Ok(()) @@ -365,7 +372,14 @@ fn advanced() -> Result<(), String> { } Some(4) => { screen("Configure API access", Some("Choose who can reach the API:")); - configure::apply_access(&ApiOpts { mode: None, vpn_ip: None, interface: None, allow: Vec::new(), yes: false }) + configure::apply_access(&ApiOpts { + mode: None, + vpn_ip: None, + interface: None, + allow: Vec::new(), + yes: false, + dangerously_allow_all_inbound: false, + }) .map(|api| Some(format!("API endpoint: {api}"))) } Some(5) => crate::keys_menu::session().map(|s| s.summary()), diff --git a/engine/crates/core/src/packs.rs b/engine/crates/core/src/packs.rs index b3971e2..c816b8c 100644 --- a/engine/crates/core/src/packs.rs +++ b/engine/crates/core/src/packs.rs @@ -256,12 +256,14 @@ fn arch_runs(arch: &str, cc: &str) -> String { } pub fn sku_why(dev: &Device, req: &SkuReq) -> String { - // Why one GPU cannot run one variant ("" when it can), by goethe-engine's rule: the arch on the compute capability, the grid's SMs co-resident, the shared memory per block; an unknown SM count or shared memory never refuses. + // Why one GPU cannot run one variant ("" when it can): arch, SM count, shared memory. Unknown device limits refuse when the pack asks for any. let why = arch_runs(&req.arch, &dev.cc); if !why.is_empty() { return format!("device cc {}: {why}", dev.cc); } match (dev.sms, dev.smem_block) { + (None, _) if req.sms > 0 => format!("the plan's grid needs {} SMs co-resident; this device's SM count is unknown", req.sms), + (_, None) if req.smem_block > 0 => format!("the plan needs {} B shared memory per block; this device's shared-memory limit is unknown", req.smem_block), (Some(have), _) if req.sms > have => format!("the plan's grid needs {} SMs co-resident, the device has {have}", req.sms), (_, Some(have)) if req.smem_block > have => format!("the plan needs {} B shared memory per block, the device allows {have}", req.smem_block), _ => String::new(), @@ -569,9 +571,7 @@ pub fn drafter(e: &Entry) -> Option { } fn sha256_file(p: &Path) -> Option { - let mut h = Sha256::new(); - std::io::copy(&mut std::fs::File::open(p).ok()?, &mut h).ok()?; - Some(format!("{:x}", h.finalize())) + crate::update::sha256_file(p).ok() } pub fn safe_name(f: &str) -> bool { @@ -1094,7 +1094,7 @@ mod tests { let line = sku_refusal(&reqs, &[gpu("NVIDIA GeForce RTX 5090", "12.0", Some(170))]).unwrap(); assert_eq!(line, "the kernel pack is built for rtx_pro_6000_maxq (sm_120a, 188 SMs); none runs on NVIDIA GeForce RTX 5090 (cc 12.0, 170 SMs) (rtx_pro_6000_maxq: the plan's grid needs 188 SMs co-resident, the device has 170)"); assert_eq!(sku_refusal(&reqs, &[gpu("NVIDIA RTX PRO 6000 Blackwell Max-Q", "12.0", Some(188))]), None, "the card it was built for runs it"); - assert_eq!(sku_refusal(&reqs, &[gpu("a card CUDA does not show", "12.0", None)]), None, "an unknown SM count never refuses"); + assert!(sku_refusal(&reqs, &[gpu("a card CUDA does not show", "12.0", None)]).unwrap().contains("SM count is unknown")); assert!(sku_refusal(&reqs, &[gpu("an Ada card", "8.9", Some(128))]).unwrap().contains("sm_120a SASS runs only on cc 12.0")); } diff --git a/engine/crates/core/src/platform_api.rs b/engine/crates/core/src/platform_api.rs index 8992a38..600adcf 100644 --- a/engine/crates/core/src/platform_api.rs +++ b/engine/crates/core/src/platform_api.rs @@ -104,7 +104,7 @@ impl Client { pub fn auth_start(&self, hostname: &str, fingerprint: &str) -> Result { let body = json!({"action": "start", "hostname": hostname, "version": env!("CARGO_PKG_VERSION"), "fingerprint": fingerprint}); - parse_started(&self.post("/api/device/auth", body.to_string().as_bytes(), None)?, self.base().starts_with("http://")) + parse_started(&self.post("/api/device/auth", body.to_string().as_bytes(), None)?, &self.base()) } pub fn auth_poll(&self, device_code: &Secret) -> Result { @@ -250,7 +250,7 @@ fn malformed(what: &str) -> ApiError { ApiError::Malformed(what.to_string()) } -pub fn parse_started(v: &Value, dev: bool) -> Result { +pub fn parse_started(v: &Value, base: &str) -> Result { #[derive(Deserialize)] struct Raw { device_code: String, @@ -266,8 +266,11 @@ pub fn parse_started(v: &Value, dev: bool) -> Result { if r.user_code.is_empty() || r.user_code.len() > 32 || !r.user_code.chars().all(|c| c.is_ascii_alphanumeric() || c == '-') { return Err(malformed("the user code is not valid")); } - let check = if dev { crate::platform::check_dev_link } else { crate::platform::check_link }; + let check = if base.starts_with("http://") { crate::platform::check_dev_link } else { crate::platform::check_link }; let uri = check(&r.verification_uri_complete).map_err(|_| malformed("the verification link is not a valid https URL"))?; + if !verification_on_base(base, &uri) { + return Err(malformed("the verification link is not on the platform")); + } Ok(Started { device_code: Secret::new(r.device_code), user_code: r.user_code, @@ -277,6 +280,26 @@ pub fn parse_started(v: &Value, dev: bool) -> Result { }) } +fn verification_on_base(base: &str, uri: &str) -> bool { + let host = |u: &str| -> String { + let rest = u.split_once("://").map(|(_, r)| r).unwrap_or(""); + let auth = rest.split(['/', '?', '#']).next().unwrap_or(""); + let auth = auth.rsplit_once('@').map(|(_, a)| a).unwrap_or(auth); + let h = if let Some(v6) = auth.strip_prefix('[') { + v6.split_once(']').map(|(h, _)| h).unwrap_or("") + } else { + match auth.rsplit_once(':') { + Some((h, p)) if p.parse::().is_ok() => h, + _ => auth, + } + }; + let h = h.to_ascii_lowercase(); + h.strip_prefix("www.").unwrap_or(&h).to_string() + }; + let (b, u) = (host(base), host(uri)); + !b.is_empty() && b == u +} + pub fn parse_approved(v: &Value) -> Result { #[derive(Deserialize)] struct RawWorkspace { @@ -474,6 +497,18 @@ mod tests { let m = mock::serve(vec![(200, r#"{"device_code":"d","user_code":"AB\u001b[2J","verification_uri_complete":"http://evil.example.com/x","expires_in":900,"interval":3}"#.into())]); let c = Client::new(&m.url, Duration::from_secs(5)).unwrap(); assert!(c.auth_start("h", "f").is_err()); + let m = mock::serve(vec![(200, r#"{"device_code":"d","user_code":"ABCD-1234","verification_uri_complete":"https://evil.example.com/x","expires_in":900,"interval":3}"#.into())]); + let c = Client::new(&m.url, Duration::from_secs(5)).unwrap(); + assert!(c.auth_start("h", "f").is_err(), "https on another origin is refused"); + } + + #[test] + fn verification_link_must_share_the_platform_host() { + assert!(verification_on_base("https://goethe.sh", "https://goethe.sh/device?code=1")); + assert!(verification_on_base("https://goethe.sh", "https://www.goethe.sh/device?code=1"), "www twin"); + assert!(!verification_on_base("https://goethe.sh", "https://evil.example.com/device")); + assert!(!verification_on_base("https://goethe.sh", "https://goethe.sh.evil.com/device")); + assert!(verification_on_base("http://127.0.0.1:9", "http://127.0.0.1:9/device")); } #[test] diff --git a/engine/crates/core/src/public_share.rs b/engine/crates/core/src/public_share.rs index e6c684d..9987fe8 100644 --- a/engine/crates/core/src/public_share.rs +++ b/engine/crates/core/src/public_share.rs @@ -51,20 +51,16 @@ impl Share { } fn settle(&mut self) { - // One gateway for every shared key: open until the last key's time ends; nginx admits the union of their addresses, or anyone when a key has none. + // One gateway for every shared key: open until the last key's time ends; nginx admits the union of their allowlists. self.keys = self.grants.iter().map(|g| g.key.clone()).collect(); self.until = self.grants.iter().map(|g| g.until).max().unwrap_or(self.until); - self.allow = if self.grants.iter().all(|g| !g.allow.is_empty()) { - let mut all: Vec = Vec::new(); - for c in self.grants.iter().flat_map(|g| g.allow.iter()) { - if !all.contains(c) { - all.push(c.clone()); - } + let mut all: Vec = Vec::new(); + for c in self.grants.iter().flat_map(|g| g.allow.iter()) { + if !all.contains(c) { + all.push(c.clone()); } - all - } else { - Vec::new() - }; + } + self.allow = all; } fn grant(&self, key: &str) -> Option { @@ -197,7 +193,7 @@ fn decide(share: Option<&Share>, key_id: &str, is_admin: bool, now: i64, client: if now >= g.until { return Access::KeyExpired; } - if !g.allow.is_empty() && !client.is_some_and(|ip| g.allow.iter().any(|c| cidr_contains(c, ip))) { + if g.allow.is_empty() || !client.is_some_and(|ip| g.allow.iter().any(|c| cidr_contains(c, ip))) { return Access::NotFromHere; } Access::Allowed @@ -218,6 +214,13 @@ fn key_rows(home: &Path) -> Result, String> { rows } +fn strip_tokens_for_reopen(keys: &mut [SharedKey]) { + // Reopening remembered keys must not reprint bearer tokens (goethe api / keys create already showed them). + for k in keys { + k.token = None; + } +} + fn resolve_keys(home: &Path, wanted: &[String], until: i64, warnings: &mut Vec) -> Result, String> { // Each name or id must match exactly one existing key that is not an admin key and has not expired. let rows = key_rows(home)?; @@ -255,6 +258,70 @@ fn port_free(port: u16) -> bool { TcpListener::bind(SocketAddr::from(([0, 0, 0, 0], port))).is_ok() } +fn share_allow(cli: &[String], reuse_keys: bool, kept: &[String]) -> Vec { + // --allow wins; reopening the same keys keeps the addresses from their last share. Never the VPN/API allowlist. + if !cli.is_empty() { + return cli.to_vec(); + } + if reuse_keys && !kept.is_empty() { + return kept.to_vec(); + } + Vec::new() +} + +fn need_client_allow(home: &Path) -> String { + // Who may connect is the client's public IP, not this machine's (that one is only the URL). + let here = crate::platform::lookup_public_ip(home) + .ok() + .map(|ip| format!(" this machine's public address is {ip} (that becomes the URL, not --allow);")) + .unwrap_or_default(); + format!( + "name who may connect with --allow /32 (their phone or laptop's public IP, not this machine's).{here} ask them; to open to every address use --dangerously-allow-all-inbound" + ) +} + +fn reject_bad_share_cidrs(home: &Path, allow: &[String]) -> Result<(), String> { + let machine = crate::platform::lookup_public_ip(home).ok(); + for c in allow { + if c.ends_with("/0") { + return Err( + "the public gateway will not open to every address unless you pass --dangerously-allow-all-inbound; otherwise name the client's public IP with --allow /32" + .into(), + ); + } + if let Some(ip) = machine { + let host = format!("{ip}/{}", if ip.is_ipv4() { 32 } else { 128 }); + if *c == host { + return Err(format!( + "{c} is this machine's public address (the URL), not who may connect; use their phone or laptop's public IP" + )); + } + } + } + Ok(()) +} + +fn resolve_share_cidrs( + home: &Path, + cli_allow: &[String], + reuse_keys: bool, + kept: &[String], + dangerously_allow_all_inbound: bool, +) -> Result, String> { + if dangerously_allow_all_inbound { + if !cli_allow.is_empty() { + return Err("--dangerously-allow-all-inbound cannot be combined with --allow".into()); + } + return Ok(vec!["0.0.0.0/0".into()]); + } + let allow = share_allow(cli_allow, reuse_keys, kept); + if allow.is_empty() { + return Err(need_client_allow(home)); + } + reject_bad_share_cidrs(home, &allow)?; + Ok(allow) +} + fn mux_public_listening(home: &Path) -> bool { nginx::persisted_mux_port(home) .is_ok_and(|p| TcpStream::connect_timeout(&public_mux_addr(p), Duration::from_millis(500)).is_ok()) @@ -276,6 +343,7 @@ pub struct OpenArgs<'a> { pub keys: &'a [String], pub allow: &'a [String], pub port: Option, + pub dangerously_allow_all_inbound: bool, } pub fn open(home: &Path, a: OpenArgs) -> Result { @@ -297,8 +365,12 @@ pub fn open(home: &Path, a: OpenArgs) -> Result { } else { a.keys.to_vec() }; - let keys = resolve_keys(home, &wanted, until, &mut warnings)?; - let allow = a.allow.iter().map(|c| security::canonical_cidr(c)).collect::, _>>()?; + let reopen = a.keys.is_empty(); + let mut keys = resolve_keys(home, &wanted, until, &mut warnings)?; + if reopen { + strip_tokens_for_reopen(&mut keys); + } + let cli_allow = a.allow.iter().map(|c| security::canonical_cidr(c)).collect::, _>>()?; let port = a.port.or(previous.as_ref().map(|p| p.port)).unwrap_or(DEFAULT_PORT); let mut grants: Vec = previous .as_ref() @@ -307,8 +379,18 @@ pub fn open(home: &Path, a: OpenArgs) -> Result { .unwrap_or_default(); for k in &keys { let kept = previous.as_ref().and_then(|p| p.grant(&k.id)).map(|g| g.allow).unwrap_or_default(); + let allow = resolve_share_cidrs(home, &cli_allow, reopen, &kept, a.dangerously_allow_all_inbound)?; grants.retain(|g| g.key != k.id); - grants.push(Grant { key: k.id.clone(), until, allow: if a.allow.is_empty() && a.keys.is_empty() { kept } else { allow.clone() } }); + grants.push(Grant { key: k.id.clone(), until, allow }); + } + if a.dangerously_allow_all_inbound { + warnings.push(format!( + "every address may connect; rate limit is {RATE_PER_IP} requests/s per client (burst {}), {CONN_PER_IP} connections", + 2 * RATE_PER_IP + )); + } + if reopen { + warnings.push("the key secret is not printed again; goethe api shows it".into()); } let gateway_port = nginx::public_port_u16(home); let mux_port = nginx::persisted_mux_port(home)?; @@ -462,16 +544,17 @@ fn ensure_cert(home: &Path, ip: Option) -> Result<(), String> { names.insert(0, format!("IP:{ip}")); } let san = format!("subjectAltName={}", names.join(",")); + let stamp = format!("{san}\nleaf"); let san_file = dir.join("public_sans.txt"); - let current = fs::read_to_string(&san_file).map(|s| s.trim() == san).unwrap_or(false); + let current = fs::read_to_string(&san_file).map(|s| s.trim() == stamp).unwrap_or(false); if current && crt.is_file() && key.is_file() { return Ok(()); } - nginx::self_signed(&key, &crt, true, 30, "goethe-public", &[&san, "basicConstraints=critical,CA:TRUE"]) + nginx::self_signed(&key, &crt, true, 30, "goethe-public", &[&san, "basicConstraints=critical,CA:FALSE"]) .map_err(|_| "openssl could not create the public gateway certificate".to_string())?; use std::os::unix::fs::PermissionsExt; fs::set_permissions(&crt, fs::Permissions::from_mode(0o644)).map_err(|e| e.to_string())?; - nginx::write_private(&san_file, san.as_bytes()) + nginx::write_private(&san_file, stamp.as_bytes()) } fn fingerprint(crt: &Path) -> Option { @@ -487,13 +570,8 @@ pub fn nginx_block(home: &Path, mux_port: u16, modern_http2: bool, max_body_mb: if !cert_path(home).is_file() && ensure_cert(home, None).is_err() { return String::new(); } - let allow = if share.allow.is_empty() { - String::new() - } else { - let mut lines: String = share.allow.iter().map(|c| format!(" allow {c};\n")).collect(); - lines.push_str(" deny all;\n"); - lines - }; + let mut allow: String = share.allow.iter().map(|c| format!(" allow {c};\n")).collect(); + allow.push_str(" deny all;\n"); const TMPL: &str = include_str!("../../../nginx/goethe_public.conf.tmpl"); TMPL.replace("{{PORT}}", &share.port.to_string()) .replace("{{HTTP2_LISTEN}}", if modern_http2 { "" } else { " http2" }) @@ -535,7 +613,7 @@ mod tests { use super::*; fn share(open: bool, until: i64) -> Share { - Share { open, until, port: 8444, keys: vec!["k1".into()], allow: vec![], grants: vec![] } + Share { open, until, port: 8444, keys: vec!["k1".into()], allow: vec!["203.0.113.0/24".into()], grants: vec![] } } fn granted(grants: &[(&str, i64, &[&str])]) -> Share { @@ -547,12 +625,13 @@ mod tests { #[test] fn two_keys_share_one_gateway_and_one_that_ended_is_refused_while_the_other_works() { - let s = granted(&[("k1", 1000, &[]), ("k4", 5000, &[])]); + let ip = |s: &str| Some(s.parse::().unwrap()); + let s = granted(&[("k1", 1000, &["203.0.113.0/24"]), ("k4", 5000, &["203.0.113.0/24"])]); assert_eq!((s.until, s.keys.clone()), (5000, vec!["k1".to_string(), "k4".to_string()]), "open until the last key ends"); - assert_eq!(decide(Some(&s), "k1", false, 999, None), Access::Allowed); - assert_eq!(decide(Some(&s), "k1", false, 1000, None), Access::KeyExpired); - assert_eq!(decide(Some(&s), "k4", false, 1000, None), Access::Allowed, "the gateway stays open for the other key"); - assert_eq!(decide(Some(&s), "k4", false, 5000, None), Access::Closed); + assert_eq!(decide(Some(&s), "k1", false, 999, ip("203.0.113.1")), Access::Allowed); + assert_eq!(decide(Some(&s), "k1", false, 1000, ip("203.0.113.1")), Access::KeyExpired); + assert_eq!(decide(Some(&s), "k4", false, 1000, ip("203.0.113.1")), Access::Allowed, "the gateway stays open for the other key"); + assert_eq!(decide(Some(&s), "k4", false, 5000, ip("203.0.113.1")), Access::Closed); } #[test] @@ -563,10 +642,10 @@ mod tests { assert_eq!(decide(Some(&s), "k1", false, 0, ip("203.0.113.9")), Access::Allowed); assert_eq!(decide(Some(&s), "k1", false, 0, ip("198.51.100.7")), Access::NotFromHere, "not the other key's address"); assert_eq!(decide(Some(&s), "k1", false, 0, None), Access::NotFromHere, "no address, no entry"); - let open = granted(&[("k1", 1000, &["203.0.113.0/24"]), ("k4", 1000, &[])]); - assert!(open.allow.is_empty(), "a key without addresses opens nginx to anyone; the router still limits k1"); - assert_eq!(decide(Some(&open), "k1", false, 0, ip("8.8.8.8")), Access::NotFromHere); - assert_eq!(decide(Some(&open), "k4", false, 0, ip("8.8.8.8")), Access::Allowed); + let mixed = granted(&[("k1", 1000, &["203.0.113.0/24"]), ("k4", 1000, &[])]); + assert_eq!(mixed.allow, vec!["203.0.113.0/24".to_string()], "empty allowlists do not open nginx to the world"); + assert_eq!(decide(Some(&mixed), "k1", false, 0, ip("8.8.8.8")), Access::NotFromHere); + assert_eq!(decide(Some(&mixed), "k4", false, 0, ip("8.8.8.8")), Access::NotFromHere); assert!(cidr_contains("2001:db8::/32", "2001:db8::1".parse().unwrap()) && !cidr_contains("2001:db8::/32", "2001:db9::1".parse().unwrap())); } @@ -583,13 +662,14 @@ mod tests { #[test] fn only_listed_non_admin_keys_pass_while_the_window_is_open() { + let ip = |s: &str| Some(s.parse::().unwrap()); let s = share(true, 1000); - assert_eq!(decide(Some(&s), "k1", false, 999, None), Access::Allowed); - assert_eq!(decide(Some(&s), "k1", false, 1000, None), Access::Closed, "closes at the end second"); - assert_eq!(decide(Some(&s), "k2", false, 999, None), Access::KeyNotShared); - assert_eq!(decide(Some(&s), "k1", true, 999, None), Access::KeyNotShared, "admin keys never"); - assert_eq!(decide(Some(&share(false, 5000)), "k1", false, 999, None), Access::Closed, "shut down"); - assert_eq!(decide(None, "k1", false, 0, None), Access::Closed); + assert_eq!(decide(Some(&s), "k1", false, 999, ip("203.0.113.1")), Access::Allowed); + assert_eq!(decide(Some(&s), "k1", false, 1000, ip("203.0.113.1")), Access::Closed, "closes at the end second"); + assert_eq!(decide(Some(&s), "k2", false, 999, ip("203.0.113.1")), Access::KeyNotShared); + assert_eq!(decide(Some(&s), "k1", true, 999, ip("203.0.113.1")), Access::KeyNotShared, "admin keys never"); + assert_eq!(decide(Some(&share(false, 5000)), "k1", false, 999, ip("203.0.113.1")), Access::Closed, "shut down"); + assert_eq!(decide(None, "k1", false, 0, ip("203.0.113.1")), Access::Closed); } fn home_with_keys() -> tempfile::TempDir { @@ -621,6 +701,7 @@ mod tests { let got = resolve_keys(h, &["app".into(), "k1".into()], now() + 60, &mut w).unwrap(); assert_eq!(got.len(), 1, "the same key twice counts once"); assert!(w.is_empty()); + assert!(got[0].token.as_deref() == Some("tok")); assert!(resolve_keys(h, &["root".into()], now() + 60, &mut w).err().unwrap().contains("admin")); assert!(resolve_keys(h, &["old".into()], now() + 60, &mut w).err().unwrap().contains("expired")); assert!(resolve_keys(h, &["dup".into()], now() + 60, &mut w).err().unwrap().contains("several")); @@ -629,6 +710,18 @@ mod tests { assert!(w.iter().any(|x| x.contains("every model"))); } + #[test] + fn reopen_without_named_keys_strips_bearer_tokens() { + let dir = home_with_keys(); + let h = dir.path(); + let mut w = Vec::new(); + let mut keys = resolve_keys(h, &["app".into()], now() + 60, &mut w).unwrap(); + assert_eq!(keys[0].token.as_deref(), Some("tok")); + strip_tokens_for_reopen(&mut keys); + assert!(keys[0].token.is_none()); + assert_eq!(keys[0].name, "app"); + } + #[test] fn no_block_is_rendered_without_an_open_window() { let dir = home_with_keys(); @@ -646,19 +739,90 @@ mod tests { assert!(block.contains("public.crt")); } + #[test] + fn nginx_block_always_denies_all_even_with_an_empty_allowlist() { + let dir = home_with_keys(); + let h = dir.path(); + store(h, &Share { allow: vec![], ..share(true, now() + 600) }).unwrap(); + let block = nginx_block(h, 30000, true, 64); + assert!(block.contains("deny all;"), "{block}"); + assert!(!block.contains("allow "), "empty allowlist must not admit the world: {block}"); + } + + #[test] + fn share_allow_prefers_cli_then_kept_never_gateway() { + assert_eq!(share_allow(&["203.0.113.9/32".into()], false, &["198.51.100.1/32".into()]), vec!["203.0.113.9/32".to_string()]); + assert_eq!(share_allow(&[], true, &["198.51.100.1/32".into()]), vec!["198.51.100.1/32".to_string()]); + assert!(share_allow(&[], false, &[]).is_empty(), "VPN/API allowlist must not open public share"); + assert!(share_allow(&[], true, &[]).is_empty()); + } + + #[test] + fn need_client_allow_names_the_client_not_this_machine() { + let dir = tempfile::tempdir().unwrap(); + let msg = need_client_allow(dir.path()); + assert!(msg.contains("their phone or laptop") && msg.contains("--allow"), "{msg}"); + assert!(msg.contains("dangerously-allow-all-inbound"), "{msg}"); + assert!(!msg.contains("goethe api allow"), "{msg}"); + } + + #[test] + fn share_refuses_everyone_and_this_machines_public_ip_unless_dangerously() { + let dir = tempfile::tempdir().unwrap(); + let home = dir.path(); + assert!(reject_bad_share_cidrs(home, &["0.0.0.0/0".into()]).unwrap_err().contains("dangerously-allow-all-inbound")); + assert!(reject_bad_share_cidrs(home, &["::/0".into()]).unwrap_err().contains("dangerously-allow-all-inbound")); + let ip: IpAddr = "203.0.113.50".parse().unwrap(); + std::env::set_var("GOETHE_PUBLIC_IP", ip.to_string()); + let err = reject_bad_share_cidrs(home, &[format!("{ip}/32")]).unwrap_err(); + std::env::remove_var("GOETHE_PUBLIC_IP"); + assert!(err.contains("this machine's public address"), "{err}"); + assert!(reject_bad_share_cidrs(home, &["198.51.100.7/32".into()]).is_ok()); + + let all = resolve_share_cidrs(home, &[], false, &[], true).unwrap(); + assert_eq!(all, vec!["0.0.0.0/0".to_string()]); + assert!(resolve_share_cidrs(home, &["198.51.100.7/32".into()], false, &[], true) + .unwrap_err() + .contains("cannot be combined")); + assert!(resolve_share_cidrs(home, &["0.0.0.0/0".into()], false, &[], false) + .unwrap_err() + .contains("dangerously-allow-all-inbound")); + assert_eq!( + resolve_share_cidrs(home, &["198.51.100.7/32".into()], false, &[], false).unwrap(), + vec!["198.51.100.7/32".to_string()] + ); + assert_eq!( + resolve_share_cidrs(home, &[], true, &["203.0.113.9/32".into()], false).unwrap(), + vec!["203.0.113.9/32".to_string()] + ); + assert!(resolve_share_cidrs(home, &[], false, &[], false).unwrap_err().contains("--allow")); + } + + #[test] + fn nginx_block_with_everyone_still_ends_in_deny_all() { + let dir = home_with_keys(); + let h = dir.path(); + store(h, &Share { allow: vec!["0.0.0.0/0".into()], ..share(true, now() + 600) }).unwrap(); + let block = nginx_block(h, 30000, true, 64); + assert!(block.contains("allow 0.0.0.0/0;\n deny all;"), "{block}"); + assert!(block.contains(&format!("rate={RATE_PER_IP}r/s")), "{block}"); + assert!(block.contains(&format!("limit_conn goethe_pub_conn {CONN_PER_IP}")), "{block}"); + } + #[test] fn the_router_sees_every_change_at_once() { let dir = home_with_keys(); let h = dir.path(); - assert_eq!(check(h, "k1", false, None), Access::Closed); + let ip = Some("203.0.113.1".parse::().unwrap()); + assert_eq!(check(h, "k1", false, ip), Access::Closed); store(h, &share(true, now() + 600)).unwrap(); - assert_eq!(check(h, "k1", false, None), Access::Allowed); + assert_eq!(check(h, "k1", false, ip), Access::Allowed); store(h, &Share { keys: vec!["k4".into()], ..share(true, now() + 600) }).unwrap(); - assert_eq!(check(h, "k1", false, None), Access::KeyNotShared, "a replaced key stops at the next request"); + assert_eq!(check(h, "k1", false, ip), Access::KeyNotShared, "a replaced key stops at the next request"); store(h, &Share { open: false, ..share(true, now() + 600) }).unwrap(); - assert_eq!(check(h, "k1", false, None), Access::Closed); + assert_eq!(check(h, "k1", false, ip), Access::Closed); fs::remove_file(path(h)).unwrap(); - assert_eq!(check(h, "k1", false, None), Access::Closed); + assert_eq!(check(h, "k1", false, ip), Access::Closed); } #[test] @@ -681,6 +845,6 @@ mod tests { ensure_cert(dir.path(), Some("203.0.113.9".parse().unwrap())).unwrap(); let out = Command::new("openssl").args(["x509", "-noout", "-text", "-in"]).arg(cert_path(dir.path())).output().unwrap(); let text = String::from_utf8_lossy(&out.stdout); - assert!(text.contains("203.0.113.9") && text.contains("CA:TRUE"), "{text}"); + assert!(text.contains("203.0.113.9") && text.contains("CA:FALSE"), "{text}"); } } diff --git a/engine/crates/core/src/recommend.rs b/engine/crates/core/src/recommend.rs index d28ab03..711347d 100644 --- a/engine/crates/core/src/recommend.rs +++ b/engine/crates/core/src/recommend.rs @@ -146,6 +146,7 @@ impl View { let mut trial = input.clone(); trial.ctx = plan.ctx; trial.max_conc = plan.max_conc; + trial.overhead_gb = crate::engines::by_id(runtime).map_or(calc::BASE_OVERHEAD_GB, |e| e.overhead_gb); let base = chooser::need(&trial, plan.tp, &plan.cc); let learned = self.con.as_ref().map(|c| crate::engine_overhead::read_learned(c, model_id, &self.gpu_name(plan.gpus[0]), plan.tp as u64, runtime)); let learned = learned.unwrap_or_default(); @@ -238,7 +239,10 @@ pub fn run(json_out: bool) -> Result<(), String> { Some(n) => format!("{} ({n})", item.model_id), None => item.model_id.clone(), }; - let mut cells = vec![name, format!("{total:.1} GB")]; + let mut cells = vec![ + name, + format!("{:.1} GB/GPU · ctx {} · {} conc", item.vram_per_gpu_gb, item.ctx, item.max_conc), + ]; if multi { cells.push(format_gpu_list(&item.gpus)); } @@ -262,10 +266,11 @@ pub fn preview(home: &Path, model: &RegistryModel) -> Result, Str let pinned = model.default_tp.filter(|t| *t > 0).map(|t| t as usize); match view.fit(&model.id, &input, pinned) { Ok(f) => Ok(Some(format!( - "fit: ctx {} on GPU {} ({:.1} GB per GPU)", + "fit: {:.1} GB per GPU (ctx {}, {} concurrent) on GPU {}", + f.need.total(), f.ctx, + f.max_conc, format_gpu_list(&f.gpus), - f.need.total() ))), Err(e) => Err(format!("does not fit the allowed GPUs: {e}")), } @@ -407,6 +412,7 @@ pub fn registry_input(m: &RegistryModel) -> ChooserInput { crate::memory_spec::apply_engine_args(&mut spec, None, &m.extra_sglang_args); spec }), + overhead_gb: crate::engines::by_id(m.effective_runtime()).map_or(crate::calc::BASE_OVERHEAD_GB, |e| e.overhead_gb), } } @@ -582,7 +588,8 @@ mod tests { let llama = reg.models.iter().find(|m| m.id == "nvidia/Llama-3.1-8B-Instruct-NVFP4").unwrap(); let line = preview(home.path(), llama).unwrap().unwrap(); assert!(line.contains("ctx 65536"), "{line}"); - assert!(line.contains("23.4 GB"), "{line}"); + assert!(line.contains("23.4 GB per GPU"), "{line}"); + assert!(line.contains("concurrent"), "{line}"); let big = reg.models.iter().find(|m| m.id == "Qwen/Qwen3.8-27B").unwrap(); assert!(preview(home.path(), big).is_err()); } diff --git a/engine/crates/core/src/scan.rs b/engine/crates/core/src/scan.rs index 20d877d..91c5e08 100644 --- a/engine/crates/core/src/scan.rs +++ b/engine/crates/core/src/scan.rs @@ -275,7 +275,19 @@ fn candidate_dirs(home: &Path, dir: Option<&str>) -> Result, String out.sort(); return Ok(out); }; - let canonical = fs::canonicalize(d).map_err(|e| format!("{d}: {e}"))?; + let raw = Path::new(d); + let candidate = if raw.is_absolute() || d.contains('/') || d.contains('\\') { + raw.to_path_buf() + } else { + models.join(d) + }; + let canonical = fs::canonicalize(&candidate).map_err(|e| { + format!( + "{}: {e} (a folder inside {}, e.g. org__name; or goethe install --name org/name)", + candidate.display(), + models.display() + ) + })?; let root = fs::canonicalize(&models).map_err(|e| e.to_string())?; match canonical.strip_prefix(&root) { Ok(rel) if !rel.as_os_str().is_empty() => Ok(vec![models.join(rel)]), @@ -335,7 +347,10 @@ pub fn survey(home: &Path, con: &Connection, dirs: Vec, m: &Machine) -> continue; } if meta.is_ok_and(|m| m.file_type().is_symlink()) { - let why = format!("a symlink; register the folder it points to with goethe install {}", canonical.display()); + let why = format!( + "a symlink; register it with: goethe install {} --name ", + canonical.display() + ); out.push(Finding { path, name, state: State::Unsupported(why) }); continue; } @@ -927,6 +942,21 @@ mod tests { ); } + #[test] + fn folder_name_resolves_under_models_not_cwd() { + let home = tempfile::tempdir().unwrap(); + let h = home.path(); + crate::paths::ensure_layout(h).unwrap(); + let models = h.join("models"); + fs::write(model(&models, "org__fresh", "LlamaForCausalLM").join("model.safetensors"), b"x").unwrap(); + let got = candidate_dirs(h, Some("org__fresh")).unwrap(); + assert_eq!(got.len(), 1); + assert_eq!(got[0], models.join("org__fresh")); + let err = candidate_dirs(h, Some("nope__missing")).unwrap_err(); + assert!(err.contains("models"), "{err}"); + assert!(err.contains("goethe install") || err.contains("org__name") || err.contains("No such"), "{err}"); + } + #[test] fn parses_entry_classes_and_quant_table() { let src = "class A:\n pass\nEntryClass = [\n LlamaForCausalLM,\n Phi3ForCausalLM,\n]\n"; diff --git a/engine/crates/core/src/setup.rs b/engine/crates/core/src/setup.rs index 555542c..3dd2ab1 100644 --- a/engine/crates/core/src/setup.rs +++ b/engine/crates/core/src/setup.rs @@ -6,15 +6,32 @@ use std::path::{Path, PathBuf}; pub const DEFAULT_PKG_URL: &str = "https://pkg.goethe.sh"; +pub fn migrate_retired_pkg_env(home: &Path) -> Result<(), String> { + let text = fs::read_to_string(home.join("goethe.env")).unwrap_or_default(); + if !text.contains("pkg.meshapplied.com") { + return Ok(()); + } + let m = crate::paths::parse_env_file(&text); + write_env_file( + home, + &env_file_text( + m.get("GOETHE_PKG_URL").map(|s| s.as_str()).unwrap_or(DEFAULT_PKG_URL), + m.get("GOETHE_REGISTRY_URL").map(|s| s.as_str()), + m.get("GOETHE_PKG_TOKEN").map(|s| s.as_str()), + true, + ), + ) +} + pub fn env_file_text(pkg: &str, registry: Option<&str>, token: Option<&str>, keep_file_url: bool) -> String { // GOETHE_HOME/goethe.env holds only what differs from the defaults; a file:// package is a one-off unless asked to keep it. let pkg = pkg.trim_end_matches('/'); let mut out = String::new(); let local = pkg.starts_with("file://"); - if pkg != DEFAULT_PKG_URL && !pkg.is_empty() && (!local || keep_file_url) { + if pkg != DEFAULT_PKG_URL && !pkg.is_empty() && !pkg.contains("pkg.meshapplied.com") && (!local || keep_file_url) { out.push_str(&format!("GOETHE_PKG_URL={pkg}\n")); } - if let Some(r) = registry.map(str::trim).filter(|r| !r.is_empty() && *r != format!("{pkg}/v1/registry.json")) { + if let Some(r) = registry.map(str::trim).filter(|r| !r.is_empty() && !r.contains("pkg.meshapplied.com") && *r != format!("{pkg}/v1/registry.json")) { out.push_str(&format!("GOETHE_REGISTRY_URL={r}\n")); } if let Some(t) = token.map(str::trim).filter(|t| !t.is_empty()) { @@ -268,6 +285,7 @@ pub fn run(o: Opts) -> Result<(), String> { // Verify the signed package, place goethe, goethed and nginx (handing a running goethed's engines over), then settings, links, certificate, fixes and login. let home = crate::paths::goethe_home(); crate::paths::ensure_layout(&home).map_err(|e| format!("{}: {e}", home.display()))?; + let _ = migrate_retired_pkg_env(&home); fs::set_permissions(&home, std::os::unix::fs::PermissionsExt::from_mode(0o700)).map_err(|e| format!("{}: {e}", home.display()))?; let macos = cfg!(target_os = "macos"); let from = o.from.clone().unwrap_or_else(|| o.pkg.clone()).trim_end_matches('/').to_string(); @@ -413,6 +431,21 @@ mod tests { assert_eq!(login_question(&linked(false)), None); } + #[test] + fn retired_pkg_host_is_dropped_from_goethe_env() { + let t = tempfile::tempdir().unwrap(); + let home = t.path(); + fs::create_dir_all(home).unwrap(); + #[cfg(unix)] + fs::set_permissions(home, std::os::unix::fs::PermissionsExt::from_mode(0o700)).unwrap(); + crate::platform::write_private(&home.join("goethe.env"), b"GOETHE_PKG_URL=https://pkg.meshapplied.com\nGOETHE_PKG_TOKEN=tok\n").unwrap(); + migrate_retired_pkg_env(home).unwrap(); + let text = fs::read_to_string(home.join("goethe.env")).unwrap_or_default(); + assert!(!text.contains("pkg.meshapplied.com"), "{text}"); + assert!(text.contains("GOETHE_PKG_TOKEN=tok"), "{text}"); + migrate_retired_pkg_env(home).unwrap(); + } + #[test] fn the_path_line_goes_into_the_shells_startup_file_once() { let home = Path::new("/home/u"); diff --git a/engine/crates/core/src/sglang_infer.rs b/engine/crates/core/src/sglang_infer.rs index 80da76d..3a72c46 100644 --- a/engine/crates/core/src/sglang_infer.rs +++ b/engine/crates/core/src/sglang_infer.rs @@ -59,8 +59,13 @@ pub const RESERVED_FLAGS: &[(&str, &str)] = &[ ("--context-length", "use --ctx"), ("--max-running-requests", "use --max-conc"), ("--mem-fraction-static", "goethed derives it from the VRAM budget"), + ("--max-mamba-cache-size", "goethe sizes the state pool from the budget"), + ("--swa-full-tokens-ratio", "goethe sizes the window pool from the budget"), ("--served-model-name", "goethe serves the model under its id"), ("--api-key", "goethed sets a per-engine key"), + ("--pack", "goethe-engine sets the pack path"), + ("--weights", "goethe-engine sets the weight paths"), + ("--tokenizer", "goethe-engine sets the tokenizer dir"), ]; #[derive(Debug, Clone, PartialEq, Eq)] diff --git a/engine/crates/core/src/update.rs b/engine/crates/core/src/update.rs index e96f981..df2f471 100644 --- a/engine/crates/core/src/update.rs +++ b/engine/crates/core/src/update.rs @@ -28,6 +28,7 @@ pub fn run(opts: UpdateOpts) -> Result<(), String> { // Fetch and verify the signed manifest, compare versions, stage files, then swap atomically. let home = goethe_home(); ensure_layout(&home).map_err(|e| e.to_string())?; + let _ = crate::setup::migrate_retired_pkg_env(&home); let pkg = pkg_base_url(); let tmp = home.join(".update-staging"); let _ = fs::remove_dir_all(&tmp); @@ -694,4 +695,17 @@ mod tests { assert_eq!(compare_versions("v0.1.1+abc", "0.1.1"), Ordering::Equal); assert_eq!(compare_versions("0.0.9", "0.1.0"), Ordering::Less); } + + #[test] + fn one_sha256_file_matches_ninfer_wrapper() { + let t = tempfile::tempdir().unwrap(); + let p = t.path().join("blob"); + fs::write(&p, b"goethe-sha256-equiv").unwrap(); + let a = sha256_file(&p).unwrap(); + let b = crate::engines::ninfer::sha256_file(&p).unwrap(); + assert_eq!(a, b); + assert_eq!(a.len(), 64); + assert!(a.chars().all(|c| c.is_ascii_hexdigit())); + assert!(sha256_file(&t.path().join("missing")).is_err()); + } } diff --git a/engine/crates/goethe/src/complete.rs b/engine/crates/goethe/src/complete.rs index 083e71b..262c493 100644 --- a/engine/crates/goethe/src/complete.rs +++ b/engine/crates/goethe/src/complete.rs @@ -64,13 +64,12 @@ pub fn authkey_stdin() -> Vec { } pub fn expose_targets() -> Vec { - // goethe api expose: the three modes, then the addresses and interfaces it can listen on directly. + // Modes first. Raw 0.0.0.0 is not offered: that bind goes through `expose public` with guards. let mut out = vec![ candidate("local", "this machine only"), candidate("vpn", "a VPN or private network"), - candidate("public", "a public network, allowlisted clients"), + candidate("public", "a public network, allowlisted clients (needs --allow and --yes)"), candidate("127.0.0.1", "this machine only"), - candidate("0.0.0.0", "every address"), ]; out.extend(interfaces()); out diff --git a/engine/crates/goethe/src/main.rs b/engine/crates/goethe/src/main.rs index 2fd6b55..0567cc9 100644 --- a/engine/crates/goethe/src/main.rs +++ b/engine/crates/goethe/src/main.rs @@ -74,24 +74,20 @@ const EXPOSE_HELP: &str = "Who can reach the API: \t→ goethe api expose local (this machine only) \t→ goethe api expose vpn --interface wg0 -\t→ goethe api expose public --allow 203.0.113.7/32 +\t→ goethe api expose public --allow 203.0.113.7/32 --yes -Or listen on one address or interface directly: - -\t→ goethe api expose 0.0.0.0 -\t→ goethe api expose wg0 +Public always needs --allow /32 and a confirm (--yes, or y on a terminal). It never opens on a leftover allowlist alone. goethe api expose alone shows where the API listens and who may connect."; const SHARE_HELP: &str = "Examples: -\t→ goethe api share 5h --key my-app -\t→ goethe api share 1d (the keys of the last opening) +\t→ goethe api share 5h --key my-app --allow /32 +\t→ goethe api share 5h --key my-app --dangerously-allow-all-inbound \t→ goethe api share (status) -\t→ goethe api share stop --key my-app (that key only) \t→ goethe api share stop -Every shared key uses the one public gateway, each until its own time and from its own addresses; it closes when the last one ends."; +--allow is who may connect (their public IP: ask them, or https://ifconfig.me on their device). This machine's IP is only the URL goethe prints. Reopening the same keys keeps their last addresses. --dangerously-allow-all-inbound opens to every address (rate limits still apply). Closes when the last key ends."; #[derive(Parser)] #[command( @@ -460,16 +456,22 @@ struct TraceArgs { #[derive(clap::Args)] struct ExposeArgs { - #[arg(add = clap_complete::engine::ArgValueCandidates::new(complete::expose_targets), value_name = "WHERE", help = "local, vpn or public; or an address (127.0.0.1, 0.0.0.0, an IP of this host) or an interface such as wg0 to listen on.")] + #[arg(add = clap_complete::engine::ArgValueCandidates::new(complete::expose_targets), value_name = "WHERE", help = "local, vpn or public; or an address (127.0.0.1, an IP of this host) or an interface such as wg0 to listen on.")] target: Option, - #[arg(add = clap_complete::engine::ArgValueCandidates::new(complete::interfaces), long, value_name = "NAME", conflicts_with = "ip", help = "With vpn or public: the network interface to serve, e.g. wg0; the gateway follows its address.")] + #[arg(add = clap_complete::engine::ArgValueCandidates::new(complete::interfaces), long, value_name = "NAME", conflicts_with = "ip", help = "With vpn: the network interface to serve, e.g. wg0; the gateway follows its address.")] interface: Option, - #[arg(long, value_name = "IP", help = "With vpn or public: a fixed address of this machine to listen on, instead of following an interface.")] + #[arg(long, value_name = "IP", hide = true, help = "Advanced: with vpn, a fixed address of this machine to listen on, instead of following an interface.")] ip: Option, - #[arg(long, value_name = "CIDR", num_args = 1.., value_delimiter = ',', help = "With vpn or public: client ranges to allow (default for vpn: the ranges the network reaches).")] + #[arg(long, value_name = "CIDR", num_args = 1.., value_delimiter = ',', help = "With vpn or public: client ranges to allow (required for public; default for vpn: the ranges the network reaches).")] allow: Vec, - #[arg(long, help = "Apply without questions (public still refuses 0.0.0.0/0).")] + #[arg(long, help = "Apply without questions. Public still needs --allow /32 (or --dangerously-allow-all-inbound).")] yes: bool, + #[arg( + long, + hide = true, + help = "Advanced / public only: allow every client address (0.0.0.0/0). Needs --yes. Not with --allow." + )] + dangerously_allow_all_inbound: bool, #[arg(long, help = "With no WHERE: the status as JSON.")] json: bool, } @@ -482,6 +484,8 @@ struct ShareArgs { key: Vec, #[arg(long, value_name = "CIDR", num_args = 1.., value_delimiter = ',', help = "Only these client addresses or ranges may use the keys given, such as 203.0.113.7.")] allow: Vec, + #[arg(long, help = "Allow every address on the public gateway (0.0.0.0/0). Rate limits still apply. Not with --allow.")] + dangerously_allow_all_inbound: bool, #[arg(long, help = "The public port (default 8444, or the last one used).")] port: Option, } @@ -677,7 +681,7 @@ const HELP_SECTIONS: &[(&str, &[(&str, &str)])] = &[ ("goethe api expose vpn --interface wg0", "serve a VPN or private network (also local or public)"), ("goethe api allow add 203.0.113.7/32", "let a client address in (remove takes it out)"), ("goethe api domain check DOMAIN", "check that a domain reaches the API"), - ("goethe api share 5h --key my-app", "open a public address for a while (share stop closes it)"), + ("goethe api share 5h --key my-app --allow /32", "open a public address for a while (share stop closes it)"), ]), ("Learning and traces", &[ ("goethe learn on", "sessions keep what compaction drops (--model M for one model)"), @@ -911,9 +915,22 @@ fn main() { Some(domain) => goethe_core::domain::check(&domain).map(|text| print!("{text}")), None => goethe_core::domain::explain().map(|text| print!("{text}")), }, - Commands::Api { cmd: Some(ApiCommands::Share(ShareArgs { duration, key, allow, port })), .. } => { - do_share(duration.as_deref().unwrap_or("status"), &key, &allow, port) - } + Commands::Api { + cmd: Some(ApiCommands::Share(ShareArgs { + duration, + key, + allow, + dangerously_allow_all_inbound, + port, + })), + .. + } => do_share( + duration.as_deref().unwrap_or("status"), + &key, + &allow, + dangerously_allow_all_inbound, + port, + ), Commands::Learn(LearnArgs { action, model, print, json }) => do_learn(action.as_deref(), &model, print.as_deref(), json), Commands::Trace(TraceArgs { action, @@ -1085,14 +1102,24 @@ fn do_allow(a: AllowArgs) -> Result<(), String> { fn do_expose(e: ExposeArgs) -> Result<(), String> { // goethe api expose: the status; local|vpn|public runs the access setup; an address or interface is listened on directly. - let ExposeArgs { target, interface, ip, allow, yes, json } = e; - let mode_only = interface.is_some() || ip.is_some() || !allow.is_empty() || yes; + let ExposeArgs { target, interface, ip, allow, yes, dangerously_allow_all_inbound, json } = e; + let mode_only = interface.is_some() || ip.is_some() || !allow.is_empty() || yes || dangerously_allow_all_inbound; match target.as_deref() { None if mode_only => usage("give local, vpn or public with --interface, --ip, --allow or --yes"), None => do_gateway_status(json), Some(m @ ("local" | "vpn" | "public")) => { let mode = goethe_core::configure::Mode::parse(m).unwrap_or_else(|e| usage(e)); - goethe_core::configure::run_api(goethe_core::configure::ApiOpts { mode: Some(mode), vpn_ip: ip, interface, allow, yes }) + if dangerously_allow_all_inbound && mode != goethe_core::configure::Mode::Public { + usage("--dangerously-allow-all-inbound is only for public"); + } + goethe_core::configure::run_api(goethe_core::configure::ApiOpts { + mode: Some(mode), + vpn_ip: ip, + interface, + allow, + yes, + dangerously_allow_all_inbound, + }) } Some(_) if mode_only => usage("--interface, --ip, --allow and --yes go with local, vpn or public"), Some(addr) => do_gateway_bind(addr), @@ -1601,10 +1628,13 @@ fn do_ps(json: bool) -> Result<(), String> { } fn do_gateway_bind(addr: &str) -> Result<(), String> { - // An address, or an interface name whose address the gateway follows; persist, apply, and remind about the allowlist. + // An address or interface; unspecified binds go through expose public with its guards. let home = goethe_home(); ensure_layout(&home).map_err(|e| format!("layout: {e}"))?; let loopback = match addr.trim().parse::() { + Ok(ip) if ip.is_unspecified() => usage( + "listening on every address needs: goethe api expose public --allow /32 --yes", + ), Ok(ip) => { println!("{}", nginx::set_public_bind(&home, &ip.to_string())?); ip.is_loopback() @@ -1613,7 +1643,9 @@ fn do_gateway_bind(addr: &str) -> Result<(), String> { println!("{}", nginx::set_public_bind_interface(&home, addr.trim())?); false } - Err(_) => usage(format!("invalid address {addr:?}: use 0.0.0.0, 127.0.0.1, an IP address of this host or an interface name")), + Err(_) => usage(format!( + "invalid address {addr:?}: use 127.0.0.1, an IP address of this host or an interface name" + )), }; if !loopback && security::allowlist_list()?.is_empty() { print!("\n{}", empty_allowlist_text()); @@ -1710,7 +1742,13 @@ fn do_status() -> Result<(), String> { Ok(()) } -fn do_share(public: &str, keys: &[String], allow: &[String], port: Option) -> Result<(), String> { +fn do_share( + public: &str, + keys: &[String], + allow: &[String], + dangerously_allow_all_inbound: bool, + port: Option, +) -> Result<(), String> { // goethe api share | stop | (status). use goethe_core::public_share as share; let home = goethe_home(); @@ -1736,7 +1774,7 @@ fn do_share(public: &str, keys: &[String], allow: &[String], port: Option) } "status" => { if !keys.is_empty() || !allow.is_empty() || port.is_some() { - usage("give a duration to open the public gateway, such as goethe api share 5h --key my-app"); + usage("give a duration to open the public gateway, such as goethe api share 5h --key my-app --allow /32"); } match share::active(&home) { Some(s) => { @@ -1744,33 +1782,47 @@ fn do_share(public: &str, keys: &[String], allow: &[String], port: Option) println!("Public gateway: open until {} ({} left)", share::local_time(s.until), left(s.until - share::now())); println!("URL: https://{}/v1", host_port(&ip, s.port)); for (name, g) in share::shared(&home) { - let from = if g.allow.is_empty() { "any address".to_string() } else { g.allow.join(", ") }; + let from = if g.allow.is_empty() { "no addresses (unreachable)".to_string() } else { g.allow.join(", ") }; println!("Key: {name} until {} ({} left), from {from}", share::local_time(g.until), left(g.until - share::now())); } println!("\nStop one key with: goethe api share stop --key ; close it with: goethe api share stop"); } - None => println!("The public gateway is closed. Open it with: goethe api share 5h --key "), + None => println!("The public gateway is closed. Open it with: goethe api share 5h --key --allow /32"), } Ok(()) } dur => { let secs = share::parse_duration(dur)?; - let o = share::open(&home, share::OpenArgs { secs, keys, allow, port })?; + let o = share::open( + &home, + share::OpenArgs { + secs, + keys, + allow, + port, + dangerously_allow_all_inbound, + }, + )?; let ip = o.public_ip.map(|i| i.to_string()).unwrap_or_else(|| "".into()); let url = format!("https://{}/v1", host_port(&ip, o.share.port)); println!("Shared until {} ({}); the public gateway is open until {}\n", share::local_time(share::now() + secs), left(secs), share::local_time(o.share.until)); println!("URL: {url}"); for k in &o.keys { let models = k.models.as_deref().filter(|m| !m.is_empty()).unwrap_or("every model"); - let token = k.token.as_deref().unwrap_or("(shown when it was created)"); - println!("Key: {} {token} models: {models}", k.name); + match k.token.as_deref() { + Some(token) => println!("Key: {} {token} models: {models}", k.name), + None => println!("Key: {} (id {}) models: {models}", k.name, k.id), + } } - if !allow.is_empty() { - println!("Only: {}", allow.join(", ")); + let clients: Vec = o.share.allow.clone(); + if clients.iter().any(|c| c.ends_with("/0")) { + println!("Only these clients: every address (--dangerously-allow-all-inbound)"); + } else if !clients.is_empty() { + println!("Only these clients: {}", clients.join(", ")); } println!("\nTLS: self-signed for this address. Send {} with the key (it holds no secret); the recipient", o.cert.display()); println!(" verifies the gateway with it, for example:"); - let token = o.keys.first().and_then(|k| k.token.as_deref()).unwrap_or(""); + let token = o.keys.first().and_then(|k| k.token.as_deref()).unwrap_or(""); println!(" curl --cacert public.crt {url}/models -H \"Authorization: Bearer {token}\""); if let Some(f) = &o.fingerprint { println!(" SHA-256 fingerprint: {f}"); @@ -2029,6 +2081,12 @@ mod tests { assert!(parse(&["api", "expose", "vpn", "--interface", "wg0"]).is_ok() && parse(&["api", "expose", "wg0"]).is_ok()); assert!(matches!(parse(&["api", "share", "5h", "--key", "a,b"]).unwrap().command, Commands::Api { cmd: Some(ApiCommands::Share(ShareArgs { duration: Some(_), .. })), .. })); + let Commands::Api { cmd: Some(ApiCommands::Share(s)), .. } = + parse(&["api", "share", "5h", "--key", "my-app", "--dangerously-allow-all-inbound"]).unwrap().command + else { + panic!() + }; + assert!(s.dangerously_allow_all_inbound && s.allow.is_empty()); assert!(parse(&["models", "alias", "set", "fast", "org/m"]).is_ok() && parse(&["models", "scan", "mydir"]).is_ok()); assert!(parse(&["models", "--json", "scan"]).is_err() && parse(&["api", "--json", "keys"]).is_err()); assert!(parse(&["learn", "on", "--model", "m"]).is_ok() && parse(&["trace", "on", "--all-keys"]).is_ok()); diff --git a/engine/tests/e2e/cuda_shim.c b/engine/tests/e2e/cuda_shim.c new file mode 100644 index 0000000..5c0221a --- /dev/null +++ b/engine/tests/e2e/cuda_shim.c @@ -0,0 +1,79 @@ +/* Stand-in libcuda.so.1 for e2e: SM count and shared memory for the same UUIDs as nvml_shim. + FAKE_CUDA_SMS (default 170) and FAKE_CUDA_SMEM (default 101376) set the attributes. */ +#include +#include +#include + +#define MULTIPROCESSOR_COUNT 16 +#define MAX_SHARED_MEMORY_PER_BLOCK_OPTIN 97 +#define MAX_GPUS 16 + +static int n_devices; + +static int device_count(void) { + const char *spec = getenv("FAKE_NVML"); + char *copy, *entry, *save = NULL; + int n = 0; + if (!spec || !*spec) return 0; + copy = strdup(spec); + for (entry = strtok_r(copy, ";", &save); entry && n < MAX_GPUS; entry = strtok_r(NULL, ";", &save)) { + if (strchr(entry, '|')) n++; + } + free(copy); + return n; +} + +static int sms_of(void) { + const char *v = getenv("FAKE_CUDA_SMS"); + return v && *v ? atoi(v) : 170; +} + +static int smem_of(void) { + const char *v = getenv("FAKE_CUDA_SMEM"); + return v && *v ? atoi(v) : 101376; +} + +int cuInit(unsigned flags) { + (void)flags; + n_devices = device_count(); + return 0; +} + +int cuDeviceGetCount(int *count) { + *count = n_devices; + return 0; +} + +int cuDeviceGet(int *device, int ordinal) { + if (ordinal < 0 || ordinal >= n_devices) return 1; + *device = ordinal; + return 0; +} + +int cuDeviceGetUuid(unsigned char *uuid, int device) { + /* Match nvml_shim: GPU-0000000{i}-0000-0000-0000-000000000000 */ + char hex[33]; + int i; + if (device < 0 || device >= n_devices) return 1; + snprintf(hex, sizeof hex, "%08x000000000000000000000000", (unsigned)device); + for (i = 0; i < 16; i++) { + unsigned v = 0; + sscanf(hex + 2 * i, "%2x", &v); + uuid[i] = (unsigned char)v; + } + return 0; +} + +int cuDeviceGetAttribute(int *pi, int attrib, int device) { + if (device < 0 || device >= n_devices) return 1; + if (attrib == MULTIPROCESSOR_COUNT) { + *pi = sms_of(); + return 0; + } + if (attrib == MAX_SHARED_MEMORY_PER_BLOCK_OPTIN) { + *pi = smem_of(); + return 0; + } + *pi = 0; + return 0; +} diff --git a/engine/tests/e2e/fake/goethe_engine/openai_server.py b/engine/tests/e2e/fake/goethe_engine/openai_server.py index 44685b6..0188ad2 100644 --- a/engine/tests/e2e/fake/goethe_engine/openai_server.py +++ b/engine/tests/e2e/fake/goethe_engine/openai_server.py @@ -1,6 +1,7 @@ """A model server speaking the OpenAI routes goethe uses: health, models, chat, completions, embeddings.""" import json +import os import sys from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer @@ -15,7 +16,8 @@ def option(args, *names): def serve(args): """Run until killed: port, key and served name come from the engine's own flags.""" - port, key = int(option(args, "--port")), option(args, "--api-key") or "" + port = int(option(args, "--port")) + key = option(args, "--api-key") or os.environ.get("GOETHE_ENGINE_API_KEY") or os.environ.get("LLAMA_API_KEY") or "" name = option(args, "--alias", "--served-model-name") or "model" class Handler(BaseHTTPRequestHandler): diff --git a/engine/tests/e2e/fake/uv b/engine/tests/e2e/fake/uv index e8ecdef..0325845 100755 --- a/engine/tests/e2e/fake/uv +++ b/engine/tests/e2e/fake/uv @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""uv for the two commands bootstrap.sh runs: a venv, and pip installs that only lay down dist-info and the fake goethe-engine.""" +"""uv for the commands bootstrap.sh runs: a venv, lockfile installs (-r), and a wheel or editable with no -r.""" import os import re import shutil @@ -18,28 +18,37 @@ def site(python): return next((env / "lib").glob("python3*/site-packages")) +def lay_dist(sp: Path, name: str, version: str) -> None: + name = name.replace("-", "_").lower() + (sp / f"{name}-{version}.dist-info").mkdir(exist_ok=True) + (sp / f"{name}-{version}.dist-info" / "METADATA").write_text(f"Name: {name}\nVersion: {version}\n") + + if args[:1] == ["venv"]: target = next(a for a in args[1:] if not a.startswith("-") and a not in ("3.11",)) shutil.rmtree(target, ignore_errors=True) subprocess.run([sys.executable, "-m", "venv", "--without-pip", target], check=True) elif args[:2] == ["pip", "install"]: python = args[args.index("--python") + 1] - req = Path(args[args.index("-r") + 1]) sp = site(python) time.sleep(float(os.environ.get("FAKE_UV_SECONDS", "0"))) - for line in req.read_text().splitlines(): - m = re.match(r"^([A-Za-z0-9_.-]+)\s*(?:==\s*([^\s;\\]+))?", line.strip()) - if not m or line.lstrip().startswith(("#", "-")): - continue - name, version = m.group(1).replace("-", "_").lower(), m.group(2) or "0" - if name == "goethe_engine": - version = re.search(r"goethe_engine-([0-9.]+)-", line).group(1) - shutil.copytree(FAKE / "goethe_engine", sp / "goethe_engine", dirs_exist_ok=True) - (sp / f"{name}-{version}.dist-info").mkdir(exist_ok=True) - (sp / f"{name}-{version}.dist-info" / "METADATA").write_text(f"Name: {name}\nVersion: {version}\n") - if name == "huggingface_hub": - hf = Path(python).parent / "hf" - hf.write_text(f"#!{os.path.abspath(python)}\n" + (FAKE / "hf").read_text().split("\n", 1)[1]) - hf.chmod(0o755) + if "-r" in args: + req = Path(args[args.index("-r") + 1]) + for line in req.read_text().splitlines(): + m = re.match(r"^([A-Za-z0-9_.-]+)\s*(?:==\s*([^\s;\\]+))?", line.strip()) + if not m or line.lstrip().startswith(("#", "-")): + continue + name, version = m.group(1).replace("-", "_").lower(), m.group(2) or "0" + if name == "goethe_engine": + version = re.search(r"goethe_engine-([0-9.]+)-", line).group(1) + shutil.copytree(FAKE / "goethe_engine", sp / "goethe_engine", dirs_exist_ok=True) + lay_dist(sp, name, version) + if name == "huggingface_hub": + hf = Path(python).parent / "hf" + hf.write_text(f"#!{os.path.abspath(python)}\n" + (FAKE / "hf").read_text().split("\n", 1)[1]) + hf.chmod(0o755) + else: + # Wheel path or editable: bootstrap's GOETHE_SGLANG_WHEEL / -e fork install. + lay_dist(sp, "sglang", "0.0.0.dev1+e2e") else: sys.exit(f"fake uv: unsupported {args}") diff --git a/engine/tests/e2e/world.py b/engine/tests/e2e/world.py index 04c2a2d..7c4a660 100644 --- a/engine/tests/e2e/world.py +++ b/engine/tests/e2e/world.py @@ -424,6 +424,10 @@ def __init__( ["cc", "-shared", "-fPIC", "-o", str(shim / "libnvidia-ml.so.1"), str(HERE / "nvml_shim.c")], check=True ) (shim / "libnvidia-ml.so").symlink_to("libnvidia-ml.so.1") + subprocess.run( + ["cc", "-shared", "-fPIC", "-o", str(shim / "libcuda.so.1"), str(HERE / "cuda_shim.c")], check=True + ) + (shim / "libcuda.so").symlink_to("libcuda.so.1") shutil.copytree(FAKE, self.fakes / "fake") ports = [] for _ in range(2): diff --git a/engine/tests/lib/goethe_test.py b/engine/tests/lib/goethe_test.py index fe9388d..2e38b4f 100644 --- a/engine/tests/lib/goethe_test.py +++ b/engine/tests/lib/goethe_test.py @@ -728,8 +728,10 @@ def write_stub_python(pyenv_bin: Path, real: Path) -> None: "#!/bin/bash\n" f'REAL="{real}"\n' f'STUB="{STUB_ENGINE}"\n' - 'if [[ "$1" == "-m" && "$2" == "sglang.launch_server" ]]; then\n' - ' exec "$REAL" "$STUB" "${@:3}"\n' + 'if [[ "$1" == "-m" && ( "$2" == "sglang.launch_server" || "$2" == "goethe_sglang_boot" ) ]]; then\n' + ' EXTRA=()\n' + ' if [[ -n "${GOETHE_ENGINE_API_KEY:-}" ]]; then EXTRA=(--api-key "$GOETHE_ENGINE_API_KEY"); fi\n' + ' exec "$REAL" "$STUB" "${@:3}" "${EXTRA[@]}"\n' "fi\n" 'if [[ "$1" == "-c" && "$2" == *sglang* ]]; then echo ok; exit 0; fi\n' 'exec "$REAL" "$@"\n', diff --git a/engine/tests/lib/stub_engine.py b/engine/tests/lib/stub_engine.py index 533cda4..917095c 100644 --- a/engine/tests/lib/stub_engine.py +++ b/engine/tests/lib/stub_engine.py @@ -18,6 +18,7 @@ RECORDED_ENV = ( "CUDA_VISIBLE_DEVICES", "GOETHE_HOME", "PYTHONPATH", "GOETHE_INSTANCE_ID", "GOETHE_KERNEL_DIR", "SPT_NOENV", + "GOETHE_ENGINE_API_KEY", "LLAMA_API_KEY", ) FILE_KNOBS = ("STUB_START_DELAY", "STUB_IGNORE_SIGTERM", "STUB_SIGTERM_DELAY", "STUB_TOKEN_INTERVAL") AUTH_EXEMPT = ("/health", "/ready", "/metrics") @@ -39,6 +40,8 @@ def _parse_args(argv: list[str]) -> argparse.Namespace: parser.add_argument("--api-key") parser.add_argument("--disable-custom-all-reduce", action="store_true") args, _unknown = parser.parse_known_args(argv) + if not args.api_key: + args.api_key = os.environ.get("GOETHE_ENGINE_API_KEY") or os.environ.get("LLAMA_API_KEY") return args diff --git a/engine/tests/stage5_configure_api.py b/engine/tests/stage5_configure_api.py index 9a65453..6bac324 100644 --- a/engine/tests/stage5_configure_api.py +++ b/engine/tests/stage5_configure_api.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""goethe api expose without a terminal: local, public with named clients, public refusing 0.0.0.0/0, VPN.""" +"""goethe api expose without a terminal: local, public with named clients, refuse bare --yes / 0.0.0.0 /0, VPN.""" from __future__ import annotations @@ -22,13 +22,21 @@ def main() -> None: if rr.returncode == 0 or _status(goethe, env)["bind"] == "0.0.0.0": raise RuntimeError(f"without a terminal api expose public needs --yes: {rr.stdout} {rr.stderr}") rr = run([goethe, "api", "expose", "local", "--yes"], env=env) - if rr.returncode != 0 or "API key (default): goethe_sk_" not in rr.stdout: - raise RuntimeError(f"local mode with --yes must bind loopback and create a key: {rr.stdout} {rr.stderr}") - if _status(goethe, env)["bind"] != "127.0.0.1": - raise RuntimeError("local mode must bind 127.0.0.1") + if rr.returncode != 0 or _status(goethe, env)["bind"] != "127.0.0.1": + raise RuntimeError(f"local mode with --yes must bind loopback: {rr.stdout} {rr.stderr}") + rr = run([goethe, "api", "expose", "public", "--yes"], env=env) + if rr.returncode == 0 or _status(goethe, env)["bind"] != "127.0.0.1": + raise RuntimeError( + f"public --yes without --allow must refuse and leave bind unchanged: {rr.stdout} {rr.stderr}" + ) + if "--allow" not in (rr.stdout + rr.stderr): + raise RuntimeError(f"refusal must name --allow: {rr.stdout} {rr.stderr}") + rr = run([goethe, "api", "expose", "0.0.0.0"], env=env) + if rr.returncode == 0 or _status(goethe, env)["bind"] != "127.0.0.1": + raise RuntimeError(f"raw expose 0.0.0.0 must refuse: {rr.stdout} {rr.stderr}") rr = run([goethe, "api", "expose", "public", "--yes", "--allow", "0.0.0.0/0"], env=env) if rr.returncode == 0 or _status(goethe, env)["bind"] != "127.0.0.1": - raise RuntimeError(f"public mode must refuse 0.0.0.0/0 without a terminal and change nothing: {rr.stdout}") + raise RuntimeError(f"public mode must refuse 0.0.0.0/0 without dangerously and change nothing: {rr.stdout}") quiet = {**env, "GOETHE_PUBLIC_IP_LOOKUP": "0"} rr = run([goethe, "api", "expose", "public", "--yes", "--allow", "203.0.113.7"], env=quiet) if "Your public IP: unknown, lookup disabled" not in rr.stdout: @@ -42,8 +50,8 @@ def main() -> None: if rr.returncode == 0 or "not an address of this host" not in rr.stderr: raise RuntimeError(f"a VPN address this host does not have must be refused: {rr.stdout} {rr.stderr}") rr = run([goethe, "api", "expose", "local", "--yes"], env=env) - if rr.returncode != 0 or "API keys: 1 exist" not in rr.stdout: - raise RuntimeError(f"a second run must keep the existing key: {rr.stdout} {rr.stderr}") + if rr.returncode != 0 or _status(goethe, env)["bind"] != "127.0.0.1": + raise RuntimeError(f"local again must succeed: {rr.stdout} {rr.stderr}") print("stage5_configure_api ok")