Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

## Unreleased

- `goethe api expose public --yes` without `--allow` no longer binds every interface on a leftover allowlist; name the clients and confirm (`--yes` or `y`). Raw `goethe api expose 0.0.0.0` is refused (use `expose public`). World-open clients need the hidden `--dangerously-allow-all-inbound --yes`.
- The gateway TLS certificate SAN includes this host's interface IPs (tailnet, LAN), so `curl --cacert` to the bind address verifies; the cert refreshes when that list changes.
- `goethe models scan <foldername>` resolves the name under `GOETHE_HOME/models`, not the current working directory.
- goethe picks the engine before downloading: a goethe kernel pack on a logged-in node, else SGLang when it supports the model, else llama.cpp on the repo's GGUF files or on the GGUF build Hugging Face lists for the model (the name typed becomes an alias), else a refusal with each engine's reason. Embedding models like embeddinggemma install this way. On a Mac, llama.cpp with the same GGUF lookup instead of a name search.
- A kernel pack that cannot run gives way to the plain runtime by default, saying why (`GOETHE_ALLOW_FALLBACK=0` restores the error). The pack structure check after a download, and the goethe-engine install it needs, run only when a pack of the model's architecture exists for the GPU.
- `goethe models` shows each model's engine.
Expand Down
32 changes: 17 additions & 15 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,24 +271,24 @@ Who can reach the API. Without an argument it shows where the API listens and ho
| `goethe api expose` | Show where the API listens and who may connect (`--json` for scripts) |
| `goethe api expose local` | This machine only |
| `goethe api expose vpn` | A VPN or other private network (recommended) |
| `goethe api expose public` | A public address |
| `goethe api expose <address or interface>` | Listen on `127.0.0.1` (this machine only), `0.0.0.0` (all addresses), one IP of this machine, or an interface such as `wg0`, whose address is followed |
| `goethe api expose public` | A public address (needs `--allow <their-ip>/32` and `--yes`, or `y` on a terminal) |
| `goethe api expose <address or interface>` | Listen on `127.0.0.1`, one IP of this machine, or an interface such as `wg0` (followed). Not `0.0.0.0`: use `expose public` |

For a VPN it lists the networks this machine is on, with the ranges each one reaches, and works with any VPN that has a network interface. It then asks whether to restrict access to specific IP addresses. The gateway follows the chosen interface: when its address changes, it moves within a few seconds, and while the interface has no address only this machine can connect.

`expose public` never binds every interface on a leftover allowlist. Name the clients with `--allow`, confirm with `--yes` (or `y`), and only then does it listen on `0.0.0.0`. Opening every client address needs the hidden `--dangerously-allow-all-inbound --yes`.

| Option | Description |
|---|---|
| `--interface <name>` | With `vpn` or `public`: the network interface to serve, such as `wg0`; needed without a terminal when there are several |
| `--ip <ip>` | With `vpn` or `public`: a fixed address of this machine instead of an interface |
| `--allow <cidr>` | With `vpn` or `public`: a client address or range to allow; repeatable |
| `--yes` | Apply without questions (`public` still refuses `0.0.0.0/0`) |
| `--interface <name>` | With `vpn`: the network interface to serve, such as `wg0`; needed without a terminal when there are several |
| `--allow <cidr>` | With `vpn` or `public`: a client address or range to allow; **required for public**; repeatable |
| `--yes` | Apply without questions. Public still needs `--allow` (or `--dangerously-allow-all-inbound`) |
| `--json` | With no argument: the status as JSON |

```bash
goethe api expose local
goethe api expose vpn --interface wg0
goethe api expose public --allow 203.0.113.7/32
goethe api expose 0.0.0.0
goethe api expose public --allow 203.0.113.7/32 --yes
```

### `goethe api allow`
Expand Down Expand Up @@ -318,11 +318,12 @@ The DNS record that serves the API under your own domain name, with this machine
Opens the API on a public address for a limited time, for API keys you choose, so you can give someone outside your network a URL and a key. The normal API address, local or VPN, stays as it is.

```
goethe api share 5h --key my-app # share my-app for 5 hours
goethe api share 1d --key ci --allow 203.0.113.7 # also share ci, for a day, from one address
goethe api share # status: URL, each key with its time left and addresses
goethe api share stop --key ci # stop sharing one key
goethe api share stop # close the public gateway
goethe api share 5h --key my-app --allow 203.0.113.7/32 # prints the URL, the key and the certificate to send
goethe api share 5h --key my-app --dangerously-allow-all-inbound # every address (rate limits still apply)
goethe api share 5h --key my-app # reopen: keeps addresses from the last share of those keys
goethe api share # status: URL, each key with its time left and addresses
goethe api share stop --key ci # stop sharing one key
goethe api share stop # close the public gateway
```

Every shared key uses the same public gateway, each until its own time and from its own addresses. The gateway stays open until the last key's time ends. A key whose time has ended is refused while the others keep working, and the addresses only it allowed are dropped from the gateway. Sharing a key again starts its time over (`5h` means 5 hours from now) and replaces its addresses; the other keys are not touched. Without `--key`, the keys of the last opening are shared again. Opening prints the URL, each key with the models it may use, the certificate to send along, and a `curl` command to try it. The keys stay the same, so a recipient can keep using the key you sent. Connections in progress are not interrupted. The menu's Public address page does the same with a form: the key, how long, and optional addresses. `goethe status` shows the gateway while it is open, and a node linked to the platform shows the public URL and when it closes on its page.
Expand All @@ -331,7 +332,8 @@ Every shared key uses the same public gateway, each until its own time and from
|---|---|
| `<duration>` | How long to share the keys: minutes, hours or days, such as `30m`, `5h` or `1d`, at most `7d`. `stop` stops sharing; nothing shows the status |
| `--key <name or id>` | API keys to share, each until the duration; repeatable or comma-separated. With `stop`, the keys to stop sharing |
| `--allow <cidr>` | Only these client addresses or ranges may use the keys given, such as `203.0.113.7`; repeatable. Without it any address may, with the key |
| `--allow <cidr>` | Only these client addresses or ranges may use the keys given, such as `203.0.113.7`; repeatable. Without it, addresses from the last share of those keys are kept; with none, Goethe refuses to open |
| `--dangerously-allow-all-inbound` | Allow every address (`0.0.0.0/0`). Rate limits still apply (10 requests/s per client, burst 20, 16 connections). Not with `--allow` |
| `--port <port>` | The public port (default `8444`, or the last one used) |

The recipient verifies the gateway with the certificate goethe prints, `GOETHE_HOME/nginx/certs/public.crt`. It holds no secret: `curl --cacert public.crt https://<address>:8444/v1/models -H "Authorization: Bearer <key>"`. Most OpenAI-compatible clients take it through `SSL_CERT_FILE` or their CA-bundle setting. If nobody can connect, forward the port to this machine on your router or allow it in your cloud firewall; goethe does not open ports on routers. The public listener uses IPv4.
Expand All @@ -347,7 +349,7 @@ The recipient verifies the gateway with the certificate goethe prints, `GOETHE_H
- **API routes only:** health, readiness, admin and metrics endpoints are not served publicly.
- **HTTPS only:** TLS 1.2 or 1.3, with its own certificate for the public listener. Plain HTTP connections are dropped.
- **Stricter limits than the normal address:** 10 requests per second per client address (bursts of 20) and 16 connections per address, plus the same body size limit and header and body timeouts.
- **Optional address limit per key:** a key shared with `--allow` works only from those addresses, checked on every request; nginx also admits only the addresses of the shared keys when every key has a limit.
- **Client address required:** every shared key has an allowlist (`--allow`, or addresses kept from the last share of those keys). Nginx admits only that union and always ends with `deny all`. Every address (`0.0.0.0/0`) needs `--dangerously-allow-all-inbound`; this machine's own public IP is refused as `--allow`. Goethe also checks the client on every public request.
- **Checks before opening:** nothing changes unless every check passes: the gateway and goethed are running, the keys are valid, and the port is free and is not the normal gateway's port. If nginx does not accept the new listener, the previous state is restored.

## Learning and traces
Expand Down
20 changes: 20 additions & 0 deletions docs/e2e-findings-pr180.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# E2E findings (PR180)

Isolated-home session against the PR binary. Session logs with host paths and keys stay off-repo.

## Status

| Sev | Finding | Status |
|-----|---------|--------|
| P0 | `expose public --yes` binds every interface without a client `--allow` | Fixed |
| P0 | Gateway cert SAN missing interface IPs (`curl --cacert` to the bind address fails) | Fixed |
| P1 | `models scan <foldername>` resolved from the current directory | Fixed |
| P1 | Symlink under `models/` refused by scan | Intentional; use `goethe install <path> --name` |
| P1 | Non-TTY load next to a display manager needs `--yes` | Intentional |
| P2 | A second `GOETHE_HOME` may pick another gateway port | Warned; read status URL |
| P2 | `goethe run` has no `-p`; input is positional | Documented |

## CI coverage

- Unit: public `--yes` without `--allow` refuses; scan folder names under `models/`; cert SAN includes interface IPs
- `engine/tests/stage5_configure_api.py`: bare public `--yes`, raw `0.0.0.0`, and `0.0.0.0/0` refuse; named client allow succeeds
14 changes: 7 additions & 7 deletions docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ Models that were loaded when Goethe stopped come back when it starts again, incl
| `goethe trace` | Record requests and replies as training traces (`on`, `off`, `--key`); see [traces.md](traces.md) |
| `goethe learn` | Let a session keep what compaction would drop (`on`, `off`, `--model`); see [learn.md](learn.md) |
| `goethe api domain` | DNS record for your own hostname (`check <name>`) |
| `goethe api share 5h --key <name>` | Open the API on a public address for a while for chosen keys (`stop` closes it) |
| `goethe api share 5h --key <name> --allow <their-ip>/32` | Open the API on a public address for a while for chosen keys from their IP (`stop` closes it) |

### GPUs and system

Expand Down Expand Up @@ -307,7 +307,7 @@ goethe api expose vpn # or local, or public; asks who may connect
- **VPN or private network (recommended):** connect your VPN first, then pick it from the list. Any VPN with a network interface works, and so does a cloud VM's private network. Goethe lists tunnels and interfaces on private addresses, leaves out container bridges and public network cards, and shows the range each one reaches, taken from the system's routes. When a VPN routes each device separately, the private block its address is in (such as `100.64.0.0/10` or `10.0.0.0/8`) is used, so devices added later can connect too; you can narrow it to specific addresses. Default routes are never allowed. The gateway follows the interface: a new address after a reconnect or a reboot is picked up within a few seconds, and while the interface is down only this machine can connect (`goethe status` says so). The allowlist itself only changes when you run `goethe api expose` again; `goethe doctor` warns if it no longer lets the network's clients in. A VPN without an interface (userspace mode) can't be listened on directly: use local mode and the VPN's own port forwarding.
- **Public network:** any address that can reach this machine. You open the port in your firewall yourself.

It then asks whether to restrict access to specific IP addresses, and offers to create the first key. In scripts: `goethe api expose vpn --yes`, or `goethe api expose public --allow 203.0.113.7 --yes`.
It then asks whether to restrict access to specific IP addresses, and offers to create the first key. In scripts: `goethe api expose vpn --yes`, or `goethe api expose public --allow 203.0.113.7/32 --yes`. Public always needs `--allow` and a confirm (`--yes` or `y`); it will not open every interface on a leftover allowlist alone.

### Keys

Expand Down Expand Up @@ -339,13 +339,13 @@ goethe api allow remove 192.168.1.0/24
To give someone outside your network access for a limited time, without changing how the API is normally reached:

```bash
goethe api share 5h --key my-app # prints the URL, the key and the certificate to send
goethe api share 1d --key ci # another key on the same address, for a day
goethe api share stop --key ci # stop sharing one key
goethe api share stop # close early
goethe api share 5h --key my-app --allow 203.0.113.7/32 # prints the URL, the key and the certificate to send
goethe api share 1d --key ci --allow 203.0.113.7/32 # another key on the same address, for a day
goethe api share stop --key ci # stop sharing one key
goethe api share stop # close early
```

The public address exists only while some key is shared, accepts only the keys you chose (never admin keys), each until its own time, and closes by itself when the last one ends. Keys stay the same every time. The menu's Public address page does the same with a form. Details and the full list of security precautions: [`goethe api share`](cli.md#goethe-api-share-duration).
The public address exists only while some key is shared, accepts only the keys you chose (never admin keys), each until its own time and from the client addresses you named with `--allow` (or kept from the last share of those keys), and closes by itself when the last one ends. Keys stay the same every time. The menu's Public address page does the same with a form. Details and the full list of security precautions: [`goethe api share`](cli.md#goethe-api-share-duration).

### Address, aliases and your own domain

Expand Down
4 changes: 0 additions & 4 deletions engine/crates/core/src/calc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,6 @@ const CUDA_CONTEXT_GB: f64 = crate::engine_overhead::DEFAULT_CONTEXT_GB;
pub const BASE_OVERHEAD_GB: f64 = 1.8;
pub const METAL_OVERHEAD_GB: f64 = 0.5;

pub fn base_overhead_gb(cc: &str) -> f64 {
// llama.cpp on Metal has no CUDA context or graph pools, only its compute buffers.
if crate::gpu_arch::is_apple(cc) { METAL_OVERHEAD_GB } else { BASE_OVERHEAD_GB }
}
const LEARN_SLACK_GB: f64 = 0.25;

#[derive(Debug, Clone, Copy, PartialEq, Default)]
Expand Down
18 changes: 10 additions & 8 deletions engine/crates/core/src/chooser.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Tensor parallel size and GPU set selection for goethe load.

use crate::calc::{base_overhead_gb, missing_gb, shortfall, Need};
use crate::calc::{missing_gb, shortfall, Need, METAL_OVERHEAD_GB};
use crate::memory_spec::MemorySpec;

const GIB: f64 = 1024.0 * 1024.0 * 1024.0;
Expand All @@ -23,6 +23,7 @@ pub struct ChooserInput {
pub supported_sm: Vec<String>,
pub memory: Option<MemorySpec>,
pub window_gb_per_request: f64,
pub overhead_gb: f64,
}

pub struct CardFacts {
Expand Down Expand Up @@ -50,6 +51,7 @@ impl ChooserInput {
marlin_extra_gb: crate::install::card_marlin_extra_gb(card),
supported_sm: crate::install::card_supported_sm(card),
memory,
overhead_gb: crate::engines::of_card(card).map_or(crate::calc::BASE_OVERHEAD_GB, |e| e.overhead_gb),
}
}
}
Expand Down Expand Up @@ -131,13 +133,14 @@ pub fn need(input: &ChooserInput, tp: usize, cc: &str) -> Need {
let tokens = kv_tokens(input.ctx, input.max_conc) as f64;
let conc = input.max_conc.max(1);
let request_table = ((conc + 1) * (input.ctx + 4) * 4) as f64 / GIB;
let oh = if gpu_arch::is_apple(cc) { METAL_OVERHEAD_GB } else { input.overhead_gb } + request_table;
let Some(m) = &input.memory else {
let shard = kv_shard(tp, input.num_kv_heads, &input.attention);
return Need {
weights: (input.static_vram_gb + marlin) / tp as f64,
kv: tokens * input.kv_bytes_per_token as f64 / GIB / shard,
window: conc as f64 * input.window_gb_per_request / shard,
overhead: base_overhead_gb(cc) + request_table,
overhead: oh,
..Need::default()
};
};
Expand All @@ -152,7 +155,7 @@ pub fn need(input: &ChooserInput, tp: usize, cc: &str) -> Need {
/ tp as f64
+ ((conc + 1) * m.request_bytes + m.fixed_bytes) as f64)
/ GIB,
overhead: base_overhead_gb(cc) + request_table,
overhead: oh,
load_peak: m.load_peak_bytes as f64 / tp as f64 / GIB,
}
}
Expand All @@ -168,10 +171,6 @@ pub fn cap_default_ctx(input: &ChooserInput, tp: usize, cc: &str, free_gb: f64)
trial.ctx
}

pub fn per_gpu_budget_gb(input: &ChooserInput, tp: usize, cc: &str) -> f64 {
need(input, tp, cc).total()
}

pub fn with_learned(base: Need, learned: Learned) -> Need {
// Fold what a model taught us on a GPU type and tp into the budget parts: static extra and non-static overhead.
Need {
Expand Down Expand Up @@ -464,13 +463,16 @@ mod tests {
marlin_extra_gb: 0.0,
supported_sm: Vec::new(),
memory: None,
overhead_gb: crate::calc::BASE_OVERHEAD_GB,
}
}

#[test]
fn metal_pays_no_cuda_overhead() {
let (cuda, metal) = (need(&inp(), 1, "12.0"), need(&inp(), 1, "metal"));
assert!((cuda.overhead - metal.overhead - (crate::calc::BASE_OVERHEAD_GB - crate::calc::METAL_OVERHEAD_GB)).abs() < 1e-9);
assert!((cuda.overhead - metal.overhead - (crate::calc::BASE_OVERHEAD_GB - METAL_OVERHEAD_GB)).abs() < 1e-9);
let llama = need(&ChooserInput { overhead_gb: METAL_OVERHEAD_GB, ..inp() }, 1, "12.0");
assert!((llama.overhead - metal.overhead).abs() < 1e-9);
}

fn gpu(idx: u32, free_gb: f64, foreign: bool) -> GpuPlacementInput {
Expand Down
Loading
Loading