Skip to content
meshappliedPublic

About

Serve your models fast on your own infra in 3 steps.

Topics

Resources

Stars

81 stars

Watchers

0 watching

Forks

Repository files navigation

Goethe

NVIDIA CUDA 12.8 or newer GPU architectures sm_86 to sm_120 Written in Rust Linux x86_64 and aarch64 Apache 2.0 license CI status
Documentation at goethe.sh/docs Join us on Discord

Run and share your models on your own infra in 3 steps, and run them fast.

If Goethe is useful to you, a ⭐ on GitHub helps others find it.

Goethe turns a Linux machine into your own API, compatible with OpenAI, Anthropic and LiteLLM clients. Install it, add a model, and point your apps, your team or your other machines at one address. Your prompts, your data and your models stay on hardware you control.

curl -fsSL https://goethe.sh/install.sh | bash
goethe

goethe on its own starts Goethe and, the first time, walks you through three steps: install a model, serve it through the API, and configure who can reach it. If you prefer plain commands, these do the same:

# 1. Start Goethe and load a model
goethe up
goethe install nvidia/Llama-3.1-8B-Instruct-NVFP4 --load

# 2. Share it
goethe api expose vpn                # or local, or public: this machine only, your VPN (recommended), or a public address
goethe api keys create my-laptop         # one key per app or person
goethe api                           # the address to give your apps

On a 32 GB Blackwell GPU, goethe models recommend lists meshapplied/Qwen3.8-27B-NVFP4-NInfer first.

Anything that speaks the OpenAI, Anthropic or LiteLLM API can use it right away, all at the same address: the OpenAI and Anthropic SDKs, LiteLLM, LangChain, LlamaIndex, Open WebUI, coding assistants, or plain curl.

from openai import OpenAI

client = OpenAI(base_url="https://<your-goethe-address>:8443/v1", api_key="goethe_sk_...")
reply = client.chat.completions.create(
    model="nvidia/Llama-3.1-8B-Instruct-NVFP4",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(reply.choices[0].message.content)

Why Goethe

  • Easy to serve. Every model you load is served at the same URL; a request picks its model by name.
  • Fast. Use the custom kernel attachments to accelerate and optimise model capability.
  • Private by default. Nothing reaches the API until you allow it. Rate-limit, expire, scope or revoke keys at any time.
  • Fits your network. This machine only, a VPN, or a public IP. goethe api expose sets it.
  • Many models, many GPUs. Run several models side by side, split large models across GPUs, and keep other GPU software running next to Goethe.
  • Always on. Loaded models come back after a reboot or a crash.

Where it runs

Environment Examples
Your own hardware Workstations and servers with one or more NVIDIA GPUs, including DGX Spark
Cloud GPU virtual machines AWS, Google Cloud, Azure, Oracle Cloud, Lambda, CoreWeave, Paperspace
GPU containers RunPod, Vast.ai, and Docker with the NVIDIA Container Toolkit

Linux on x86_64 or ARM64 (glibc 2.30 or newer, for example Ubuntu 20.04+ or Debian 11+). Goethe recognises the cloud provider or container it runs in and shows the address to use from outside.

GPU and CUDA compatibility

Architecture Compute capability GPUs
Blackwell 12.0, 12.1 GeForce RTX 5090, RTX PRO 6000 Blackwell, DGX Spark (GB10)
Blackwell data center 10.0, 10.3 B200, GB200, B300, GB300
Hopper 9.0 H100, H200, GH200
Ada 8.9 GeForce RTX 4090, RTX 6000 Ada, L40S, L40, L4
Ampere 8.0, 8.6 A100, A30, A40, A10, RTX A6000, GeForce RTX 3090

Older GPUs (Volta, Turing) are not supported. A machine can mix GPU generations; each model runs on GPUs of one generation, and large models can be split across several GPUs.

Goethe ships its own CUDA 13 runtime and compiler, so no CUDA toolkit is needed on the machine. What it needs is the NVIDIA driver:

NVIDIA driver CUDA it offers Goethe runs on
580 or newer 13.0 and newer Every supported GPU
535 to 579 12.2 to 12.9 Data-center GPUs (A100, H100, H200, B200, L40S, L4 and similar), through NVIDIA's CUDA forward compatibility
Older than 535 Not supported; the installer offers to update the driver

GeForce and TITAN cards need driver 580 or newer. Enough GPU memory for the model is the only other requirement; goethe models recommend lists the models that fit.

The llama.cpp build goethe downloads for GGUF models uses CUDA 12.8 libraries; NVIDIA lists the minimum driver of each CUDA release in its CUDA compatibility table. Drivers older than the 580 line above are unverified with goethe.

Supported model families and quantization formats are in the user guide. GGUF models (single GPU) and GGUF embedding models are served through llama.cpp.

Known limits

  • Gated models need the license accepted on Hugging Face and a read token (install with --hf-token). This includes orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4.
  • Optimised kernel packs (the 27B NVFP4 model, the ternary models and others) need goethe login on the node. A model that has only a pack and none for your GPU is refused before anything is downloaded.
  • If a model's kernel pack cannot run on your GPU, install or load stops with the reason. Add --fallback (or set GOETHE_ALLOW_FALLBACK=1) to serve it with the plain runtime instead.

Documentation

Repository layout

Path What it is
goethe/ Goethe: the goethe command, the background service, the gateway and the installer
lab/ goethe-lab: fit/registry helpers used in development and tests
nvfp4-kernels/, ternary-kernels/ GPU kernels behind our models
docs/ User guide, command reference and model notes

Our models on Hugging Face and at goethe.sh/models.

License

Goethe is licensed under the Apache License 2.0. The license of each model you serve still applies.

Goethe is built on third-party software, among them SGLang, PyTorch, FlashInfer, Triton, Hugging Face Transformers, nginx and NVIDIA's CUDA libraries. NOTICE lists them with their licenses.

About

Serve your models fast on your own infra in 3 steps.

Topics

Resources

Stars

81 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages