llmman serve listens on 127.0.0.1:17434 by default (LLMMAN_HOST
changes it; see configuration.md) and speaks the
Ollama, OpenAI and Anthropic wire formats, plus a small API of its own.
| API | Endpoints |
|---|---|
| Ollama | /api/generate, /api/chat, /api/embed, /api/embeddings, /api/tags, /api/show, /api/pull, /api/push, /api/copy, /api/create, /api/blobs/{digest}, /api/ps, /api/delete, /api/version |
| OpenAI | /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, /v1/responses, /v1/responses/input_tokens, /v1/audio/transcriptions (also /audio/transcriptions), /v1/systemone (llama-server's own API for decision models) |
| Anthropic | /v1/messages |
| llmman | /llmman/providers, /llmman/providers/{id}, /llmman/providers/{id}/key, /llmman/node, /llmman/search, /llmman/search/model, /llmman/search/popular, /llmman/search/avatar, /llmman/shell |
| Web UI | /, /ui/* and, for a browser, any other path outside the prefixes above (/models, /c/<id>) — see webui.md |
| llama.cpp | /props |
| Prometheus | /metrics (off unless LLMMAN_METRICS is 1, true, yes or on) |
Every route but the web UI's own files requires an API key when the daemon has any configured; see Authentication.
Use it as a drop-in Ollama server:
OLLAMA_HOST=127.0.0.1:17434 ollama run unsloth/Qwen3.5-0.8B-GGUF
Or with any Ollama, OpenAI or Anthropic client. http://127.0.0.1:17434/
in a browser is llmman's own web UI (webui.md).
Models load on demand, each in its own backend subprocess
(llama-server, vllm, sglang, mlx_lm.server or ds4-server; see backends.md)
on a random loopback port, reused by later requests.
An idle model unloads after keep_alive (default 5 minutes, as in
Ollama; per-request, or daemon-wide via LLMMAN_KEEP_ALIVE);
llmman ps//api/ps report each model's expires_at.
LLMMAN_MAX_LOADED_MODELS caps how many stay loaded, evicting the
least-recently-used idle one. llmman stop <model> (or keep_alive: 0
on /api/generate) unloads immediately.
/api/chat supports Ollama's tools (function calling, streamed back as
message.tool_calls with their id, and matched to tool_call_id on
the way back in), images (vision, base64, same as Ollama's own wire
format), think, and format ("json" or a JSON Schema object, for
constrained structured output). /api/generate takes system and
images too; raw, suffix and template are refused with a 400,
since llmman leaves templating to the backend.
Of options, temperature, top_p, top_k, min_p, seed, stop,
num_predict, repeat_penalty, presence_penalty, frequency_penalty,
num_thread and num_ctx apply. num_ctx is the window a llama-server
model is loaded with, over LLMMAN_CONTEXT_LENGTH (LLMMAN_EMBEDDING_CONTEXT_LENGTH for an embedding model) and cut to its trained
context. As in Ollama, a model loaded with another window is reloaded once
its requests finish, and requests that arrive meanwhile wait. A request with
no num_ctx uses whichever window is loaded. The done
chunk carries Ollama's counts and durations (prompt_eval_count,
eval_count, total_duration, ...) and done_reason is stop or
length as in Ollama, which reports a tool-calling turn as stop.
prompt_eval_count is the whole prompt, as in Ollama, and
prompt_eval_cached_count how much of it came from the cache.
/api/embed and /api/embeddings work with any GGUF embedding model
(one with a pooling type, e.g. embeddinggemma, nomic-embed-text):
llama-server is started with --embeddings for it, so /v1/embeddings
works too. /api/embed and /v1/embeddings honour dimensions: each vector
is cut to that many values and re-normalised. An input longer than the
model's context is cut to it by /api/embed (unless truncate is false)
and is a 400 on /v1/embeddings, as on OpenAI's.
/api/tags lists each model with the details /api/show gives it (format,
family, parameter_size, quantization_level), read from the GGUF header
once per model.
think is true/false or a level (minimal, low, medium,
high, xhigh, max), forwarded as llama-server's
chat_template_kwargs. /api/show returns capabilities and, for a
local model with one, template. They are completion (embedding for a
pooling model, decision for a decision model), vision with a projector,
tools when the chat template renders a request's tools, and thinking
when it handles reasoning. A llama-server model that thinks also gets Ollama's
thinking: {"values": [...], "default": ...}: false plus the levels its
template names (Qwen3.8: low, medium, xhigh), or minimal, low, medium
and high for a switch-only template (Qwen3.5, Gemma 4). On /api/chat and
/api/generate a think level outside values becomes the default, as in
Ollama; a think that is not a boolean or string is a 400.
On /api/chat and /api/generate, think: "minimal" and "low" also
get the thinking budget a local /v1/chat/completions gives those
levels (see OpenAI API notes), sized from
num_predict.
/api/create supports from (alias a model) and files (GGUFs uploaded
via /api/blobs/{digest}, as ollama create does). Modelfile fields such
as system or quantize are refused with a 400: the GGUF's own chat
template applies.
These are not 500s:
- A model that is neither in the store nor on its registry (or Hugging Face)
is a
404naming it, on every route that loads one. Docker Hub and Hugging Face answer an anonymous request for a missing repository as for a private one; that is a404too, with their words after the name. Any other failed pull, a refused login or another registry's denial included, is a500. - A prompt longer than the context is a
400on/api/chat,/api/generateand/v1/messages, as on/v1/chat/completions.
A chat template that needs alternating roles (Gemma, Llama 2, Mistral)
refuses two messages of one role in a row. The chat routes then retry once
with adjacent user (or assistant) messages merged, joined by a blank
line; a message with tool calls is never merged.
/v1/models carries llama-server's meta and architecture, which clients
size a model by (meta.n_ctx_train is OpenClaw's context window). A loaded
llama-server's own are passed through, meta.n_ctx included. Before a model
is loaded, meta.n_ctx_train comes from the GGUF header, meta.n_ctx is the
per-slot context the llama-server a load starts would report (from
LLMMAN_CONTEXT_LENGTH (LLMMAN_EMBEDDING_CONTEXT_LENGTH for an embedding model) and LLMMAN_NUM_PARALLEL, never past n_ctx_train; a
request's num_ctx or an out-of-memory retry can still change it), and
architecture.input_modalities comes from what the model can do (text, plus
image with a vision projector); a model that does not generate text has no
architecture. meta.n_ctx is left out for a model llama-server does not
serve (media, or a GGUF that goes to ds4-server), and when LLMMAN_LLAMA_ARGS
or a LLAMA_ARG_* variable sets llama-server's context size, slots or KV
unification itself.
/props is llama-server's router-mode endpoint: the router's own, or with
?model= that model's llama-server's (context window, slots, modalities,
chat template). A model that is not a loaded llama-server is a 404. Unlike
llama-server's router it never loads a model (autoload is ignored): a GET
that started one is one any web page could send.
/v1/responses implements the OpenAI Responses API (the dialect
OpenAI Codex requires), including
streaming SSE and function-tool-call re-mapping. For a local model this
is a plain pass-through to llama-server's own native /v1/responses
support, so a recent enough llama-server build is required for it to
work. For a provider without the route, the daemon
translates to and from /v1/chat/completions itself, and for a provider
that speaks the Anthropic Messages API from there to /v1/messages (see
wire formats).
The model in a /v1/chat/completions, /v1/completions or /v1/responses
reply (every streamed chunk too) is the one the client asked for, not the
path llama-server loaded the weights from.
A request bound for a provider loses llama-server's own fields
(repeat_penalty, top_k, min_p, chat_template_kwargs, ...), which
a strict provider would reject the whole request over; OpenAI's reasoning
models get max_completion_tokens and lose the sampling overrides they
refuse. previous_response_id is refused when /v1/responses has to be
bridged through chat completions: nothing is stored to resolve it against.
A local /v1/chat/completions with reasoning_effort also gets the
chat_template_kwargs Ollama's think would (none →
enable_thinking: false; a level → enable_thinking: true plus
reasoning_effort), so it works on llama-server builds that do not read
reasoning_effort themselves. The caller's own kwargs are kept.
On a llama-server model whose template thinks but takes no
reasoning_effort level (Qwen3, Qwen3.5, Gemma 4), minimal and low
also get a reasoning_budget_tokens of an eighth or a quarter of the
token limit (at most 1024 or 2048, also the budget without a limit) and
a reasoning_budget_message that closes it, so thinking cannot use up
max_tokens. A budget from the caller or from --reasoning-budget in
LLMMAN_LLAMA_ARGS, or a --chat-template[-file] there, means llmman
adds neither; a message from the caller or --reasoning-budget-message
replaces llmman's.
A streamed /v1/chat/completions or /v1/completions to an openai-wire
provider (Cohere aside) or a non-llama.cpp backend goes upstream with
stream_options.include_usage for llmman usage; a
client that did not ask gets the usage chunk stripped back out.
llama-server's timings need no asking.
/v1/audio/transcriptions is likewise a pass-through. The model needs
audio support (an --mmproj projector, supplied when the model image
carries one). Bodies up to 200 MiB are accepted. A llama-server is given
a transcription prompt if the request has none (not for LFM2 models,
which have their own).
/v1/systemone answers typed questions about a state with a decision model
(a GGUF with a <arch>.decision.type, such as Clef), with a probability for
each answer; llmman run MODEL --ask ... is its client. It is
llama-server's own route,
plus what Ollama's takes: images as bare base64 and keep_alive. The reply
names the model as the request did, and questions and answers keep the
order written.
Such a model generates no text and makes no embeddings, so chat, completion,
responses, messages and embedding requests get a 400, as in Ollama, without
loading it. /api/show reports it as decision.
It reads its whole prompt in one batch: LLMMAN_CONTEXT_LENGTH up to the
trained context, 8192 tokens when unset. A longer text is a 400. Bodies up
to 32 MiB are accepted.
/v1/messages implements the Anthropic Messages API (the dialect
Claude Code requires). For
a local model or a provider on the openai wire the daemon translates
it to a chat completion and the reply back: system-role turns fold into
one leading system message, tool_use/tool_result become
tool_calls/role: "tool", tools and tool_choice become functions
(names over 64 characters shortened and restored), thinking and
output_config.effort (Claude Code's /effort) become
reasoning_effort and llama-server's chat_template_kwargs,
output_format becomes response_format. Text,
reasoning (as thinking) and tool input stream back as they arrive,
with the real stop_reason and usage. Anthropic's server tools, cache
breakpoints and anthropic-beta headers have no chat-completion form
and are dropped; a tool_choice naming a server tool is a 400. A
provider on the anthropic wire gets the request relayed as sent (see
wire formats).
/llmman/... is llmman's own API, not a compatibility surface: no
upstream API has a notion of a models.dev provider
(see providers.md). /llmman/providers lists the ones
this daemon can route to, each with its API-key variable, whether the
daemon has that key, and how many models it serves;
/llmman/providers/{id} adds those models and what each costs in US
dollars per million tokens (absent, not zero, where models.dev publishes
no price), including cache_read/cache_write/reasoning where it publishes those.
?model=<id> filters nothing; a catalog that lacks it is re-fetched first
(at most every five minutes). llmman providers, list --provider, run --provider and
launch --provider are all clients of it, so the catalog is fetched and
cached in one process: the one that forwards the request upstream.
A provider defined in llmman.conf
appears alongside the catalog ones with key_optional: true, no
key_env unless the file names one, and — on /llmman/providers/{id}
— for the openai wire, whatever model ids its own GET /models
reports, unpriced.
PUT /llmman/providers/{id}/key with {"api_key": "..."} stores a
provider's key the way llmman config set providers.<id>.api_key does
(the per-user llmman.conf, chmod 600), and the daemon spends it from
the next request on, without a restart. DELETE removes that file's key;
one in /etc/llmman/llmman.conf or the provider's variable still
applies. Both re-read every llmman.conf's provider keys before they
answer (a DELETE with nothing to remove too), so a 200 means the
keys in force are what the files now say; only these routes reload,
and a key set with llmman config set or a hand edit waits for a
restart. If the files cannot be read (a hand edit that broke one), the
keys in force are kept and the answer is a 500 that points to the
daemon's log, which has the detail. Both answer {id, key_usable, key_env, env_override} and never the key; env_override: true means
the variable is set in the daemon's environment and wins over the
stored key. An id neither in the catalog
nor configured is a 404, a blank or multi-line key a 400. They are
refused (403) where the daemon would not spend its own provider key for
the caller: a daemon reachable beyond loopback without LLMMAN_API_KEYS,
or a page on another site (see authentication).
/llmman/node reports this node's memory and loaded/stored models; it
is what aggregation peers ask each other. See aggregation.md.
/llmman/search?q=<query> is llmman search over HTTP: the same Docker
Hub and Hugging Face rows in the same order, as {"models": [{name, pulls, likes, updated}]}, each name ready for /api/pull. limit
caps the rows per registry as --limit does (default 25, at most 64).
It leaves out the FIT column llmman search adds, which costs a
registry request per row; the web UI shows fit on a model's card.
/llmman/search/popular has the same shape and needs no query: Docker
Hub's most pulled models, then Hugging Face's most downloaded GGUF
text-generation repos.
/llmman/search/model?name=<a row's name> expands one row: every tag
pull can take for it with its size (variants, the one a tagless
pull takes marked default), plus the repo's pulls, likes, license,
tags and README (Hugging Face's README.md, Docker Hub's overview; the
first 64 KiB). For a GGUF repo on Hugging Face the variants are its
quantizations, each one that resolves to its own file the way pull
picks one. /llmman/search/avatar?name= redirects to the repo owner's
picture on Hugging Face or Gravatar, or is a 404 when it has none.
These ask the registries live; a registry that cannot be reached is a
502.
With LLMMAN_API_KEYS (or [auth] api_keys in llmman.conf; see
configuration.md) set, every request
must present one of the keys, in either spelling the compatible surfaces
already use:
Authorization: Bearer <key> # OpenAI clients
x-api-key: <key> # Anthropic clientsAnything else is a 401 with WWW-Authenticate: Bearer realm="llmman"
and the daemon's usual {"error": ...} body. Only GET/HEAD of /,
/ui/* and the web UI's own routes (an unrouted path, outside the API
prefixes, from a client that accepts HTML) are exempt, so the page can
load and ask its user for the key; it then sends the key on every call,
and, since a browser cannot set a header on a WebSocket, opens
/llmman/shell with the subprotocol llmman.bearer.<base64url(key)>.
The shell also needs its own token, keys or not
(webui.md), as the subprotocol
llmman.shell.<token>, which the daemon echoes back.
The llmman CLI sends LLMMAN_API_KEY, defaulting to the first
configured server key; llmman launch hands the same key to the
integration it starts.
A key that opens the daemon is not a provider key: the header it
arrived in is removed before routing, so it is never relayed upstream.
An authenticated caller is the operator, though, so the daemon spends
its own provider keys for it whatever LLMMAN_HOST it is bound to —
where an open daemon spends them only on a loopback bind. A caller that
wants to use its own provider key sends it in the other header
(llmman run --provider does this on its own).
Without keys the daemon is open, and only allowed to be so on loopback:
bound anywhere else it refuses to start unless LLMMAN_AUTH=off.
TLS is terminated by the daemon itself with LLMMAN_TLS_CERT and
LLMMAN_TLS_KEY, together with an https:// LLMMAN_HOST so clients
in the same environment connect the same way (the daemon refuses one
without the other). LLMMAN_TLS_CA trusts a private CA — for every
connection the process makes, providers included.
/metrics is a Prometheus scrape target, off by default because an
open daemon has no authentication; with keys configured it requires one
like every other route. LLMMAN_METRICS=1 llmman serve turns it on;
the seventeen metric families and how to read them are in
metrics.md.
Browser clients on localhost, 127.0.0.1, 0.0.0.0 and [::1] (any
scheme, any port) are always allowed. LLMMAN_ORIGINS adds more; see
configuration.md.