Repository navigation
Add LLMMAN_EMBEDDING_CONTEXT_LENGTH; de-flake the vllm/sglang e2e - #822
Conversation
|
[Medium impact] The PR appears safe to merge; no blocking issues were found. Reviews (1) · Last reviewed commit: "Add LLMMAN_EMBEDDING_CONTEXT_LENGTH; de-..." · Reviewed by Greptile |
Walkthrough
Severity of issue fixed: Medium Merge Risk:
|
There was a problem hiding this comment.
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 2
docs/configuration.md: 1
426-426:Minor · Functional Correctness · Quick win Limit the
options.num_ctxoverride to the endpoints that forward it.The embedding handlers do not forward a request's
options.num_ctx.handle_embedandhandle_embeddingsboth callembed_via_backend, which passesLoadOptions::default()to the load. WhenLLMMAN_EMBEDDING_CONTEXT_LENGTHis set, an/api/embedor/api/embeddingsrequest that sendsoptions.num_ctxuses the configured embedding context instead. The documented override does not apply.Qualify the statement to the endpoints that forward
options.num_ctx. Alternatively, pass the request options intoembed_via_backendso the embedding endpoints honor the override.Suggested documentation fix
-| `LLMMAN_EMBEDDING_CONTEXT_LENGTH` | Context size for embedding models, in place of `LLMMAN_CONTEXT_LENGTH`. An embedding model reads its whole input in one batch as large as its context, so at the default it costs memory in proportion to its trained context (3.6 GiB for a 0.6B model trained to 32768); this lets one daemon serve a large-context chat model and a small embedder. Capped to the model's trained context, and `0` means that context. A request's `options.num_ctx` still wins. Unset, embedding models follow `LLMMAN_CONTEXT_LENGTH`. | +| `LLMMAN_EMBEDDING_CONTEXT_LENGTH` | Context size for embedding models, in place of `LLMMAN_CONTEXT_LENGTH`. An embedding model reads its whole input in one batch as large as its context, so at the default it costs memory in proportion to its trained context (3.6 GiB for a 0.6B model trained to 32768); this lets one daemon serve a large-context chat model and a small embedder. Capped to the model's trained context, and `0` means that context. On `/api/chat` and `/api/generate`, a request's `options.num_ctx` still wins; `/api/embed` and `/api/embeddings` ignore it. Unset, embedding models follow `LLMMAN_CONTEXT_LENGTH`. |tests/launch_e2e.rs: 1
3204-3206:Minor · Security & Privacy · Quick win Enumerate interface addresses instead of relying on a default route.
non_loopback_ipv4()finds an address by connecting a UDP socket to192.0.2.1:9. That connect needs a route to the destination. On a host with a private interface and no default route, the connect fails, so the helper returnsNone. The test then returns early, and the wildcard-bind check does not run.The skip comment says such a host "has no address another machine could reach the engine on." That is not true for a private interface. A peer on the same network can reach it. An engine bound to
0.0.0.0on that host would pass unnoticed.The helper also turns every
bind,connect, andlocal_addrerror intoNone. A socket error therefore produces the same skip as a host with no non-loopback address. Enumerate the interfaces instead, for example withgetifaddrsthrough the existinglibcdependency. Return a non-loopback IPv4 address when one exists. Skip only when enumeration succeeds and finds none. Treat enumeration errors as test failures.
An embedding model loads with its trained context and a batch as large, so a 0.6B model costs 3.6 GiB at 32768, and LLMMAN_CONTEXT_LENGTH is daemon-wide (#820). LLMMAN_EMBEDDING_CONTEXT_LENGTH sets it for embedding models only: capped to the trained context, 0 meaning that context. The /v1/models n_ctx prediction reads the same setting. main's e2e job failed in the vllm and sglang tests for two reasons: - The aarch64 runner ran out of memory and was killed: SGLang's CPU engine sized a 4-8 GB KV pool for one 1024-token request. Cap it with --max-total-tokens, and log MemAvailable around each engine test. - The loopback probe found its non-loopback address through the default route, and panicked on a runner without one. List the interfaces instead, and skip only when there is no such address.
56c9c52 to
2701e36
Compare
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the embedding-context setting and related CI changes.
No open issues 🎉
What changed in this PR
Adds an embedding-only context setting to address #820, allowing smaller embedding allocations without reducing chat-model context. It also targets known vLLM/SGLang end-to-end test failures.
Changes:
- Adds
LLMMAN_EMBEDDING_CONTEXT_LENGTHwith trained-context capping and request-level override precedence. - Updates unloaded-model context predictions, test cases, and documentation.
- Caps SGLang’s Linux CI token pool, adds memory diagnostics, and skips the external reachability check when no non-loopback IPv4 address is available.
| File | Description |
|---|---|
tests/launch_e2e.rs |
Adds memory logging and handles missing non-loopback IPv4 addresses. |
src/cmd/serve/tests.rs |
Extends context prediction and load-parity tests with embedding settings. |
src/cmd/serve/test_support.rs |
Initializes the embedding context setting in test fixtures. |
src/cmd/serve/ollama.rs |
Uses shared context selection for unloaded-model metadata. |
src/cmd/serve/config.rs |
Parses the new embedding context environment variable. |
src/cmd/serve.rs |
Wires embedding context selection into loading and CLI help. |
docs/configuration.md |
Documents the setting’s behavior and memory implications. |
docs/api.md |
Clarifies context override precedence and metadata predictions. |
.github/workflows/ci.yml |
Limits SGLang’s Linux CI token pool to reduce memory pressure. |
Review effort: Balanced
There was a problem hiding this comment.
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 1
src/cmd/serve.rs: 1
335-335:Minor · Functional Correctness · Heavy lift Apply
LLMMAN_EMBEDDING_CONTEXT_LENGTHto safetensors embedding models, or document that it is GGUF-only.The override applies only when
ctx_settingreceivesembedding=true. That flag comes from GGUF metadata alone. A safetensors load therefore always gets the daemonLLMMAN_CONTEXT_LENGTHand its explicit flag. WhenLLMMAN_CONTEXT_LENGTHis set, the vLLM or SGLang launch can receive that larger value asmax_model_len, not the embedding value.This predates the PR, so it is a scope gap rather than a regression. The help text and the general row in
docs/configuration.mddescribe the variable for embedding models without naming a backend. Either make the safetensors embedding path honor the override, or state in those docs that the variable applies only to GGUF embedding models. The fix for the first option needs a safetensors embedding signal that the inspected code does not provide.
Fixes #820.
LLMMAN_EMBEDDING_CONTEXT_LENGTH: an embedding model's context, in place ofLLMMAN_CONTEXT_LENGTH(capped to the trained context,0= trained)./v1/modelsn_ctxfollows it; tests and docs updated.--max-total-tokens 4096, plusMemAvailablelogging around each engine test.getifaddrs, and skips only when there is no non-loopback address.Not run locally (the Go shim's modules were unreachable):
cargo check,clippy -D warningsandfmtare clean. The e2e job is skipped on PRs, so the memory fix is untested until main.