Skip to content

Add LLMMAN_EMBEDDING_CONTEXT_LENGTH; de-flake the vllm/sglang e2e - #822

Merged
ericcurtin merged 1 commit into
mainfrom
fix/embedding-context-length-and-e2e-flakes
Nov 16, 2026
Merged

ericcurtin merged 1 commit into
mainfrom
fix/embedding-context-length-and-e2e-flakes

Conversation

@ericcurtin

@ericcurtin ericcurtin commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #820.

  • LLMMAN_EMBEDDING_CONTEXT_LENGTH: an embedding model's context, in place of LLMMAN_CONTEXT_LENGTH (capped to the trained context, 0 = trained). /v1/models n_ctx follows it; tests and docs updated.
  • Runner OOM (runs 43822452189, 43646085397): SGLang's CPU engine sized a 4-8 GB KV pool on the 16 GB aarch64 runner. Now --max-total-tokens 4096, plus MemAvailable logging around each engine test.
  • Loopback probe (run 43639331493): panicked on a runner with no default route. It now lists interfaces with getifaddrs, and skips only when there is no non-loopback address.

Not run locally (the Go shim's modules were unreachable): cargo check, clippy -D warnings and fmt are clean. The e2e job is skipped on PRs, so the memory fix is untested until main.

@greptile-apps

greptile-apps Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

RetriggerView in Greptile

[Medium impact] The PR appears safe to merge; no blocking issues were found.

Reviews (1) · Last reviewed commit: "Add LLMMAN_EMBEDDING_CONTEXT_LENGTH; de-..." · Reviewed by Greptile

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

The serve command now reads LLMMAN_EMBEDDING_CONTEXT_LENGTH and uses it for embedding model loads when set. Tests and documentation cover its precedence, trained-context cap, and zero-value behavior. The Linux engine test environment adds a SGLang token limit. Engine test runs log available memory, and the reachability check skips when no non-loopback IPv4 address is available.

Severity of issue fixed: Medium

Merge Risk: Low · up to 2701e

The new embedding context setting works for GGUF embedding models but not for safetensors embedders served through vLLM or SGLang, although the docs describe it generally. Either honor it there or document the limit. Otherwise the change is low risk.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 2

docs/configuration.md: 1

426-426: Minor · Functional Correctness · Quick win

Limit the options.num_ctx override to the endpoints that forward it.

The embedding handlers do not forward a request's options.num_ctx. handle_embed and handle_embeddings both call embed_via_backend, which passes LoadOptions::default() to the load. When LLMMAN_EMBEDDING_CONTEXT_LENGTH is set, an /api/embed or /api/embeddings request that sends options.num_ctx uses the configured embedding context instead. The documented override does not apply.

Qualify the statement to the endpoints that forward options.num_ctx. Alternatively, pass the request options into embed_via_backend so the embedding endpoints honor the override.

Suggested documentation fix

-| `LLMMAN_EMBEDDING_CONTEXT_LENGTH` | Context size for embedding models, in place of `LLMMAN_CONTEXT_LENGTH`. An embedding model reads its whole input in one batch as large as its context, so at the default it costs memory in proportion to its trained context (3.6 GiB for a 0.6B model trained to 32768); this lets one daemon serve a large-context chat model and a small embedder. Capped to the model's trained context, and `0` means that context. A request's `options.num_ctx` still wins. Unset, embedding models follow `LLMMAN_CONTEXT_LENGTH`. |
+| `LLMMAN_EMBEDDING_CONTEXT_LENGTH` | Context size for embedding models, in place of `LLMMAN_CONTEXT_LENGTH`. An embedding model reads its whole input in one batch as large as its context, so at the default it costs memory in proportion to its trained context (3.6 GiB for a 0.6B model trained to 32768); this lets one daemon serve a large-context chat model and a small embedder. Capped to the model's trained context, and `0` means that context. On `/api/chat` and `/api/generate`, a request's `options.num_ctx` still wins; `/api/embed` and `/api/embeddings` ignore it. Unset, embedding models follow `LLMMAN_CONTEXT_LENGTH`. |
tests/launch_e2e.rs: 1

3204-3206: Minor · Security & Privacy · Quick win

Enumerate interface addresses instead of relying on a default route.

non_loopback_ipv4() finds an address by connecting a UDP socket to 192.0.2.1:9. That connect needs a route to the destination. On a host with a private interface and no default route, the connect fails, so the helper returns None. The test then returns early, and the wildcard-bind check does not run.

The skip comment says such a host "has no address another machine could reach the engine on." That is not true for a private interface. A peer on the same network can reach it. An engine bound to 0.0.0.0 on that host would pass unnoticed.

The helper also turns every bind, connect, and local_addr error into None. A socket error therefore produces the same skip as a host with no non-loopback address. Enumerate the interfaces instead, for example with getifaddrs through the existing libc dependency. Return a non-loopback IPv4 address when one exists. Skip only when enumeration succeeds and finds none. Treat enumeration errors as test failures.

An embedding model loads with its trained context and a batch as large,
so a 0.6B model costs 3.6 GiB at 32768, and LLMMAN_CONTEXT_LENGTH is
daemon-wide (#820). LLMMAN_EMBEDDING_CONTEXT_LENGTH sets it for embedding
models only: capped to the trained context, 0 meaning that context. The
/v1/models n_ctx prediction reads the same setting.

main's e2e job failed in the vllm and sglang tests for two reasons:
- The aarch64 runner ran out of memory and was killed: SGLang's CPU
  engine sized a 4-8 GB KV pool for one 1024-token request. Cap it with
  --max-total-tokens, and log MemAvailable around each engine test.
- The loopback probe found its non-loopback address through the default
  route, and panicked on a runner without one. List the interfaces
  instead, and skip only when there is no such address.
@ericcurtin
ericcurtin force-pushed the fix/embedding-context-length-and-e2e-flakes branch from 56c9c52 to 2701e36 Compare November 16, 2026 13:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the embedding-context setting and related CI changes.

No open issues 🎉

What changed in this PR

Adds an embedding-only context setting to address #820, allowing smaller embedding allocations without reducing chat-model context. It also targets known vLLM/SGLang end-to-end test failures.

Changes:

  • Adds LLMMAN_EMBEDDING_CONTEXT_LENGTH with trained-context capping and request-level override precedence.
  • Updates unloaded-model context predictions, test cases, and documentation.
  • Caps SGLang’s Linux CI token pool, adds memory diagnostics, and skips the external reachability check when no non-loopback IPv4 address is available.
File Description
tests/​launch_e2e.rs Adds memory logging and handles missing non-loopback IPv4 addresses.
src/​cmd/​serve/​tests.rs Extends context prediction and load-parity tests with embedding settings.
src/​cmd/​serve/​test_support.rs Initializes the embedding context setting in test fixtures.
src/​cmd/​serve/​ollama.rs Uses shared context selection for unloaded-model metadata.
src/​cmd/​serve/​config.rs Parses the new embedding context environment variable.
src/​cmd/​serve.rs Wires embedding context selection into loading and CLI help.
docs/​configuration.md Documents the setting’s behavior and memory implications.
docs/​api.md Clarifies context override precedence and metadata predictions.
.github/​workflows/​ci.yml Limits SGLang’s Linux CI token pool to reduce memory pressure.

Review effort: Balanced

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 1

src/cmd/serve.rs: 1

335-335: Minor · Functional Correctness · Heavy lift

Apply LLMMAN_EMBEDDING_CONTEXT_LENGTH to safetensors embedding models, or document that it is GGUF-only.

The override applies only when ctx_setting receives embedding=true. That flag comes from GGUF metadata alone. A safetensors load therefore always gets the daemon LLMMAN_CONTEXT_LENGTH and its explicit flag. When LLMMAN_CONTEXT_LENGTH is set, the vLLM or SGLang launch can receive that larger value as max_model_len, not the embedding value.

This predates the PR, so it is a scope gap rather than a regression. The help text and the general row in docs/configuration.md describe the variable for embedding models without naming a backend. Either make the safetensors embedding path honor the override, or state in those docs that the variable applies only to GGUF embedding models. The fix for the first option needs a safetensors embedding signal that the inspected code does not provide.

@ericcurtin
ericcurtin merged commit 69b3d90 into main Nov 16, 2026
16 checks passed
@ericcurtin
ericcurtin deleted the fix/embedding-context-length-and-e2e-flakes branch November 16, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embedding models run with batch equal to a 32768 context, 3.6 GiB for a 0.6B model

2 participants