Skip to content

metrics: add per-model requests in flight and queued - #824

Merged
ericcurtin merged 1 commit into
mainfrom
feat/817-model-queue-gauges
Nov 16, 2026
Merged

ericcurtin merged 1 commit into
mainfrom
feat/817-model-queue-gauges

Conversation

@ericcurtin

@ericcurtin ericcurtin commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #817. Adds per-model llmman_requests_in_flight and llmman_requests_queued. Queued is an estimate and absent when the slot count is unknown (diffusion, --parallel in LLMMAN_LLAMA_ARGS, no --ctx-size).

Tested: unit tests.

@greptile-apps

greptile-apps Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

RetriggerView in Greptile

[Low impact] The change affects what operators can see in the existing metrics scrape.

Fix the queue calculation before merging; it can both hide waiting requests and report queues that do not exist.

Reviews (1) · Last reviewed commit: "metrics: add per-model requests in fligh..." · Reviewed by Greptile

Comment thread src/cmd/serve.rs Resolved
Comment thread src/cmd/serve.rs Resolved
Comment thread src/cmd/serve.rs Resolved
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Explain Change →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML; Path: .coderabbit.yaml
  • Review profile: QUIET
  • Plan: Advanced
  • Run ID: 239f1f21-bffc-4833-9031-8e17c246140f
Commits

Reviewing files that changed from the base of the PR and between 50f1169 and e56547c.

Files selected for processing: 7

  • docs/api.md
  • docs/metrics.md
  • src/cmd/serve.rs
  • src/cmd/serve/backend.rs
  • src/cmd/serve/test_support.rs
  • src/cmd/serve/tests.rs
  • src/metrics.rs
Walkthrough

The serve path now records the configured parallel-slot count for LlamaServer models. Metrics include a per-model gauge for in-flight requests and, when the slot count is known, a gauge for queued requests. Tests cover metric output for known and unknown queue counts. Documentation describes the new gauges and updates the metric-family count.

Severity of issue fixed: Medium

Merge Risk: Low · previous review 50f11

Models loaded through the diffusion path may report a misleading queued-request metric. Serving behavior is unaffected. Fix the slot recording before merge or accept the minor inaccuracy.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 1

src/cmd/serve.rs: 1

2759-2759: Minor · Functional Correctness · Quick win

Do not record slot capacity for diffusion loads.

Both ModelPath::Diffusion spawn branches set Engine::LlamaServer on the returned process. Neither branch passes llama_opts or num_parallel to its worker. Line 2759 still records num_parallel for these loads because it checks only the engine tag. When num_parallel is Some, /metrics then emits llmman_requests_queued from a capacity the worker never received. Limit the slot count to spawn paths that actually forward num_parallel, so the metric stays absent for diffusion loads.

Adds llmman_requests_in_flight and llmman_requests_queued (#817).
Queued is an estimate, absent when the slot count is unknown: diffusion,
--parallel in LLMMAN_LLAMA_ARGS, or no --ctx-size.
@ericcurtin
ericcurtin force-pushed the feat/817-model-queue-gauges branch from 50f1169 to e56547c Compare November 16, 2026 22:28
@ericcurtin
ericcurtin merged commit 3b65c0f into main Nov 16, 2026
15 of 16 checks passed
@ericcurtin
ericcurtin deleted the feat/817-model-queue-gauges branch November 16, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

metrics: expose in-flight and queued requests per model

1 participant