Repository navigation
metrics: add per-model requests in flight and queued - #824
Conversation
|
[Low impact] The change affects what operators can see in the existing metrics scrape. Fix the queue calculation before merging; it can both hide waiting requests and report queues that do not exist. Reviews (1) · Last reviewed commit: "metrics: add per-model requests in fligh..." · Reviewed by Greptile |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 20 minutes. View limit details
Walkthrough
Severity of issue fixed: Medium Merge Risk:
|
There was a problem hiding this comment.
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 1
src/cmd/serve.rs: 1
2759-2759:Minor · Functional Correctness · Quick win Do not record slot capacity for diffusion loads.
Both
ModelPath::Diffusionspawn branches setEngine::LlamaServeron the returned process. Neither branch passesllama_optsornum_parallelto its worker. Line 2759 still recordsnum_parallelfor these loads because it checks only the engine tag. Whennum_parallelisSome,/metricsthen emitsllmman_requests_queuedfrom a capacity the worker never received. Limit the slot count to spawn paths that actually forwardnum_parallel, so the metric stays absent for diffusion loads.
Adds llmman_requests_in_flight and llmman_requests_queued (#817). Queued is an estimate, absent when the slot count is unknown: diffusion, --parallel in LLMMAN_LLAMA_ARGS, or no --ctx-size.
50f1169 to
e56547c
Compare
Fixes #817. Adds per-model
llmman_requests_in_flightandllmman_requests_queued. Queued is an estimate and absent when the slot count is unknown (diffusion,--parallelinLLMMAN_LLAMA_ARGS, no--ctx-size).Tested: unit tests.