[mo4islona]

Closes #114.

hotblocks_active_queries counts a query task from admission to its end, so it mixes tasks that wait in the query pool's queue with tasks that run. A slower stream at a flat request rate can mean either, and nothing exported told the two apart.

What

QuerySlot::run takes a step (first opens a stream, next fills each later data pack) and records:

• hotblocks_query_task_queue_seconds{step}: spawn → the task starts on a worker
• hotblocks_query_task_run_seconds{step}: start → end of the task, panics included
• hotblocks_query_tasks_queued: tasks spawned and not yet started; active_queries - query_tasks_queued is the number of running tasks

Buckets are exponential_buckets(0.0001, 2.0, 18) (100 µs to ~13 s). No dataset or client label. The cost is three Instant::now() calls and two observations per task.

Reading the run time

The query code runs par_iter inside a task. A worker that waits for its parallel work can take another query task from the pool's queue and run it on the same stack, so a task's run time can include a whole other task. A long run time is not always a slow query.

Testing

tests/query_task_metrics.rs runs one query against the real binary and checks the /metrics delta: one first observation in each histogram, equal next counts, and the queued gauge back at 0.

Dashboard

Queue and run p50/p99 side by side:

histogram_quantile(0.99, sum by (le, step) (rate(hotblocks_query_task_queue_seconds_bucket[$__rate_interval])))
histogram_quantile(0.99, sum by (le, step) (rate(hotblocks_query_task_run_seconds_bucket[$__rate_interval])))

🤖 Generated with Claude Code