Skip to content

feat(metrics): add request lifecycle, queue-scoped stage, and batch shape metrics - #890

Open
roychying wants to merge 5 commits into
mainfrom
chenghan.ying/sqv2-dashboard-metrics
Open

roychying wants to merge 5 commits into
mainfrom
chenghan.ying/sqv2-dashboard-metrics

Conversation

@roychying

Copy link
Copy Markdown
Contributor

Why?

Operating a SubmitQueue deployment today means reading per-stage internals with no queue-level picture. The service emits per-delivery consumer metrics, but it cannot answer the questions an operator asks first:

  • How long does a land take? Nothing measures a request from acceptance to its terminal outcome.
  • How many requests finish, and how? There is no throughput metric broken down by landed / error / cancelled.
  • Which queue is affected? Every <stage>_controller.process.* counter in the orchestrator is untagged, so per-queue filtering only works for the consumer's own metrics.
  • Is storage actually failing? Every storage.*.finish histogram records a missing-record read as result=error, so store error rates are inflated by expected not-found lookups.
  • What shape is the work? Nothing records how large batches are, how many dependencies they serialize behind, or how many speculation paths a batch needed.

This PR adds those signals so a service dashboard can lead with land time and throughput, and filter every stage panel by queue.

What?

Five commits, each independently reviewable:

  1. platform/metrics: shared buckets and an explicit-result completion

    • LandLatencyBuckets (1m to 24h, densest between 5 and 60 minutes) for end-to-end request lifecycles.
    • CountBuckets (0 to 144, Fibonacci-spaced, with zero in its own bucket) for counts recorded with RecordValue.
    • Op.CompleteAs(result, tags...): like Complete, but with a caller-chosen result tag, for outcomes the error alone does not classify. platform/metrics stays unaware of domain error types.
  2. Storage: not-found reads are not errors

    • storage.CompleteMetricsOp(op, err) records ErrNotFound as result=not_found and defers to op.Complete otherwise.
    • All 33 gateway and orchestrator MySQL store operations use it.
  3. Orchestrator: every stage counter carries queue

    • All 238 metrics.NamedCounter calls in submitqueue/orchestrator/controller/** pass metrics.TagsFromContext(ctx).... The consumer already puts queue on the delivery context, and this is the pattern it uses for its own counters.
    • speculate.(*Controller).attributed now takes ctx so its failure counter is tagged too.
  4. Gateway: request lifecycle metrics from the log materializer

    • The materializer emits on the summary update that crosses each milestone:

      Metric Type Tags Emitted when
      request_lifecycle.accepted counter queue the summary first leaves accepting
      request_lifecycle.terminated counter queue, status, from_status the summary moves from a non-terminal to a terminal status
      request_lifecycle.end_to_end_latency histogram (LandLatencyBuckets) queue, status same as terminated; terminal log timestamp minus receipt time; skipped when the receipt time is unknown
    • Each request counts once because of a before/after comparison, not logWins. An orchestrator retry that hits AlreadyInTargetState re-publishes the terminal log with the same request version and a newer timestamp. logWins accepts it and the summary is rewritten, but the previous status is already terminal, so nothing is counted twice.

    • Every termination path publishes a terminal log, so all of them are covered without touching the orchestrator.

    • Wiring is backward compatible. It uses a variadic option, NewMaterializer(stores, WithMetricsScope(scope)), and defaults to tally.NoopScope, so existing NewMaterializer(stores) callers keep compiling. The reference gateway server passes its scope.

    • Behaviours worth knowing:

      • A client retry of Land gets a new request ID that the orchestrator rejects as a duplicate. It shows up as status=error with near-zero latency; from_status separates validation rejections from pipeline failures.
      • Land time is measured from gateway acceptance, so it includes the hop to the orchestrator and any terminal-publish retry delay.
  5. Orchestrator: batch and speculation shape

    • dependency_analysis_controller.process.batch_size and .batch_dependencies are recorded on the Creating→Created transition. That happens once per batch, and batches abandoned before promotion are never measured.
    • speculate_controller.process.paths is recorded when a batch's outcome is committed.
    • Batch size reads 1 until batching groups more than one request. It is included so the panel works unchanged once it does.

Test Plan

New table-driven unit tests:

  • platform/metrics: TestOp_CompleteAs (result, extra tags, result wins over a same-key tag); sorted-bucket checks for both new sets.
  • submitqueue/extension/storage: TestCompleteMetricsOp (success, wrapped not-found, caller-wrapped not-found, other error, version mismatch).
  • submitqueue/gateway/core/request: TestMaterializer_RecordsLifecycleMetrics covers:
    • accepted; a later status overtaking the accepted log
    • terminal with land time; a missing receipt time
    • a re-published terminal log with a newer timestamp (no second count)
    • an identical redelivery; cancelling treated as non-terminal
  • speculate: TestAttributed_CountsFailureWithContextQueueTag, plus a paths histogram assertion on a landing run.
  • dependencyanalysis: the batch_size / batch_dependencies assertions on promotion, and no histograms on a redelivery of an already-created batch.

Run locally:

make test
make lint check-gazelle check-tidy check-mocks

Issue

@roychying
roychying requested review from a team, behinddwalls and sbalabanov as code owners October 19, 2026 23:09
@roychying
roychying added this pull request to stack #892 October 19, 2026 23:13
@roychying roychying changed the title fear(metrics): add request lifecycle, queue-scoped stage, and batch shape metrics feat(metrics): add request lifecycle, queue-scoped stage, and batch shape metrics Oct 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant