Repository navigation
Conversation
…osen operation results
…ad of result=error
…l outcome, and end-to-end land time
…ed paths per batch as histograms
roychying
requested review from
a team,
behinddwalls and
sbalabanov
as code owners
October 19, 2026 23:09
roychying
added this pull request to stack #892
October 19, 2026 23:13
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
Operating a SubmitQueue deployment today means reading per-stage internals with no queue-level picture. The service emits per-delivery consumer metrics, but it cannot answer the questions an operator asks first:
<stage>_controller.process.*counter in the orchestrator is untagged, so per-queue filtering only works for the consumer's own metrics.storage.*.finishhistogram records a missing-record read asresult=error, so store error rates are inflated by expected not-found lookups.This PR adds those signals so a service dashboard can lead with land time and throughput, and filter every stage panel by queue.
What?
Five commits, each independently reviewable:
platform/metrics: shared buckets and an explicit-result completionLandLatencyBuckets(1m to 24h, densest between 5 and 60 minutes) for end-to-end request lifecycles.CountBuckets(0 to 144, Fibonacci-spaced, with zero in its own bucket) for counts recorded withRecordValue.Op.CompleteAs(result, tags...): likeComplete, but with a caller-chosenresulttag, for outcomes the error alone does not classify.platform/metricsstays unaware of domain error types.Storage: not-found reads are not errors
storage.CompleteMetricsOp(op, err)recordsErrNotFoundasresult=not_foundand defers toop.Completeotherwise.Orchestrator: every stage counter carries
queuemetrics.NamedCountercalls insubmitqueue/orchestrator/controller/**passmetrics.TagsFromContext(ctx).... The consumer already putsqueueon the delivery context, and this is the pattern it uses for its own counters.speculate.(*Controller).attributednow takesctxso its failure counter is tagged too.Gateway: request lifecycle metrics from the log materializer
The materializer emits on the summary update that crosses each milestone:
request_lifecycle.acceptedqueueacceptingrequest_lifecycle.terminatedqueue,status,from_statusrequest_lifecycle.end_to_end_latencyLandLatencyBuckets)queue,statusterminated; terminal log timestamp minus receipt time; skipped when the receipt time is unknownEach request counts once because of a before/after comparison, not
logWins. An orchestrator retry that hitsAlreadyInTargetStatere-publishes the terminal log with the same request version and a newer timestamp.logWinsaccepts it and the summary is rewritten, but the previous status is already terminal, so nothing is counted twice.Every termination path publishes a terminal log, so all of them are covered without touching the orchestrator.
Wiring is backward compatible. It uses a variadic option,
NewMaterializer(stores, WithMetricsScope(scope)), and defaults totally.NoopScope, so existingNewMaterializer(stores)callers keep compiling. The reference gateway server passes its scope.Behaviours worth knowing:
Landgets a new request ID that the orchestrator rejects as a duplicate. It shows up asstatus=errorwith near-zero latency;from_statusseparates validation rejections from pipeline failures.Orchestrator: batch and speculation shape
dependency_analysis_controller.process.batch_sizeand.batch_dependenciesare recorded on the Creating→Created transition. That happens once per batch, and batches abandoned before promotion are never measured.speculate_controller.process.pathsis recorded when a batch's outcome is committed.Test Plan
New table-driven unit tests:
platform/metrics:TestOp_CompleteAs(result, extra tags, result wins over a same-key tag); sorted-bucket checks for both new sets.submitqueue/extension/storage:TestCompleteMetricsOp(success, wrapped not-found, caller-wrapped not-found, other error, version mismatch).submitqueue/gateway/core/request:TestMaterializer_RecordsLifecycleMetricscovers:cancellingtreated as non-terminalspeculate:TestAttributed_CountsFailureWithContextQueueTag, plus apathshistogram assertion on a landing run.dependencyanalysis: thebatch_size/batch_dependenciesassertions on promotion, and no histograms on a redelivery of an already-created batch.Run locally:
Issue