Skip to content

rw2: add stateless and stateless-st modes (rc.6) - #3147

Open
bwplotka wants to merge 4 commits into
mainfrom
bwplotka/rw2-rc6-stateless-st
Open

bwplotka wants to merge 4 commits into
mainfrom
bwplotka/rw2-rc6-stateless-st

Conversation

@bwplotka

@bwplotka bwplotka commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Alternative to #3129 and #3142.

Specify two progressive modes (stateless and stateless-st) for io.prometheus.write.v2.Request in Remote-Write 2.0-rc.6, following the Prometheus Dev Summit consensus and RFC 9413.

By default, when no ;mode= parameter is set on Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request, all field permutations permitted by the base specification may be sent. 2.0-rc.6 adds two optional, progressive modes where senders MUST guarantee:

  1. stateless mode (Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless):
    • Exemplars are attached to a TimeSeries that contains at least one sample or histogram, so no exemplar-only TimeSeries are sent.
    • Histograms use the Histogram message, so classic histograms are sent as NHCB instead of _bucket, _sum, and _count float series.
    • Metadata.type is set and not METRIC_TYPE_UNSPECIFIED. For series with an unknown metric type, such as untyped scraped metrics or recording rule outputs, senders MAY assume METRIC_TYPE_GAUGE.
  2. stateless-st mode (Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless-st):
    • All requirements of stateless mode.
    • start_timestamp is set (non-zero) on every sample and histogram, except stale markers, for METRIC_TYPE_COUNTER, METRIC_TYPE_HISTOGRAM, and METRIC_TYPE_SUMMARY _sum and _count.

Both modes are optional for senders and receivers and remain valid io.prometheus.write.v2.Request payloads on the wire.

Why this split?

  • stateless mode bundles everything needed for self-contained, typed TimeSeries messages: attached exemplars, structured histograms, and an explicit Metadata.type. Any sender can enforce stateless mode without tracking cross-scrape counter start timestamps, and receivers can validate histogram structure because Metadata.type is always set.
  • stateless-st isolates start_timestamp into a second progressive level because start_timestamp depends on upstream instrumentation exposing created timestamps or stateful sender-side synthesis, and adds per-sample wire overhead on cumulative series that not all receivers need.
  • Progressive modes (default < stateless < stateless-st) give 3 well-defined protocol states rather than $2^N$ capability combinations (RFC 9413), keep Content-Type matching to a single parameter value, and make stateless a candidate for a future 3.0 default while keeping stateless-st optional.

Draft of the implementation on Prometheus side: prometheus/prometheus#20098

[ENHANCEMENT] Remote-Write 2.0: Add optional `stateless` and `stateless-st` modes (`rc.6`).

Define two progressive modes for io.prometheus.write.v2.Request: stateless (requiring Exemplars attached to samples/histograms, Histogram messages for histograms, and Metadata.type with optional METRIC_TYPE_GAUGE fallback for unknown types) and stateless-st (extending stateless with mandatory start_timestamp on cumulative metric types).

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Comment thread docs/specs/prw/remote_write_spec_2_0.md
Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
…TYPE_COUNTER fallback

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Comment thread docs/specs/prw/remote_write_spec_2_0.md Outdated
Comment thread docs/specs/prw/remote_write_spec_2_0.md Outdated
@bwplotka

bwplotka commented Oct 21, 2026 •

Copy link
Copy Markdown
Member Author

TODO: address @kgeckhart comments from #3142 (comment) Done

…xemplars

Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
Comment thread docs/specs/prw/remote_write_spec_2_0.md Outdated
Comment thread docs/specs/prw/remote_write_spec_2_0.md Outdated
#### Unsupported Request Content

Receivers MUST return [415 HTTP Unsupported Media Type](https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type) status code if they don't support a given content type or encoding provided by Senders.
Receivers MUST return [415 HTTP Unsupported Media Type](https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type) status code if they don't support a given content type or encoding provided by Senders. When returning `415 HTTP Unsupported Media Type`, Receivers SHOULD include the `Accept` (listing supported `Content-Type` values, including supported `;proto=` and `;mode=` parameters) and `Accept-Encoding` (listing supported `Content-Encoding` values) response headers as defined in [RFC 7694](https://www.rfc-editor.org/rfc/rfc7694.html) so Senders can determine what the Receiver supports without guessing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: @kgeckhart I think this solves our discoverability of supported modes problem 👍🏽

For the message introduced in PRW 2.0, identified by `io.prometheus.write.v2.Request`:

* `Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request`
* `Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=none` (equivalent to omitting `;mode=`; see [Modes](#modes))

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressing #3142 (comment) - I think "none" is a good name

@bwplotka

bwplotka commented Oct 21, 2026 •

Copy link
Copy Markdown
Member Author

Should be ready to go @kgeckhart @krajorama

@kgeckhart do you plan to propose gzip (can help with that if you won't have time to do it) in rc.6?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant