[bwplotka]

Alternative to #3129.

Specify two progressive modes (stateless and stateless-typed) for io.prometheus.write.v2.Request in Remote-Write 2.0-rc.6, following the Prometheus Dev Summit consensus and RFC 9413.

By default, when no ;mode= parameter is set on Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request, all field permutations permitted by the base specification may be sent. 2.0-rc.6 adds two optional, progressive modes where senders MUST guarantee:

1. stateless mode (Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless):
   • Exemplars are attached to a TimeSeries that contains at least one sample or histogram, so no exemplar-only TimeSeries are sent.
   • Histograms use the Histogram message, so classic histograms are sent as NHCB instead of _bucket, _sum, and _count float series.
2. stateless-typed mode (Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless-typed):
   • All requirements of stateless mode.
   • Metadata.type is set and not METRIC_TYPE_UNSPECIFIED.
   • start_timestamp is set (non-zero) on every sample and histogram for cumulative and delta metric types (METRIC_TYPE_COUNTER, METRIC_TYPE_HISTOGRAM, and METRIC_TYPE_SUMMARY _sum and _count).

Both modes are optional for senders and receivers and remain valid io.prometheus.write.v2.Request payloads on the wire.

Why two progressive modes instead of one mode or separate capabilities?

• stateless rules, attaching exemplars to samples or histograms and converting classic histograms to NHCB, are sender-side formatting choices. Any sender such as Prometheus or Alloy can enforce them by configuration alone without dropping data, even when scraping legacy targets.
• stateless-typed rules, Metadata.type and start_timestamp, require every series to have a known metric type and counter start time. When scraping targets that expose untyped metrics or omit _created timestamps, a sender can only satisfy stateless-typed by synthesizing state, inferring types, or dropping non-conforming series.
• stateless-typed includes stateless because receivers that require explicit metric types and start timestamps also require structured histograms and attached exemplars, and validating start_timestamp already requires knowing Metadata.type.
• Progressive modes (default < stateless < stateless-typed) give 3 well-defined protocol states rather than $2^N$ capability combinations (RFC 9413), keep Content-Type matching to a single parameter value, and keep prometheus/compliance testing linear.

[ENHANCEMENT] Remote-Write 2.0: Add optional `stateless` and `stateless-typed` modes (`rc.6`).

[bwplotka]

This is to address here some discussions points of @krajorama

1. Receiver strict validation of spec and mode (IMO "SHOULD" not "MUST")

The existing spec was relaxed on this. Receiver are not responsible for validating the spec (e.g. ST on gauges), although they MAY. Somewhat exception are core elements like Sample, Histogram and Example (response headers are MUST).

Rationales for keeping MAY for both spec and mode validation:

a. Forward compatibility:

• Proto allows us to add new fields etc. Expecting receiver to validate would mean new fields would be a breaking change
• Similarly we wouldn't be able to add new modes as technically old receivers would need to add special validation per mode. Our goal is to ensure generic receiver implementation is possible - one that can support all modes similar to HTTP Media Type Profiles (RFC 6906):

A profile parameter on Content-Type constrains what the sender puts on the wire so profile-aware receivers can rely on it, while explicitly guaranteeing that any generic receiver of the base media type can accept the payload even if it ignores the profile's extra semantics.

b. Makes RW2 adoption on receive harder (e.g. my RW2 sink that only checks certain things is not "compliant").
c. Efficiency - burning CPU on things you don't care about?
d. Adoption: If client ever supports stateless - receiver might want to accept it and partially reject untyped metrics.

As a result, it's all SHOULD here.

2. Receiver accepting a mode means it will use all it's data (IMO that's explicitly NOT an intention)

We mentioned this concept that sender might want to know that e.g. if ST is sent and stateless-typed is used and receiver returns 200 code it means ST is actually used for querying (!).

This has some uses, but I think it's hard to rely on, plus all the problems above.

So we don't expect that - modes are to:

• Simplify communication between sender and receiver (e.g. simplify sender configuration, reject undesired modes early on receive, clearly document which mode is how expensive)
• Help with adoption of more advanced modes (e.g. letting sender know that some receivers will only support certain mode, monitoring mode adoption etc)

[bwplotka]

Proposed implementation of those modes in Prom:

• storage/remote: add stateless and stateless-typed modes to remote_write prometheus#20069
• storage/remote: coalesce exemplars onto matching v2 TimeSeries in batch prometheus#20070

[dashpole]

This split makes sense to me

[krajorama]

nit: if the type is not set, it's impossible to validate this, which questions the stateless name a little bit and the usefulness of this mode :(

[bwplotka]

Well, not if we specify a clear heuristic e.g. _bucket, _sum, _count UNTYPED can be rejected. But yes, this is probably the biggest downside of this solution.

[krajorama]

_sum and _count can be summary as well, so nope. This is what we ran into when migrating to native histograms internally, we couldn't tell from PromQL queries alone what the type was, so we had to cross reference with the source code all the time.

[kgeckhart]

I'm a bit naive on this front, are summaries something we still want to support to the point that this would be a blocker beyond updating the spec to say they get dropped too if they are untyped?

[krajorama]

note: I'm unsure about this combined with metadata. As we said, maybe sending ST with every sample isn't a good idea for bandwidth when it comes to cumulative metrics. But I'm unsure about the actual overhead.

[bwplotka]

Well, it's what some backends needs - true ST on cumulatives.

In Prometheus ecosystem I can imagine stateless + metric types (no ST) is the future (aka 3.0)

[krajorama]

Practically, when a client sends the new mode there's no feedback if it was actually verified. Unless of course it's flat out refused with HTTP status 415.

Which means that the user needs to ask the admin of the receiver whether the check was actually performed.

Consider returning X-Prometheus-Remote-Write-Mode-Checked: <stateless|stateless-typed> if a check was actually performed.

[bwplotka]

Practically, when a client sends the new mode there's no feedback if it was actually verified. Unless of course it's flat out refused with HTTP status 415.
Which means that the user needs to ask the admin of the receiver whether the check was actually performed.
Consider returning X-Prometheus-Remote-Write-Mode-Checked: <stateless|stateless-typed> if a check was actually performed.

I don't think we should add a response header for this:

1. ;mode= is a sender-side guarantee (similar to RFC 6906 profiles), not a request for the receiver to validate the sender. Checking that a sender implementation actually satisfies stateless or stateless-typed is the job of prometheus/compliance tests, not production receivers on every request.
2. In practice, receivers validate what their ingestion pipeline needs rather than linting every rule in the mode (for example, a receiver might require Metadata.type and start_timestamp and return 400 when missing, but not burn CPU checking if a float series name ends in _bucket). That makes a single Mode-Checked header hard to define accurately.
3. A sender has nothing actionable to do with that header at runtime on a 2xx response. If a receiver requires a mode, the user already gets immediate feedback via 415 when the mode is not sent and 400 when data cannot be ingested. If a generic receiver (like Prometheus TSDB) accepts all modes without needing the extra guarantees, paying per-response header overhead just to report that is unnecessary.

[kgeckhart]

Do we want to do anything about opening up compression options for rc6 or leave that for a different RC?

[kgeckhart]

WDYT about naming the default mode stateful? This way every mode has a named option and if we move the default at a later time there's still a named option for people to fall back on.

I think being able to explicitly configure the mode could also be helpful vs needing to know implicitly what no mode means.

[kgeckhart]

https://github.com/prometheus/docs/pull/3142/changes#diff-fb08fe6850c73166c5ad591d90f78aee4c90986316885ec04d0942ee871640b9R199 mentions a sender can retry when they get a 415 to try to find something that works. We could return the supported content types in the response so senders aren't having to iterate and guess. Doing this is probably enough to avoid needing an ahead of time content negotiation.