diff --git a/docs/specs/prw/remote_write_spec_2_0.md b/docs/specs/prw/remote_write_spec_2_0.md index 454aaf6e8..68110b916 100644 --- a/docs/specs/prw/remote_write_spec_2_0.md +++ b/docs/specs/prw/remote_write_spec_2_0.md @@ -4,9 +4,9 @@ nav_title: "2.0" sort_rank: 2 --- -* Version: 2.0-rc.5 +* Version: 2.0-rc.6 * Status: **Experimental** -* Date: May 2024 +* Date: October 2026 The Remote-Write specification, in general, is intended to document the standard for how Prometheus and Prometheus Remote-Write compatible senders send data to Prometheus or Prometheus Remote-Write compatible receivers. @@ -44,6 +44,7 @@ In this document, the following definitions are followed: * a `Wire Format` is the format of the data as it travels on the wire (i.e. in a network). In the case of Remote-Write, this is always the compressed binary protobuf format. * a `Sender` is something that sends Remote-Write data. * a `Receiver` is something that receives (writes) Remote-Write data. The meaning of `Written` is up to the Receiver e.g. usually it means storing received data in a database, but also just validating, splitting or enhancing it. +* a `Mode` is a set of additional rules on top of `Protocol` that tightens optional sender and receiver semantics into a deterministic contract. * `Written` refers to data the `Receiver` has received and is accepting. Whether or not it has ingested this data to persistent storage, written it to a WAL, etc. is up to the `Receiver`. The only distinction is that the `Receiver` has accepted this data rather than explicitly rejecting it with an error response. * a `Sample` is a triplet of (start timestamp, timestamp, value). * a `Histogram` is a triplet of (start timestamp, timestamp, [histogram value](https://github.com/prometheus/docs/blob/b9657b5f5b264b81add39f6db2f1df36faf03efe/content/docs/concepts/native_histograms.md)). @@ -96,9 +97,10 @@ Content encoding request header MUST follow [the RFC 9110](https://www.rfc-edito ``` Content-Type: application/x-protobuf Content-Type: application/x-protobuf;proto= +Content-Type: application/x-protobuf;proto=;mode= ``` -Content type request header MUST follow [the RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-type). Senders MUST use `application/x-protobuf` as the only media type. Senders MAY add `;proto=` parameter to the header's value to indicate the fully qualified name of the Protobuf Message that was used, from the two mentioned above. As a result, Senders MUST send any of the three supported header values: +Content type request header MUST follow [the RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-type). Senders MUST use `application/x-protobuf` as the only media type. Senders MAY add `;proto=` parameter to the header's value to indicate the fully qualified name of the Protobuf Message that was used, from the two mentioned above. Senders MAY also add `;mode=` parameter to indicate the [mode](#modes) (additional semantic rules) for the Protobuf Message, if any is defined for it. As a result, Senders MUST send one of the following supported header values: For the deprecated message introduced in PRW 1.0, identified by `prometheus.WriteRequest`: @@ -108,10 +110,12 @@ For the deprecated message introduced in PRW 1.0, identified by `prometheus.Writ For the message introduced in PRW 2.0, identified by `io.prometheus.write.v2.Request`: * `Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request` +* `Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless` (see [Modes](#modes)) +* `Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless-typed` (see [Modes](#modes)) -When talking to 1.x Receivers, Senders SHOULD use `Content-Type: application/x-protobuf` for backward compatibility. Otherwise, Senders SHOULD use `Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request`. More Protobuf Messages might come in 2.x or beyond. +When talking to 1.x Receivers, Senders SHOULD use `Content-Type: application/x-protobuf` for backward compatibility. Otherwise, Senders SHOULD use `Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request` (optionally with `;mode=stateless` or `;mode=stateless-typed`). More Protobuf Messages or modes might come in 2.x or beyond. -Receivers MUST use the content type header to identify the Protobuf Message schema to use. Accidental wrong schema choices may result in non-deterministic behaviour (e.g. corruptions). +Receivers MUST use the content type header to identify the Protobuf Message schema to use, and MAY use it to identify the [mode](#modes). Accidental wrong schema choices may result in non-deterministic behaviour (e.g. corruptions). > NOTE: Thanks to reserved fields in [`io.prometheus.write.v2.Request`](#protobuf-message), Receiver accidental use of wrong schema with `prometheus.WriteRequest` will result in empty message. This is generally for convenience to avoid surprising errors, but don't rely on it -- future Protobuf Messages might not have this feature. @@ -190,7 +194,7 @@ Senders SHOULD expect [400 HTTP Bad Request](https://www.rfc-editor.org/rfc/rfc9 #### Invalid Samples -Receivers MAY NOT support certain metric types or samples (e.g. a Receiver might reject sample without metadata type specified or without start timestamp, while another Receiver might accept such sample.). It’s up to the Receiver what sample is invalid. Receivers MUST return a [400 HTTP Bad Request](https://www.rfc-editor.org/rfc/rfc9110.html#name-400-bad-request) status code for write requests that contain any invalid samples unless the [partial retriable write](#retries-on-partial-writes) occurs. +Receivers SHOULD reject data (such as samples, histograms, exemplars, or entire time series) that violates this specification or the negotiated [mode](#modes). Receivers MAY reject data that violates receiver-specific limits and gaps (such as not supporting native histograms, or label count or length limits). Receivers MUST return a [400 HTTP Bad Request](https://www.rfc-editor.org/rfc/rfc9110.html#name-400-bad-request) status code for write requests that contain any invalid data (whether rejecting the entire request or performing a [partial write](#partial-write)) unless the [partial retriable write](#retries-on-partial-writes) occurs. Senders MUST NOT retry on a 4xx HTTP status codes (other than [429](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429)), which MUST be used by Receivers to indicate that the write operation will never be able to succeed and should not be retried. Senders MAY retry on the 415 HTTP status code with a different content type or encoding to see if the Receiver supports it. @@ -229,9 +233,6 @@ The 2.x protocol is breaking compatibility with 1.x by introducing a new, mandat The `io.prometheus.write.v2.Request` references the new Protobuf Message that's meant to replace and deprecate the Remote-Write 1.0's `prometheus.WriteRequest` message. - The full schema and source of the truth is in Prometheus repository in [`prompb/io/prometheus/write/v2/types.proto`](https://github.com/prometheus/prometheus/blob/main/prompb/io/prometheus/write/v2/types.proto). The `gogo` dependency and options CAN be ignored ([will be removed eventually](https://github.com/prometheus/prometheus/issues/11908)). They are not part of the specification as they don't impact the serialized format. The simplified version of the new `io.prometheus.write.v2.Request` is presented below. @@ -372,13 +373,13 @@ For every `TimeSeries` message: -* At least one element in `samples`, in `histograms`, or in `exemplars` MUST be provided. For example, a `TimeSeries` with only `metadata` and `labels_refs` is not valid. +* At least one element in `samples`, in `histograms`, or in `exemplars` MUST be provided (in [`stateless` and `stateless-typed` modes](#modes), at least one element in `samples` or in `histograms` MUST be provided). For example, a `TimeSeries` with only `metadata` and `labels_refs` is not valid. * A `TimeSeries` MUST NOT include both `samples` and `histograms`. For series which (rarely) would mix float and histogram samples, a separate `TimeSeries` message MUST be used. -* `metadata` sub-fields SHOULD be provided. Receivers MAY reject series with unspecified `Metadata.type`. +* `metadata` sub-fields SHOULD be provided (in [`stateless-typed` mode](#stateless-typed-mode), `Metadata.type` MUST be specified). Receivers MAY reject series with unspecified `Metadata.type`. * Exemplars SHOULD be provided if they exist for a series. The following subsections define some schema elements in detail. @@ -421,9 +422,9 @@ Receivers also MAY impose limits on the number and length of labels, but this is -Senders MUST send `samples` (or `histograms`) for any given `TimeSeries` in timestamp order. Senders MAY send multiple requests for different series in parallel. +Senders MUST send `samples` (or `histograms`) for any given `TimeSeries` in timestamp order. Senders MAY send multiple requests for different series in parallel. In [`stateless` and `stateless-typed` modes](#modes), histogram metrics MUST be sent using `histograms` (e.g. Native Histograms with Custom Buckets) rather than classic histogram float `samples`. -Sample's or Histogram's `start_timestamp` SHOULD be provided for types that follow counter semantics (e.g. counters and counter histograms). Receivers MAY reject those series without `start_timestamp` being set. Given optionality, the 0 value MUST be treated by receivers as unset value. To represent the unlikely 0 unix timestamp in milliseconds, "1" or "-1" value MUST be used. +Sample's or Histogram's `start_timestamp` SHOULD be provided for types that follow counter semantics (e.g. counters and counter histograms; in [`stateless-typed` mode](#stateless-typed-mode), `start_timestamp` MUST be set for these types, except on stale markers). Receivers MAY reject those series without `start_timestamp` being set. Given optionality, the 0 value MUST be treated by receivers as unset value. To represent the unlikely 0 unix timestamp in milliseconds, "1" or "-1" value MUST be used. * MAY contain labels e.g. referencing trace or request ID. If the exemplar references a trace it SHOULD use the `trace_id` label name, as a best practice. * MUST contain a timestamp. While exemplar timestamps are optional in Prometheus/Open Metrics exposition formats, the assumption is that a timestamp is assigned at scrape time in the same way a timestamp is assigned to the scrape sample. Receivers require exemplar timestamps to reliably handle (e.g. deduplicate) incoming exemplars. -* MAY be sent in a `TimeSeries` that carries neither samples nor histograms. In that form the exemplars are associated with the series identified by `labels_refs`, rather than with a particular sample or histogram. +* MAY be sent in a `TimeSeries` that carries neither samples nor histograms (except in [`stateless` and `stateless-typed` modes](#modes), where exemplars MUST only be sent on a `TimeSeries` that also contains at least one element in `samples` or in `histograms`). Exemplars are always associated with the series identified by `labels_refs` rather than with a specific element in `samples` or `histograms`, even when multiple samples or histograms are present in the `TimeSeries`. + +#### Modes + +By default (when no `;mode=` parameter is set), `io.prometheus.write.v2.Request` keeps several structural and metadata rules optional (`SHOULD` or `MAY`) so Senders can forward any scraped metric without data loss or conversion overhead. However, some common stateless or strongly typed Receivers cannot ingest all the data e.g. detached exemplars, classic histogram series or untyped metrics. + +To address this, `io.prometheus.write.v2.Request` defines two progressive modes, `stateless` and `stateless-typed`, that tighten optional rules into `MUST` requirements on Senders while keeping every payload a valid `io.prometheus.write.v2.Request` message. See the [FAQ](#faq) for the rationale. + +Senders SHOULD support `stateless` and `stateless-typed` modes. Senders MAY operate in either mode behind configuration or by default. + +Receivers MAY accept any valid `io.prometheus.write.v2.Request` regardless of mode, require at least `stateless` mode, or require `stateless-typed` mode only. If rejected, [415 HTTP Unsupported Media Type](https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type) MUST be used. + +##### `stateless` Mode + +When operating in `stateless` mode, Senders MUST: + +* Use content type of `Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless`. +* Ensure all of the following for every `TimeSeries` in a `Request`: + * `Exemplar`s MUST only be sent on a `TimeSeries` that also contains at least one element in `samples` or in `histograms`. Exemplar-only `TimeSeries` messages MUST NOT be sent. + * Histograms MUST use the `Histogram` message (e.g. using Native Histograms with Custom Buckets instead of classic histogram `_bucket`, `_sum`, and `_count` float `Sample` series). + +##### `stateless-typed` Mode + +When operating in `stateless-typed` mode, Senders MUST: + +* Use content type of `Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless-typed`. +* Satisfy all `TimeSeries` requirements of [`stateless` mode](#stateless-mode). +* Ensure all of the following for every `TimeSeries` in a `Request`: + * `Metadata.type` MUST be specified and MUST NOT be `METRIC_TYPE_UNSPECIFIED` (`0`). + * `start_timestamp` MUST be set (non-zero) on every `Sample` and `Histogram` (except stale markers) for metric types `METRIC_TYPE_COUNTER`, `METRIC_TYPE_HISTOGRAM`, as well as `METRIC_TYPE_SUMMARY` `_sum` and `_count` `Samples`. ## Out of Scope @@ -475,7 +505,7 @@ This section contains speculative plans that are not considered part of protocol * **Transactionality** There is still no transactionality defined for 2.0 specification, mostly because it makes a scalable Sender implementation difficult. Prometheus Sender aims at being "transactional" - i.e. to never expose a partially scraped target to a query. We intend to do the same with Remote-Write -- for instance, in the future we would like to "align" Remote-Write with scrapes, perhaps such that all the samples, metadata and exemplars for a single scrape are sent in a single Remote-Write request. - However, Remote-Write 2.0 specification solves an important transactionality problem for [the classic histogram buckets](https://docs.google.com/document/d/1mpcSWH1B82q-BtJza-eJ8xMLlKt6EJ9oFGH325vtY1Q/edit#heading=h.ueg7q07wymku). This is done thanks to the native histograms supporting custom bucket-ing possible with the `io.prometheus.write.v2.Request` wire format. Senders might translate all classic histograms to native histograms this way, but it's out of this specification to mandate this. However, for this reason, Receivers MAY ignore certain metric types (e.g. classic histograms). + However, Remote-Write 2.0 specification solves an important transactionality problem for [the classic histogram buckets](https://docs.google.com/document/d/1mpcSWH1B82q-BtJza-eJ8xMLlKt6EJ9oFGH325vtY1Q/edit#heading=h.ueg7q07wymku). This is done thanks to the native histograms supporting custom bucketing (NHCB) in the `io.prometheus.write.v2.Request` wire format, which [`stateless` and `stateless-typed` modes](#modes) mandate instead of classic histograms. When no mode is specified, Senders might still translate classic histograms to native histograms this way, and Receivers MAY ignore certain metric types (e.g. classic histograms). * **Alternative wire formats**. The OpenTelemetry community has shown the validity of Apache Arrow (and potentially other columnar formats) for over-wire data transfer with their OTLP protocol. We would like to do experiments to confirm the compatibility of a similar format with Prometheus’ data model and include benchmarks of any resource usage changes. We would potentially maintain both a protobuf and columnar format long term for compatibility reasons and use our content negotiation to add different Protobuf Messages for this purpose. @@ -494,6 +524,21 @@ If you use persistent HTTP/1.1 connections, they are pretty close to streaming. **Why do we send samples in order?** The in-order constraint comes from the encoding we use for time series data in Prometheus, the implementation of which is optimized for append-only workloads. However, this requirement is also shared across many other databases and vendors in the ecosystem. In fact, [Prometheus with OOO feature enabled](https://youtu.be/qYsycK3nTSQ?t=1321), allows out-of-order writes, but with the performance penalty, thus reserved for rare events. To sum up, Receivers may support out-of-order write, though it is not permitted by the specification. In the future e.g. 2.x spec versions, we could extend content type to negotiate the out-of-order writes, if needed. +**What's the purpose of `stateless` and `stateless-typed` modes?** +By default, when no mode is specified, Remote-Write 2.0 allows any valid permutation of fields in `io.prometheus.write.v2.Request` to preserve compatibility and minimal overhead for existing Prometheus TSDB pipelines. + +However, stateless or strongly-typed receivers often require structured histograms, attached exemplars, metric types, or start timestamps to ingest metrics without stateful caches or mid-stream `400 HTTP Bad Request` rejections. [Modes](#modes) have two goals: + +* Simplify communication between senders and receivers, such as configuring required sender features as a single setting, rejecting unsupported modes early on the receiver with `415 HTTP Unsupported Media Type`, and documenting the cost and requirements of each mode. +* Help the ecosystem adopt stricter semantics, such as letting receivers advertise which mode they require and letting operators monitor mode adoption across senders. + +We defined two progressive modes, `stateless` and `stateless-typed`, rather than a single mode or fine-grained capability flags: + +* `stateless` mode rules, attaching exemplars to samples or histograms and converting classic histograms to NHCB, are sender-side formatting choices that any sender can enforce without dropping data, even when scraping legacy targets. `stateless-typed` mode rules, `Metadata.type` and `start_timestamp`, depend on upstream instrumentation exposing types and start timestamps, or sender-side synthesis or filtering of untyped metrics. +* `stateless-typed` includes `stateless` because receivers that require strict types and start timestamps also require structured histograms and attached exemplars, and `start_timestamp` validation itself requires knowing `Metadata.type`. Following [RFC 9413](https://www.rfc-editor.org/rfc/rfc9413.html), three progressive levels (`default`, `stateless`, `stateless-typed`) avoid `2^N` protocol permutations, keep `Content-Type` matching to a single parameter value, and keep compliance testing linear. + +In future specification versions, we might propose making `stateless` or parts of `stateless-typed` mandatory, depending on ecosystem feedback and adoption. + **How can we parallelise requests with the in-order constraint?** Samples must be in-order _for a given series_. However, even if a Receiver does not support out-of-order write, the Remote-Write requests can be sent in parallel as long as they are for different series. Prometheus shards the samples by their labels into separate queues, and then writes happen sequentially in each queue. This guarantees samples for the same series are delivered in order, but samples for different series are sent in parallel - and potentially "out of order" between different series.