Skip to content

Extend opaque correlation to single-key writes - #90

Draft
mrattle wants to merge 1 commit into
Shopify:mainfrom
mrattle:mrattle/write-opaque-correlation
Draft

mrattle wants to merge 1 commit into
Shopify:mainfrom
mrattle:mrattle/write-opaque-correlation

Conversation

@mrattle

@mrattle mrattle commented Oct 23, 2026

Copy link
Copy Markdown

Summary

Extend the opt-in correlate_with_opaques response validation from single-key reads (#79, #84) to single-key writes. A correlated write sends a generated O token and checks the echoed token before reading anything else from the reply. Gating is the same as for reads: the client-level option, opaque_correlation_request_only, and per-request correlate_with_opaques.

Intended behaviour:

  • Covered: set/set_cas, add, replace/replace_cas, append, prepend, incr, decr, and the writes made by fetch (add) and cas/cas! (set/add).
  • Writes always use a generated token. They take no meta_flags, so there is no caller opaque to prefer.
  • A mismatched or missing token on an accepted reply code (HD/NS/EX/NF for ms, VA/NF/NS/EX for ma) returns the operation's existing failure value (false, or nil for incr/decr). The connection is closed after the request, without retrying or down-marking. A rejected VA leaves its counter line unread. Other protocol errors are unchanged.
  • The outcome of a rejected write is unknown: the server may still have applied it, so false here does not mean "not stored".
  • Unchanged: deletes (delete, delete_cas, delete_multi), writes inside quiet/multi blocks (memcached suppresses their success replies, so there is nothing to check), and multi-key writes (set_multi).
  • In the OpenTelemetry middleware, write spans now get request_opaque and the correlation_* failure attributes, kept out of the tags and attributes shared with other middleware, matching read spans.

Testing

  • Integration tests against a fake server cover the client/request gating matrix for writes, rejection of stale and missing tokens on each write path, and connection recycling without down-marking.
  • Against real memcached: every write reply code (HD, NS, NF, EX, VA) echoes the token and keeps the connection open, and deletes, quiet blocks, and multi-key writes stay uncorrelated.
  • Unit tests for the request formatter, the write reply parsers (rejected bodies are not consumed), and trace scoping on storage spans.
  • Full suite passes locally on Ruby 4.0.7 / memcached 1.6.45, excluding test_rack_session.rb, which does not load on Ruby 4.0 with or without this change (rack 2.2 requires the removed cgi/cookie).

Send a generated O token on non-quiet ms/ma requests (set, add, replace, append, prepend, incr, decr, and the writes made by fetch/cas/cas!) and verify the echoed token before reading anything else from the reply. Rejected replies return false (nil for incr/decr) and close the connection after the request without retrying or down-marking, with the same log and trace diagnostics as reads. Write spans now carry the request opaque and correlation failure attributes.

Deletes, quiet/multi blocks, and multi-key writes are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant