Skip to content

Release message streams at retention, and keep PHP streams at takeover - #801

Merged
rmcdaniel merged 11 commits into
mainfrom
rust-325-stream-retention
Oct 27, 2026
Merged

rmcdaniel merged 11 commits into
mainfrom
rust-325-stream-retention

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Part of #325. Pre-RC native Rust (tier C).

1. Release message streams at retention as PHP does (d7abee65b)

Native's retention pass held any run that had workflow_messages rows or whose instance had inbound message streams, with message_stream_cleanup_unavailable. Published PHP's MessageStreamRetentionCleanup was never ported. So a namespace that used message streams kept those runs' details forever.

prune_run now plans and applies the stream cleanup in the run's own prune transaction, under the transition lock. The plan follows PHP's planForRun, read before the run's summary goes:

  • No inbound streams on the instance: nothing to do.
  • The instance's current run is terminal and no other run summary remains: every item of the instance's streams is deleted, then the streams. Reported as message_stream_items_deleted and message_streams_deleted.
  • Otherwise: this applies to each stream whose cursor_checkpoint_run_id is a retained later run, meaning another run summary of the instance or its current run.
    • The items this run consumed lose their payloads: those at or before the stream's cursor that are not released yet. payload_blob becomes '' and payload_released_at is set.
    • The item rows stay as identity tombstones. Reported as message_stream_items_compacted.
    • cleanup_blocked_* markers naming this run are cleared.
  • The run's workflow_messages rows go with the run's other details, as before.

Not ported: PHP's markBlocked, which marks a stream when external-payload cleanup blocks a prune. Native never deletes external objects. A run holding an external reference stays held as external_payload_cleanup_unavailable, and native refuses external references as message input.

Tests, on SQLite, PostgreSQL, MySQL and MariaDB:

  • retention_releases_consumed_messages_and_removes_the_last_runs_streams: pruning a continued run compacts the two messages it consumed and keeps the third. Pruning the last run deletes the stream and its three items.
  • retention_keeps_consumed_messages_no_retained_checkpoint_covers: without a retained checkpoint, nothing is compacted and the stream stays.

2. Takeover keeps PHP message and workflow streams (4e9012c58)

Takeover refused any source with rows in workflow_messages, either kind of stream, or their items. Stream rows now carry over unchanged.

  • Each closed or errored durable stream gets its dw_stream_reclaims deadline. That is 2.11.3's reclaim_at, or for earlier layouts closed_at plus retention_seconds (3600 unless set). Reported as stream_reclaims_recorded. MySQL and MariaDB do it in journal step 10.
  • Inspection reports stream_reference_inspection. A stream row holding an external payload reference refuses with upgrade_source_external_payloads_unsupported.
  • Only workflow_messages stays refused, now as upgrade_source_workflow_messages_unsupported. Only embedded PHP workflows write it, and native does not run them.

Tests:

  • takeover_keeps_php_streams_and_records_their_reclaim_deadlines (SQLite).
  • The 2.11.3 takeover test now copies a recorded deadline.
  • The refusal list covers a durable stream item holding an external reference.
  • Takeover qualification passes on PostgreSQL (7), MySQL (8) and MariaDB (8), including a MySQL takeover killed after step 9.
  • The full SQLite suite passes: lib 338, execution 560.
  • Native records all 65 corpus fixtures.

3. Register DW_RUST_MAINTENANCE_LOOP as a native setting (01ec49bf3)

#797 added the opt-in, but the config audit did not know it. So setting it logged "Unknown DW_ variables set in environment". The local native image smoke test showed this. The audit test now includes it in the shipped environment.

4. Docs

docs/rust-server-port.md "Current state" is refreshed for October 27 and after #797 (8b2a19df1, 3da8f54fd), including the recommendation that the release candidate keeps refusing external payload takeover. Checkpoint: docs/rust-server-port-checkpoints/2026-10-27-message-stream-retention.md.

5. Fixes from an independent review (ec23a5ad7)

The review of commits 1 and 2 found no serious defects. Fixed:

  • A resumed MySQL takeover picks step 10's deadline source by layout, as SQLite and PostgreSQL do. A 2.11.3 takeover killed after reclaim_at was dropped had already copied every deadline, so it no longer adds closed_at + retention rows for streams PHP had emptied.
  • The current run counts as retained only when its run row exists, as in PHP.
  • A run whose instance has an inbound stream item holding an external reference stays held. Nothing writes one today; the hold keeps a later change from stranding stored objects.
  • The payload scan's comment no longer says stream item rows are refused.

The retention test now puts the cursor exactly on the pruned run's last consumed message (<= matters), and points the instance at a missing current run. Each new assertion was checked by mutation: it fails against the code it guards.

Local results:

  • SQLite: lib 338, execution 560.
  • MySQL and MariaDB: takeover qualification, including a takeover killed at each step and resumed.

6. One macro for the stream reference count (63de18803)

SonarCloud failed the first run's quality gate on duplicated new lines (4.5% against 3%). STREAM_REFERENCES repeated one query for each backend. A macro now builds it from the column expressions, and the SQL each backend runs is unchanged.

7. 8 GB for the Rust test builds (f9610d429)

The first full run lost execution-and-codec (native-tests): rustc was killed with signal 9 compiling the library tests in the 6 GB container. A cold cargo test --all-targets --no-run with two jobs peaks at 6.8 GB (cgroup memory.peak, measured locally on this tree), and main had passed only narrowly. The three containers that compile test targets now get 8 GB; the binary-only build keeps 6 GB. Both organization policies pass.

Still refused at takeover

workflow_messages (embedded PHP workflows only) and external payload references.

🤖 Generated with Claude Code

rmcdaniel and others added 11 commits October 27, 2026 00:42
Native's retention pass held every run whose instance had inbound message
streams, because PHP's MessageStreamRetentionCleanup was never ported. A
namespace that used message streams therefore kept those runs forever.

The prune now plans and applies the stream cleanup in the run's own
transaction. The instance's last retained run, once terminal, takes its
streams and items with it. An earlier run's consumed items lose their
payloads once a retained later run's cursor checkpoint covers them. Tested
on SQLite, PostgreSQL, MySQL and MariaDB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Native serves inbound message streams and durable workflow streams from
PHP's own tables, and now releases them at retention as PHP does, so
takeover carries their rows over instead of refusing the source.

Each closed or errored durable stream gets the reclaim deadline native's
retention pass reads: PHP 2.11.3's reclaim_at, or closed_at plus its
retention for earlier releases, as 2.11.3's migration backfills it.
Inspection now counts stream rows that hold an external payload reference,
which takeover still refuses. Only workflow_messages, which native does
not deliver yet, stays refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The opt-in for native's maintenance loop (#797) was missing from the
config audit's native-only names, so setting it logged "Unknown DW_
variables set in environment". The native image smoke test showed it.
The shipped-environment test now sets it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#797 merged; #799, this PR and the native image are next. External payload
takeover stays refused: PHP keeps references in about 30 locations in
their URI form, so it is its own project, and the release-candidate scope
question now includes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An independent review of the stream retention and stream takeover
commits found no serious defects. This fixes what it did find:
- A resumed MySQL takeover now picks step 10's deadline source by layout,
  as SQLite and PostgreSQL do, not by whether reclaim_at still exists. A
  2.11.3 takeover killed after that column's drop had copied every
  deadline already, so it adopts nothing more instead of adding
  closed_at + retention rows for streams PHP had emptied.
- The current run counts as retained only when its run row exists, as in
  PHP.
- A run whose instance has an inbound stream item holding an external
  reference stays held (external_payload_cleanup_unavailable). Nothing
  writes one today; this keeps a later change from stranding objects.
- The payload scan's comment no longer says stream item rows are refused.

The retention test now puts the cursor exactly on the pruned run's last
consumed message, and points the instance at a missing current run. Each
new assertion fails against the code it guards (checked by mutation).
SQLite: lib 338, execution 560.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SonarCloud failed this PR's quality gate on duplication (4.5% of new
lines against a 3% limit): STREAM_REFERENCES repeated the same 6-line
query for PostgreSQL, SQLite and MySQL, differing only in the ::text
casts. One macro now builds it from the payload and metadata column
expressions; SQLite and MySQL share one constant. The SQL each backend
runs is unchanged. SQLite lib: 338 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR's first full run lost execution-and-codec (native-tests): rustc
was killed with signal 9 compiling the library tests inside the 6 GB
container. A cold cargo test --all-targets --no-run with two jobs peaks
at 6.8 GB (cgroup memory.peak, measured locally on this tree); main
passed only narrowly. The three containers that compile test targets
(the all-targets run and the MySQL and PostgreSQL native test runs) now
get 8 GB. The binary-only build keeps 6 GB. A hosted runner has 16 GB,
which leaves room for the database container beside them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit 5b9eaae Oct 27, 2026
58 checks passed
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged through the merge queue at 2026-10-27T04:16:45Z, 21 minutes after enqueue (03:55:29Z, pinned to head f9610d4292168a4dc4079395a0533a2b09fb2afd).

@rmcdaniel
rmcdaniel deleted the rust-325-stream-retention branch October 27, 2026 04:19
rmcdaniel added a commit that referenced this pull request Oct 27, 2026
The checkpoint names the verified 2.11.4 artifacts, the Core 2.14
expectation changes native now follows, the poll_transport position and
the Workflow #931 boundary, with what was checked. The Rust port's
current state lists #799 and #801 as merged, #803 and this move as
next, and 2.11.4 as the compared reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant