Merge pull request #801 from durable-workflow/rust-325-stream-retention

Release message streams at retention, and keep PHP streams at takeover

Author: rmcdaniel

.github/workflows/server-parity.yml

@@ -523,8 +523,8 @@ jobs:
             -e DB_USERNAME=root -e DB_PASSWORD=parity-disposable-password \
             -e CACHE_STORE=file -e QUEUE_CONNECTION=database -e DW_AUTH_TOKEN=parity-test-token \
             "$layout" artisan server:bootstrap --force > parity-evidence/published-bootstrap.txt
-          # Compiling the library tests with two jobs peaks near 4.2 GB; a 4 GB limit OOM-killed rustc (exit 101).
-          docker run --rm --user="$(id -u):$(id -g)" --network "$RESOURCE_SCOPE" --cpus=2 --memory=6g --memory-swap=6g \
+          # Compiling the test targets with two jobs peaked at 6.8 GB cold (cgroup memory.peak, October 27); a 6 GB limit OOM-killed rustc (signal 9, PR #801).
+          docker run --rm --user="$(id -u):$(id -g)" --network "$RESOURCE_SCOPE" --cpus=2 --memory=8g --memory-swap=8g \
             -e CARGO_HOME=/cargo -e CARGO_TARGET_DIR=/target -e CARGO_BUILD_JOBS=2 -e DW_PARITY_PART -e DW_EXECUTION_SHARDS \
             -e DW_TEST_MYSQL_URL=mysql://root:parity-disposable-password@parity-mysql:3306/php_ref \
             -e DW_EXECUTION_MYSQL_URL=mysql://root:parity-disposable-password@parity-mysql:3306/php_ref \
@@ -1183,8 +1183,8 @@ jobs:
             -e DB_USERNAME=parity -e DB_PASSWORD=parity-disposable-password \
             -e CACHE_STORE=file -e QUEUE_CONNECTION=database -e DW_AUTH_TOKEN=parity-test-token \
             "$layout" artisan server:bootstrap --force > parity-evidence/published-bootstrap.txt
-          # Compiling the library tests with two jobs peaks near 4.2 GB; a 4 GB limit OOM-killed rustc (exit 101).
-          docker run --rm --user="$(id -u):$(id -g)" --network "$RESOURCE_SCOPE" --cpus=2 --memory=6g --memory-swap=6g \
+          # Compiling the test targets with two jobs peaked at 6.8 GB cold (cgroup memory.peak, October 27); a 6 GB limit OOM-killed rustc (signal 9, PR #801).
+          docker run --rm --user="$(id -u):$(id -g)" --network "$RESOURCE_SCOPE" --cpus=2 --memory=8g --memory-swap=8g \
             -e CARGO_HOME=/cargo -e CARGO_TARGET_DIR=/target -e CARGO_BUILD_JOBS=2 -e DW_PARITY_PART \
             -e DW_TEST_POSTGRES_URL=postgres://parity:parity-disposable-password@parity-pg:5432/php_ref \
             -e DW_EXECUTION_POSTGRES_URL=postgres://parity:parity-disposable-password@parity-pg:5432/php_ref \
@@ -1862,8 +1862,8 @@ jobs:
           }
           trap rust_test_failures ERR
           mkdir -p cargo-home rust-target parity-evidence
-          # Compiling the library tests with two jobs peaks near 4.2 GB; a 4 GB limit OOM-killed rustc (exit 101).
-          docker run --rm --user="$(id -u):$(id -g)" --cpus=2 --memory=6g --memory-swap=6g \
+          # Compiling the test targets with two jobs peaked at 6.8 GB cold (cgroup memory.peak, October 27); a 6 GB limit OOM-killed rustc (signal 9, PR #801).
+          docker run --rm --user="$(id -u):$(id -g)" --cpus=2 --memory=8g --memory-swap=8g \
             -e CARGO_HOME=/cargo -e CARGO_TARGET_DIR=/target -e CARGO_BUILD_JOBS=2 \
             -e DW_UPGRADE_TEST_WRITABLE=/evidence -e DW_UPGRADE_TEST_READONLY=/source/parity-evidence \
             -v "$PWD:/source:ro" -v "$PWD/cargo-home:/cargo" -v "$PWD/rust-target:/target" \

docs/rust-server-port-checkpoints/2026-10-27-message-stream-retention.md

@@ -0,0 +1,76 @@
+# October 27, message-stream retention
+
+- **Before.** Native's retention pass held any run that had
+  `workflow_messages` rows or whose instance had inbound message streams, with
+  `message_stream_cleanup_unavailable`. Published PHP's
+  `MessageStreamRetentionCleanup` was never ported. So a namespace that used
+  message streams kept those runs' details forever, and takeover refused any
+  PHP database with stream rows.
+- **Change.** `prune_run` plans and applies the stream cleanup in the run's
+  own prune transaction, under the transition lock. The plan follows PHP's
+  `planForRun`, read before the run's summary goes:
+  - **No inbound streams on the instance:** nothing to do.
+  - **The instance's current run is terminal and no other run summary
+    remains:** every item of the instance's streams is deleted, then the
+    streams. Reported as `message_stream_items_deleted` and
+    `message_streams_deleted`.
+  - **Otherwise:** on each stream whose `cursor_checkpoint_run_id` is a
+    retained later run (another run summary of the instance, or its current
+    run), the items this run consumed lose their payloads. That covers items
+    at or before the stream's cursor that are not released yet: `payload_blob`
+    becomes `''` and `payload_released_at` is set. The item rows stay as
+    identity tombstones. Reported as `message_stream_items_compacted`.
+    `cleanup_blocked_*` markers naming this run are cleared.
+  - The run's `workflow_messages` rows go with the run's other details
+    (`messages_deleted`), as in PHP. Before, such a run was held.
+  - The current run counts as retained only when its run row exists, as in
+    PHP.
+  - A run stays held with `external_payload_cleanup_unavailable` when an
+    inbound item on its instance holds an external reference. Nothing writes
+    one today; the hold keeps a later change from stranding stored objects.
+- **Not ported.** PHP's `markBlocked`, which marks a stream when
+  external-payload cleanup blocks a prune. Native never deletes external
+  objects: a run holding an external reference stays held as
+  `external_payload_cleanup_unavailable`, and native refuses external
+  references as message input.
+- **Tests** on SQLite, PostgreSQL, MySQL and MariaDB:
+  - `retention_releases_consumed_messages_and_removes_the_last_runs_streams`:
+    pruning a continued run compacts the two messages it consumed and keeps the
+    third. Pruning the last run deletes the stream and its three items.
+  - `retention_keeps_consumed_messages_no_retained_checkpoint_covers`: without
+    a retained checkpoint, nothing is compacted and the stream stays.
+
+## Takeover keeps message and workflow streams
+
+- **Before.** Takeover refused any source with rows in
+  `workflow_messages`, either kind of stream, or their items.
+- **Change.** Stream rows carry over unchanged.
+  - Each closed or errored durable stream gets its `dw_stream_reclaims`
+    deadline: 2.11.3's `reclaim_at`, or `closed_at` plus `retention_seconds`
+    (3600 unless set) for earlier layouts. Reported as
+    `stream_reclaims_recorded`; MySQL and MariaDB do it in journal step 10.
+  - Inspection reports `stream_reference_inspection`. A stream row holding an
+    external payload reference refuses with
+    `upgrade_source_external_payloads_unsupported`.
+  - Only `workflow_messages` stays refused, now as
+    `upgrade_source_workflow_messages_unsupported`.
+- **Tests.**
+  - `takeover_keeps_php_streams_and_records_their_reclaim_deadlines` (SQLite).
+  - The 2.11.3 takeover test now copies a recorded deadline.
+  - The refusal list covers a durable stream item holding an external
+    reference.
+  - Takeover qualification passes on PostgreSQL, MySQL and MariaDB,
+    including a MySQL takeover killed after step 9.
+
+## Review fixes before the PR
+
+An independent review found no serious defects. Fixed:
+- A resumed MySQL takeover decides step 10's deadline source by layout, not
+  by whether `reclaim_at` still exists. A 2.11.3 takeover killed after that
+  column's drop had already copied every deadline, so it adopts nothing more
+  instead of adding `closed_at + retention` rows for streams PHP had emptied.
+- The current run's missing row and the inbound-item hold above.
+- The payload scan's comment no longer says stream item rows are refused.
+- The retention test now puts the cursor exactly on the pruned run's last
+  consumed message (so `<=` matters) and points the instance at a missing
+  current run.

docs/rust-server-port.md

@@ -14,92 +14,60 @@ qualify a later tree.
 
 ## Current state
 
-Updated October 24, 2026.
-
-- **Merged October 24.** Train
-  [#729](https://github.com/durable-workflow/server/pull/729) (merge
-  `61219609b`):
-  - MySQL statement closes are sent with the next command
-    ([#717](https://github.com/durable-workflow/server/pull/717)), ending the
-    MySQL-family stalls of [#714](https://github.com/durable-workflow/server/issues/714);
-  - policy maps that are not JSON objects are refused, like PHP #705, and a
-    namespace quota map that is not UTF-8 is refused under either name
-    ([#719](https://github.com/durable-workflow/server/pull/719));
-  - worker history shows consumed signal sequences, hydrating only each page's
-    own signals, plus full task snapshots
-    ([#710](https://github.com/durable-workflow/server/pull/710));
-  - a newer push cancels a superseded Shared Server fixtures run
-    ([#724](https://github.com/durable-workflow/server/pull/724)).
-- **Merged October 20-23.** These are summarized in the checkpoint history:
-  - backup-first takeover on every database;
-  - multi-node kill and restart;
-  - ownership guards, search attributes, namespace update, history export,
-    retention and activity timeouts;
-  - the October 21-23 trains;
-  - the move to the published 2.11.1 reference;
-  - the GHSA-6q48-6c7w-9gw2 fixes.
-
-  Each train's PR holds its evidence.
-- **In review or qualifying, in merge order.**
-  1. CI speed and the merge queue:
-     - [#741](https://github.com/durable-workflow/server/pull/741) runs the
-       SDK direction profiles in parallel lanes, about 14 minutes off every
-       database job, and fixes a timestamp-comparison test flake;
-     - [#742](https://github.com/durable-workflow/server/pull/742) adds the
-       merge-queue prerequisites (`merge_group` triggers, one fixtures
-       result, and fixture cells skipped when a change cannot affect them).
-  2. This train: operator metrics read across PHP's redispatch steps
-     ([#708](https://github.com/durable-workflow/server/pull/708)),
-     query-authority clients that wait for the acknowledgement
-     ([#720](https://github.com/durable-workflow/server/pull/720)), and the
-     machine-checked route list below.
-  3. Condition waits without a key, task snapshots, the SQLite patch cohort and
-     message streams (#713, #718, #722, #709).
-  4. Update validation with PHP's poll request ID contract
-     ([#712](https://github.com/durable-workflow/server/pull/712)), then the
-     held-poll registration fence
-     ([#739](https://github.com/durable-workflow/server/pull/739), #734) and
-     worker sessions
-     ([#740](https://github.com/durable-workflow/server/pull/740), #736),
-     stacked in that order.
-  5. Activity cancellation acknowledgement, and pip, Composer and Cargo trust
-     in CI ([#673](https://github.com/durable-workflow/server/pull/673)).
+Updated October 27, 2026.
+
+- **Merged October 26-27, through the merge queue.**
+  - [#790](https://github.com/durable-workflow/server/pull/790): prepared
+    local-activity refusals.
+  - [#793](https://github.com/durable-workflow/server/pull/793): Nexus catalog
+    administration, service-call execute, show and cancel, shared admission
+    limits, and catalog call reads.
+  - [#795](https://github.com/durable-workflow/server/pull/795): takeover of
+    published 2.11.1 databases (fresh and upgraded from 2.9.x), keeping search
+    attribute definitions, worker sessions and service calls, plus the repair
+    pass's service-call deadline sweep.
+  - [#797](https://github.com/durable-workflow/server/pull/797): native's own
+    maintenance repair loop (opt-in, `DW_RUST_MAINTENANCE_LOOP=1`); takeover
+    keeps runtime credentials and update validation tasks; takeover of
+    published 2.11.3, fresh and upgraded from 2.9.x.
+- **In the queue or next, in order.**
+  1. [#799](https://github.com/durable-workflow/server/pull/799): the fixture
+     adapters' Cargo cache, saved only by pushes to main.
+  2. Message-stream retention (PHP's `MessageStreamRetentionCleanup`), and
+     takeover that keeps message and workflow streams.
+  3. The native release-candidate image (`Dockerfile.native`), its release
+     workflow and the takeover runbook's image commands.
+  4. The parity reference moves to published 2.11.3. Native already matches it
+     on 62 of 65 fixtures. The move waits on two published 2.11.3 defects
+     General owns: the storage diagnostic (#789) and cross-caller idempotent
+     replay (Workflow #931).
+- **Still refused at takeover:** workflow-to-workflow messages
+  (`workflow_messages`, written only by embedded PHP workflows) and external
+  payload references. PHP keeps references in about 30 table and column
+  locations in their URI form, so taking them over is its own project; the
+  recommendation is that the release candidate keeps refusing them.
 - **Shared Server fixtures.** Native, embedded and published PHP 2.11.1 are
   compared on SQLite, MySQL 8.0, MariaDB 10.11 and PostgreSQL 17. Where the
   published reference itself misbehaves, an explicit boundary separates it
-  from native, and the PHP owner has an issue for each case:
-  - `operator-metrics` across PHP's redispatch window (#708);
-  - the published-SDK patch cohort's SQLite poll refusal
-    ([#721](https://github.com/durable-workflow/server/issues/721), #722);
-  - update validation under SQLite write locks
-    ([#726](https://github.com/durable-workflow/server/issues/726), fixed in
-    PHP by #730);
-  - `worker-sdk-persistent-pressure`'s shutdown window
-    ([#728](https://github.com/durable-workflow/server/issues/728));
-  - a retried activity's cooperative cancellation
-    ([Workflow #923](https://github.com/durable-workflow/workflow/issues/923));
-  - a reacquired worker session keeping its old failure reason
-    ([#737](https://github.com/durable-workflow/server/issues/737)).
+  from native, and the PHP owner has an issue for each case. Recent ones:
+  - #796, `service-calls` under full-corpus SQLite load;
+  - #794, workflow-bound Nexus handlers.
 
   No expectation is edited to make native pass.
 - **Parity surface.** Native answers 174 of the 175 API route methods in
-  published 2.11.1's route table, with message streams, durable workflow
-  stream writes, update validation, activity cancellation acknowledgement,
-  worker sessions, workflow-task cancellation scopes, service catalog
-  administration, Nexus service-call execution, show and cancel for the
-  handler kinds that stay accepted, caller-indexed Nexus operation reads,
-  bridge adapter webhooks, and embedded v2 and Waterline v1 workflow import.
-  Eight of those are the prepared local-activity routes, which it answers
-  with PHP's refusal for a bridge without the prepared role. An execution
-  test sends each one through the native router, and the ones still
-  answering `501 rust_capability_not_implemented` must match
+  published 2.11.1's route table. Eight of those are the prepared
+  local-activity routes, which it answers with PHP's refusal for a bridge
+  without the prepared role. Nexus workflow-bound handler kinds answer 501
+  until #794 settles their contract. An execution test sends each route
+  through the native router, and the ones still answering
+  `501 rust_capability_not_implemented` must match
   [`unserved-php-routes.v2.11.1.txt`](../rust/tests/fixtures/unserved-php-routes.v2.11.1.txt).
-  Unlike published PHP, native's embedded v2 import refuses a bundle whose
-  rows reference rows outside it, links only runs of one namespace, and reads
-  the signing key only from the request body
-  ([#765](https://github.com/durable-workflow/server/issues/765)).
-- **Next action.** Land the items above in order, keep porting the unserved
-  routes fixture first, and package the native binary into the server image.
+- **Remaining for the release candidate:**
+  - the 2.11.3 reference move;
+  - the scope question with the operator: prepared local activities,
+    workflow-bound Nexus handlers and external payload takeover;
+  - publishing the native image (General runs the release);
+  - differential and performance qualification of the published image.
 
 ## API surface still missing
 
@@ -412,14 +380,14 @@ contract fixture before its correction.
   workflow task guard does not check registration freshness yet, and the
   local activity and cancellation scope checkpoints still use the older fence.
   Add a fixture for each before changing them.
-- **Retention of runs with external payloads or message streams.**
-  Published PHP deletes a pruned run's external payload objects and releases
-  its message stream items. It also compacts its instance's consumed inbound
-  stream items, and removes the instance's streams with its last run. Native
-  does not clean any of these up yet. It skips such a run, including any run
-  whose instance has inbound streams, with
-  `external_payload_cleanup_unavailable` or
-  `message_stream_cleanup_unavailable`, and keeps its details.
+- **Retention of runs with external payloads.** Published PHP deletes a
+  pruned run's external payload objects. Native does not delete them yet: it
+  skips such a run with `external_payload_cleanup_unavailable` and keeps its
+  details. A run whose instance has inbound message streams is pruned as PHP's
+  `MessageStreamRetentionCleanup` prunes it (October 27): the messages it
+  consumed lose their payloads once a retained later run's cursor checkpoint
+  covers them, and the instance's last retained run, once terminal, takes the
+  instance's streams with it.
 - **Deeply nested JSON bodies.** Published PHP decodes JSON up to 512 levels
   deep. Native checks every body as PHP does before any route, but its
   handlers parse with serde_json's 128-level limit. A body nested 129 to 512

docs/rust-server-upgrade-takeover.md

@@ -77,6 +77,20 @@ inspection is `unqualified` and takeover refuses with
 `upgrade_source_external_payload_storage_unsupported` before any change. A
 disabled policy does not block takeover.
 
+Inbound message streams and durable workflow streams carry over unchanged:
+native serves them from PHP's own tables. Each closed or errored durable
+stream gets the reclaim deadline native's retention pass reads,
+`dw_stream_reclaims`: PHP 2.11.3's recorded `reclaim_at`, or for earlier
+releases its `closed_at` plus its retention (an hour unless set), as 2.11.3's
+migration backfills it. Every inspection reports
+`stream_reference_inspection`: the number of stream items, durable streams
+and inbound messages holding an external payload reference. Native does not
+resolve those, so any such row makes the inspection `unqualified` and takeover
+refuses with `upgrade_source_external_payloads_unsupported`.
+Workflow-to-workflow messages (`workflow_messages`) are not delivered by
+native yet, so a source holding any refuses with
+`upgrade_source_workflow_messages_unsupported`.
+
 PHP 2.11.3 (Core 2.14) schedules a service call's handler in the
 `workflow_service_call_dispatches` outbox and dispatches it in a later pass.
 Native runs the handler when it accepts the call, so it cannot resume a

rust/src/config_audit.rs

@@ -502,7 +502,10 @@ pub fn audit(environment: &BTreeMap<String, String>) -> Report {
 
 /// Names native reads that PHP's contract does not list.
 fn native_only(name: &str) -> bool {
-    matches!(name, "DW_BIND_ADDRESS" | "DW_ENV" | "DW_RUST_EXPERIMENTAL" | "DW_UPGRADE_BACKUP_FINGERPRINT")
+    matches!(
+        name,
+        "DW_BIND_ADDRESS" | "DW_ENV" | "DW_RUST_EXPERIMENTAL" | "DW_RUST_MAINTENANCE_LOOP" | "DW_UPGRADE_BACKUP_FINGERPRINT"
+    )
         || ["DW_RUST_UPGRADE_QUEUE_", "DW_SCHEMA_TEST_", "DW_TAKEOVER_TEST_", "DW_TEST_"]
             .iter()
             .any(|prefix| name.starts_with(prefix))
@@ -820,6 +823,7 @@ mod tests {
             ("DW_WORKFLOW_MEMO_MIGRATION_RECOVERY", ""),
             ("DW_ENV_AUDIT_STRICT", "1"),
             ("DW_RUST_EXPERIMENTAL", "1"),
+            ("DW_RUST_MAINTENANCE_LOOP", "1"),
             ("DW_BIND_ADDRESS", "0.0.0.0:8080"),
         ];
         let report = audit(&environment(&shipped));