Repository navigation
Cooperative cancellation of a retried activity cancels the wrong task, so the stop can't be acknowledged #923
Description
The retried-activity cancellation repair is published and verified in Server 2.11.5 / Core 2.15.1. Core 2.13.1 originally delivered the library repair; Server #733 is an ancestor of the immutable published Server source. The earlier source-only 2.11.3 and publisher dependency are superseded.
Installed custom-worker protocol acceptance and normal SDK Worker/application acceptance pass with normal PyPI SDK 2.12.0. Both cancellation policies retry an actual activity once, preserve the earlier completed task, cancel the live attempt/task, snapshot that live task, and return the original event on duplicate acknowledgement. The normal Worker also stops the actual callback subprocess and acknowledges automatically. Public HTTP history is verified; both workflows finish cancelled. Exact-labelled runtime resources are removed.
Installed Waterline2.13.0 read-side/browser acceptance and matched-image SQLite snapshot recovery now pass for both policies on the published Server2.11.5/Core2.15.1 tuple. The original completed task, live cancelled task and acknowledgement identity survive restore; a post-snapshot namespace is absent and authentication refusal remains401. Actual Waterline code matches its published tag across166files. General has prepared the missing SQLite recovery instructions in Server commit c54761bd7f451b14f975cc7c87203c588a8de9b2 (docs/923-sqlite-recovery-qualification). Next: review and deliver that documentation through normal protected checks after the current shared CI repair; keep this issue open through that delivery. Native parity and the broader maintenance product remain in their owning work.
Goal
Delivering a cooperative cancellation to a retried activity cancels that activity's live attempt and task. The activity worker can then acknowledge the stop. A task that already finished keeps its recorded status.
Original behavior
WorkflowExecutor::cancelOpenWaitAtSequence (src/V2/Support/WorkflowExecutor.php, line 5832 on main and in Workflow 2.10.2) picks the activity task with:
$activityTask = $run->tasks->first(
static fn (WorkflowTask $candidate): bool => $candidate->task_type === TaskType::Activity
&& ($candidate->payload['activity_execution_id'] ?? null) === $execution->id,
);That is the first activity task ever created for the execution, whatever its status or attempt. A retried activity has one task per attempt, so this returns attempt 1's task, which already completed. ActivityCancellation::record then:
- sets the old, completed task to
cancelled; - leaves the live attempt's task
leased, while the attempt and execution becomecancelled; - records
ActivityCancelledwith attempt 2's snapshot but attempt 1's task snapshot.
The activity worker's POST /api/worker/activity-tasks/{task}/acknowledge-cancellation for the live attempt is then refused with 409 activity_cancellation_not_fenced, because its task is not cancelled. The stop can never be acknowledged.
ActivityCancellationWait::prepare (src/V2/Support/ActivityCancellationWait.php, about line 47), used by the wait_cancellation_completed policy, has the same lookup. The equivalent lookup near WorkflowExecutor.php line 7108 filters to ready or leased tasks, which is the intended behavior.
Reproduction (published Server 2.11.1, raw worker HTTP)
- Register a worker with
capabilities: ["cooperative_cancellation"]at protocol 1.20. - Start a run, and complete its workflow task with one
schedule_activitythat hasretry_policy: {max_attempts: 2, backoff_seconds: [0]}. - Poll the activity (attempt 1, task A) and
failit withnon_retryable: false. Poll again (attempt 2, task B). POST /api/workflows/{id}/runs/{run}/request-cancellation, poll the workflow task, anddeliver-cancellationwithcall_kind: "activity",sequence: 1. The answer is200 delivered: true.- Attempt 2's
statusnow showsattempt_status: cancelledandactivity_status: cancelled, buttask_status: leased.ActivityCancelled.payload.taskis task A withstatus: cancelled. acknowledge-cancellationfor task B and attempt 2, with the deliveredrequest_id, answers409 activity_cancellation_not_fenced.
With max_attempts: 1 the same flow acknowledges with 200. Found while recording the Server parity fixture for this route (Server #325).
Acceptance
- A test retries an activity once, delivers a cooperative cancellation, and shows the live attempt's task becomes
cancelled, the earlier completed task keepscompleted, andActivityCancelledsnapshots the live task. - The activity worker's acknowledgement for the live attempt succeeds, and its duplicate returns the original event.
- The
wait_cancellation_completedpath (ActivityCancellationWait::prepare) gets the same test.
Activity
rmcdaniel commentedon Oct 24, 2026
Accepted for a PHP/Core correction. I will fix both the delivery and wait-cancellation paths so they select the live task for the current attempt, retain the earlier task’s completed state, and qualify the recorded task snapshot plus successful and duplicate acknowledgement. The termination-history/#901 contract question remains a separate coordinated decision; no frozen/native fixture or published reference is changed by this acceptance. This issue stays open through normal package and affected Server-image acceptance.
rmcdaniel commentedon Oct 24, 2026
Core 2.13.1 is normally published from
b2a3369110b8cbdf060802ff91e475110e66e333, tree1384c8f6fc8d45a02ab9afae8322131914fce5a2. The immutable GitHub tag/release, Composer product-train, and Packagist source/dist references agree. Release.The exact source passed full main verification40101884828, including all database shards, complete coverage, five Laravel upgrades, quality, corpus and Sonar. Normal published verification40132873412 passed all17 jobs, including the package contract and16 Laravel/PHP upgrade cells.
A fresh normal Composer consumer on PHP8.3.36 / Laravel13.38.0 / Testbench11.3.0 / PHPUnit12.5.40 passed 11 tests /132 assertions, with no skips or warnings. It exercised cooperative cancellation after retry under both cancellation policies, completed-task preservation, the current attempt/task snapshot and successful+duplicate stop acknowledgement, plus all ten embedded lease-maintenance, genuine-progress, deadline and worker-loss controls. Stock Composer autoload resolved Activity, WorkflowExecutor and ActivityCancellationWait from the installed package; no product-source override was used. All 2,176 package files matched the tagged source before and after the tests. Composer reported zero security advisories.
This delivers the library repair and preserves the earlier failed main run as failed evidence. PHP Server adoption is continuing in durable-workflow/server#733, followed by the exact published-image acceptance. This issue remains open through the affected Server image; no native parity or rejected-outcome emitter acceptance is claimed.
rmcdaniel commentedon Oct 24, 2026
Core adoption merged in Server #733 at
922259fcbc855407961a014c697db21915c28f04, with exactly the accepted PR tree2af3195c564d7c38a1cc9ad9485449dad9a9a487. The PR head31faf34cpassed its hosted checks, and Security accepted the bounded dependency/release integration in its final review. The merged remote head branch has been deleted and independently verified absent.The source now locks normally published Core 2.13.1 (
b2a3369110b8cbdf060802ff91e475110e66e333). Local combined checks passed 80 tests / 2,983 assertions, all 3 release-generator contracts and targeted formatting. Hosted checks additionally passed the PHP feature/Nexus/corpus suites, candidate production-image readiness, database TLS/certificate refusals, Helm install, replay/query, rolling upgrades, performance, CodeQL and Sonar. The required PHP workflow took 12m54s wall time including startup/cleanup; no queued second PR was added. Native fixture compilation/execution was intentionally path-skipped and is not claimed as qualified.Independent main verification is now complete: all eight normal main workflows succeeded at exact merge
922259fc, including PHPUnit Feature, Sonar, Helm, replay/query, rolling upgrade, performance, public-boundary and CodeQL. PHP feature/Nexus/corpus, candidate production-image first-run readiness and required database TLS/certificate refusals all passed. The PHP workflow took 11m27s; the final CodeQL workflow completed at 21:57:43 UTC, 13m31s after main activation. Server 2.11.3 / chart 0.3.3 are still prepared source. A published PHP image and affected installed application/SDK/Waterline/rollback acceptance remain outstanding; this does not close Workflow #923 or the full maintenance product requirement.rmcdaniel commentedon Oct 27, 2026
Release-state correction (2026-10-27): immutable Server 2.11.5 source 721b2831975cc898cc39d7528c5ba7e8441fe744 contains Server #733 merge 922259fcbc855407961a014c697db21915c28f04 and locks Workflow 2.15.1 at 07e0d0e. That exact Workflow commit descends from the retried-cancellation fix release b2a3369. Protected release run https://github.com/durable-workflow/server/actions/runs/40478372197 completed successfully. The old Server 2.11.3 / #450 publisher dependency is superseded: no Security review, credential action, or operator action is pending. This issue remains open only for exact published-image acceptance of both cancellation policies after a retry, preservation of the earlier completed task, live task/event snapshot, successful and duplicate stop acknowledgement, and the affected application/SDK/Waterline/rollback surfaces.
rmcdaniel commentedon Oct 27, 2026
Published PHP consumer acceptance now passes for the cancellation-after-retry defect on exact Server 2.11.5 / Core 2.15.1, image
ghcr.io/durable-workflow/server@sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20. Server #733 merge922259fcbc855407961a014c697db21915c28f04is independently confirmed as an ancestor of this published source.A normal PyPI SDK 2.12.0 installation drove real control-plane and custom-worker protocol calls against a fresh isolated SQLite runtime, using its public registration, claim, command, failure, cancellation, observation and acknowledgement methods. This qualifies the installed PHP protocol through the SDK; it does not claim the SDK's automatic activity-callback cancellation loop.
Both
try_cancelandwait_cancellation_completedpass after one real retry: attempt one is failed retryably and its completed task retained; attempt two has distinct task/attempt identities. Cancellation makes the live attempt, execution and live task cancelled. Readonly stored-state checks prove the first task remains completed and the singleActivityCancelledevent snapshots the second task. Stop acknowledgement succeeds, its duplicate returns the same history event, and exactly oneActivityCancellationAcknowledgedevent exists. The waiting policy resumes after acknowledgement; both workflow cleanups finish cancelled. No timeout, lease, schema, product source or frozen/native fixture was changed.Both API and queue stopped on explicit SIGTERM with exit 0/no OOM, with zero pending jobs and two cancelled runs. Exact task-labelled containers, network and database volume were removed and label inventories are empty.
The old publisher dependency is resolved. This completes the exact installed PHP retried-task/acknowledgement criteria. The broader affected application/automatic-SDK/Waterline/rollback surfaces retained in the owning record remain separate follow-through; no native parity or full maintenance-product acceptance is claimed. Next: qualify those affected surfaces against this published behavior, retaining the published protocol evidence above.
rmcdaniel commentedon Oct 27, 2026
The normal SDK Worker/application path now also passes on the exact published Server 2.11.5 / Core 2.15.1 image, with normal PyPI SDK 2.12.0. This supplements the custom-worker protocol receipt.
For each of
try_cancelandwait_cancellation_completed, an actual activity subprocess fails once with a normal exception, retries through the Server and waits in its second callback. A workflow cancellation then stops that real callback process. The ordinary SDK Worker automatically acknowledges the stop; readonly status shows the stopped receipt with the original request/root request/cleanup deadline, and an explicit duplicate acknowledgement returns the same event ID.Both application workflows finish cancelled within the original ten-second completion wait. The old completed task remains completed, the live attempt/task is cancelled, and public HTTP history contains one
ActivityCancellednaming the live task and oneActivityCancellationAcknowledged. For the waiting policy, acknowledgement precedes cooperative delivery. The actual callback PID is independently confirmed gone. No SDK or Server source override, timing relaxation, fixture rewrite or native execution was used.Each fixture ended with two cancelled runs and zero pending queue jobs. API and queue both stopped with exit 0/no OOM using explicit SIGTERM; exact task-labelled containers, network and database volume are removed and inventories are empty. Small input/report files remain only for this issue's outstanding read-side/rollback qualification.
The PHP library, immutable Server image, protocol, normal Worker/application cancellation and public history paths are now verified. Remaining recorded scope is affected Waterline/read-side and rollback acceptance; no native parity or full maintenance-product acceptance is claimed.
rmcdaniel commentedon Oct 27, 2026
Installed Waterline read-side and isolated SQLite snapshot recovery now pass for both retried-activity cancellation policies.
Published tuple
sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20, source721b2831975cc898cc39d7528c5ba7e8441fe744; installed Core2.15.1/07e0d0e1a1185bd0a3bbd4e3fb56511cc57ca9bc.sha256:925a20c8b339991658130ebfd1688727d43ff256dc4f217e771939623a3d23d9, source30ec59eac270dafd3025afd65175670c6a0f7ab6, installed PHP SDK2.4.0/821716a92f179ec360e045c28979a86a5f587075. All166installed Waterline application/configuration/migration/public/resource/route files match that tag. No product-source overlays.Actual consumer checks
A new real SDKWorker cohort retries each activity once, then cancels the live callback under
try_cancelandwait_cancellation_completed. Both callbacks stop and acknowledge automatically; duplicate acknowledgement returns the original event. The earlier completed task remains completed; the live task is cancelled; public history snapshots that live task. The Worker emitted two fence-observation warnings after cancellation; these are retained and are not represented as zero SDK warnings.Fresh Playwright1.56.1 browser contexts open each installed Waterline run, use its “Inspect full details” action and render the cancelled status/history. The actual Waterline API preserves the live-task snapshot, request identity and single acknowledgement history position. The browser checks pass for both policies before and after recovery, with0page errors and0Waterline API errors. UI history positions11/9 map to the original retained acknowledgement event IDs; the UI's generated row identifier is not substituted for that event identity.
Snapshot recovery
Server API/queue writers stop cleanly with SIGTERM/exit0; the SDK is stopped, jobs are drained and both runs are terminal. A private offline working copy is converted with SQLite's backup API to a separate protected snapshot; integrity check returns
ok. Opening the original WAL-mode file from a read-only directory initially failed, so recovery uses the writable working copy while leaving original files untouched. The proposed instructions include that requirement.After the snapshot, a fresh namespace is created and read through the published SDK. Writers stop again. The snapshot is restored into a new directory and restarted with the same pinned Server image, encryption key and authentication configuration; the actual restored application key matches the retained value. The later namespace is absent, proving the recovery point was applied. Authorized SDK reads pass; an unauthenticated Server namespace request returns401. Both original task statuses, acknowledgement IDs and history positions remain unchanged, and both Waterline browser/API checks pass again.
This qualification covers stopped-writer, inline-payload SQLite recovery with the matched image/key snapshot. Existing MySQL/external-object procedures retain their separate qualification.
The SQLite recipe is prepared at Server commit
c54761bd7f451b14f975cc7c87203c588a8de9b2, branchdocs/923-sqlite-recovery-qualification; diff checks pass. The final documentation uses an explicit integrity-check failure instead of depending on Python assertions. The existing recovery guide only covered the MySQL Compose procedure. No new PR is opened while the current shared merge-group CI failures are being repaired. Keep923open until the reviewed documentation is delivered normally.Cleanup verified: all exact-labelled task containers and the task network are absent; source/restored databases, protected backup and bulk browser/package scratch are removed. Server API/queue exited0; the read-only Waterline process exited143 on SIGTERM. Only small receipts and inputs remain for the active documentation delivery; remove them when this issue closes. Shared base images and the tooling still needed for the active #816 source review are preserved.
Cleanup correction: the first default-profile Compose removal left the stopped UI profile container behind. Explicit UI/bootstrap-profile cleanup then completed, and fresh exact-label/project checks verify0containers,0networks and0project volumes. All listed bulk directories are now verified removed.