Repository navigation
Fence dedicated update-validation polls against replaced worker registrations #732
Description
Fixed, published and verified in Server 2.11.5 / Core 2.15.1, source 721b2831975cc898cc39d7528c5ba7e8441fe744. Installed consumer and cleanup receipt.
PR #738 merged at 4290969ee0f99751569ac853e4e98f244696c104, which is confirmed in the published source. It captures the admitted registration fence/incarnation and checks both under the worker lock in every claim/retry transaction. Source qualification passed SQLite, PostgreSQL 17.11 and MySQL 8.0.46, including the unfixed counterfactual and unchanged-current-registration control. All six independent main workflows passed; source evidence.
Actual published-image HTTP qualification with normal SDK 2.12.0 passed all four controls: replacement with the same routing/process identity, changed queue, changed capabilities and unchanged registration. Obsolete polls returned the typed stale-registration result with no task, leaving a real update-validation task pending at attempt zero/no lease. The current eligible worker claimed it once and approved the real update. The unchanged poll leased normally. Namespace, contract, queue, attempt and lease checks remain in force.
All fixture workflows were terminated after qualification, all validation tasks are approved, and pending queue jobs are zero. API/queue stopped with exit 0/no OOM on SIGTERM; exact-labelled containers, network and volume are removed. This completes the published PHP consumer criterion. Native ownership is unchanged.
Original problem
At the inspected main baseline bce669d937594aebb6b2ee45992eeb40ba63e55e, the dedicated POST /api/worker/update-validation-tasks/poll path does not fence a worker registration that changes while the request is waiting.
WorkerController::pollUpdateValidationTasks() resolves one WorkerRegistration, and WorkflowUpdateValidationTaskBroker::poll() captures that model across every repeated LongPoller::until() probe. claim() eventually calls workerMatchesTask(), but that checks WorkerPollFence::isFresh($worker) against the captured model; it does not compare the snapshot with the current registration row. A replacement registration with the same worker ID can therefore supersede the request while the old request still appears fresh in memory.
If an update-validation task wakes the old request during that window, the old request can lease it using the superseded process's routing/capability snapshot. The active replacement does not receive the task until the lease expires, so synchronous update admission can time out or be handled by the wrong worker incarnation. This is a within-namespace availability/correctness defect; no cross-tenant access or data disclosure is currently evidenced.
Workflow, activity, and query polling already carry a WorkerPollFence snapshot and compare it with current durable registration state. The dedicated update-validation poll should preserve the same replacement boundary. Closed #35 covered task-kind/version fencing for multiplexed polls, not worker-registration replacement on this dedicated path.
Acceptance
- Capture the authoritative worker-registration fence/incarnation when the dedicated validation poll is admitted.
- Before every claim, including wake and lock-pressure retry attempts, compare that fence with the current registration inside the claim transaction; do not lease when it is superseded.
- Return the existing typed
stale_worker_registrationpoll result (or an equally explicit compatible refusal) so the obsolete request terminates without consuming the task. - Add a counterfactual that starts a waiting dedicated poll, replaces the same worker ID with the same queue/capabilities before the next probe, makes a validation task ready, and proves the old request does not lease it while the replacement can.
- Cover changed routing/capabilities and the unchanged-current-registration control. Preserve namespace, workflow contract/fingerprint, build-ID, attempt, and lock-pressure fences.
- Qualify SQLite, PostgreSQL, and MySQL behavior and the published PHP consumer path. Do not weaken long-poll, lease, or update-validation timeout thresholds.
Original inspection evidence
WorkflowUpdateValidationTaskBroker::poll()closes over the initially resolved model for the wholeLongPollerloop.workerMatchesTask()calls onlyWorkerPollFence::isFresh()on that captured model.- At initial inspection, no existing regression covered replacement during the dedicated validation poll; PR Fence dedicated update-validation polls to worker registrations #738 now provides it.
- Current-main source was re-fetched and inspected at the exact SHA above; no product source was changed during this review.
Activity
rmcdaniel commentedon Oct 24, 2026
Cross-implementation scope correction — 2026-10-24
The Rust Server owner corrected an earlier implementation assumption after inspecting native. Their current finding is:
worker_idduring each validation claim, but does not bind the waiting request to a registration incarnation;stale_worker_registrationresult on replacement, while native query polling does.This is owner-reported scope, not a completed Security source review. Ownership stays explicit: General owns this issue's PHP correction; the Rust Server owner will add a published-2.11.1 replacement counterfactual and native implementation under PR #712, then follow the accepted PHP validation behavior once released. Security will review native only after a formal terminal exact-SHA handoff. No shared/frozen fixture or source overlap is authorized by this note.
Do not close the PHP defect merely because native work exists, and do not claim native parity from the PHP correction. Each implementation needs its own exact evidence and installed/published acceptance.
rmcdaniel commentedon Oct 24, 2026
Merged normally as 4290969 at 2026-10-24 23:36:35 UTC, after the exact integrated head completed 19 successful checks and 7 intended skips. The merge tree is exactly the reviewed and qualified 0e5a9ca3b65a0acca3f51a774b19fbfbd156a669; its parents are current main
dcb2d718and reviewed head83a9b557. The merged remote branch is independently verified absent.Integrated source/review evidence includes current SQLite38/532, PostgreSQL17.11/MySQL8.0.46 each29/305, the current-Core unfixed counterfactual, and Sonar with zero unresolved issues/hotspots/new duplication. Existing discovery deprecations are retained explicitly.
All six independent main workflows on that exact merge are terminal successful:
The last workflow, CodeQL, completed at 23:49:57 UTC; total main activation-to-completion was 13 minutes 20 seconds. PHP completed at 23:44:52 UTC. Published PHP image acceptance remains outstanding; #732 stays open until that user-visible delivery is verified.
rmcdaniel commentedon Oct 27, 2026
Release-state correction (2026-10-27): immutable Server 2.11.5 source 721b283 contains the #738 merge 4290969; exact ancestry was reverified. Protected release run https://github.com/durable-workflow/server/actions/runs/40478372197 completed successfully. The old fixed-image / #450 publisher dependency is superseded for this issue: no Security review, credential action, or operator action is pending. The issue remains open only for its published PHP consumer scenario on exact 2.11.5: begin a dedicated validation poll, replace the same worker ID before the next probe, make a task ready, verify the obsolete request returns the typed stale registration result without leasing, and verify the replacement receives it while the unchanged-registration control still succeeds.
rmcdaniel commentedon Oct 27, 2026
Exact immutable Server 2.11.5 / Core 2.15.1 now passes this issue's installed PHP consumer acceptance. Image:
ghcr.io/durable-workflow/server@sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20. The #738 merge4290969ee0f99751569ac853e4e98f244696c104is confirmed as an ancestor of its published source.A normal PyPI SDK 2.12.0 and real HTTP dedicated-poll calls exercised four controls: replacement with the same routing/process identity, a changed queue, changed capabilities, and an unchanged registration. Each original poll was observed waiting before replacement. A normal synchronous update request created the actual validation task; readonly SQLite observations checked its canonical lease and attempt state.
All three replacements returned
stale_worker_registrationwith no task, leaving the validation pending at attempt 0 with no lease. The eligible current worker then claimed that exact task once and approved it; the real update returnedupdate_status=accepted. In the changed-queue case, the replacement's new queue correctly received no old-queue task and a current eligible worker on the original queue claimed it. The unchanged control retained its waiting poll and leased the real validation task once. The source's PostgreSQL/MySQL/SQLite qualification remains recorded above; this new installed path used SQLite.The first harness attempt expected a Nexus-style
acceptedfield in the direct update response and stopped after the first successful fence/approval. The assertion was corrected against the published direct-update contract; all four controls then passed. No product source, timeout, native/shared/frozen fixture or claim threshold was changed.Every fixture workflow was terminated after its check; all five recorded validation tasks (including the initial harness control) are approved, with zero pending queue jobs. API and queue stopped with exit 0/no OOM on SIGTERM. Exact task-labelled containers, network and database volume were removed and label inventories are empty.
This completes #732's published consumer criterion alongside its previously qualified three-database source and counterfactual controls.