Abandoned worker polls can claim tasks and retain capacity after disconnect or restart

Current state — 2026-10-25 08:15 UTC: draft PR #758 implements stateless offers and replayable positive acceptance. Combined head be4ea6c8418d56294f0b9e968a4cede91c77fa3e passed all19 checks with7 intended skips. New head ed302326f0f73c383df88a918c8378755e0a56fe resolves expired packets against the actual task authority under the registration lock: an original live claim remains replayable, otherwise the Server returns definite expiry without touching peer authority. Local qualification:220 tests /7,642 assertions, zero failures/errors/skips; all6 new regressions fail against the preceding implementation. Current-head hosted checks are running.

PHP SDK offer/accept integration is implemented locally and remains unpublished: negotiated endpoints, immutable selected-packet retries across uncertain replies/client clones, registration-incarnation replacement, validation before payload observers, and managed-loop identity retention. Static analysis passes;105 focused tests /485 assertions pass. The first full run reached2,853 tests and found one stale registration fixture (56 existing fixture skips); that fixture was corrected and its focused suite passes. First-party integration and real proxy/restart/worker-replacement/fixed-published-image acceptance remain unfinished. No published Server fix is claimed. Process-capacity cleanup merged in #753; progress and diagnostics merged in #755. The delivery contract and full requirements remain intact.

Problem

When a proxy or load balancer closes a worker's long poll before the Server's wait ends, the Server keeps holding the abandoned request. That request can still claim a task. The poller never checks whether the client is still connected, so the task sits on a lease nobody will complete until the lease expires. Abandoned requests also keep their long-poll wait slots, so live polls are rejected as capacity-exhausted. No Server surface names the cause.

website#267 covers documenting timeout budgets. This issue is about the Server's behavior when a budget is violated, since a proxy shorter than the wait is a common deployment state, not an exotic one.

Reproduction

Published durableworkflow/server:2.9.7 from docker-compose.published.yml, with nginx proxy_read_timeout 10s in front. PyPI durable-workflow==2.6.1 worker, one-activity workflow, 5 trials per case:

Case	Start to completion
Control, 60 s proxy	0.6–1.6 s
10 s proxy, random start time	0.6–1.6 s
10 s proxy, started in the window after the SDK abandons a cut poll	62–63 s, and 123 s once

In the slow cases, the task was leased to the abandoned poll id and released only when the 60 s workflow-task lease expired. In one trial the redelivered task was leased to another abandoned poll.
The worker logged only workflow poll error: server returned 504. Over 9 minutes, live polls received 256 429 long_poll_capacity_exhausted responses, mostly on the activity poller, all logged at debug level.
While the task sat on the dead lease, run detail said "Workflow task leased to worker", task-queue repair reported needs_attention: false, and the Server logged nothing. Only nginx's own log named the cause.

Acceptance criteria

A task claimed by a poll whose client has already disconnected is made available again promptly, not after a full lease timeout.
Abandoned polls stop occupying long-poll capacity once their client is gone.
When polls are being cut short of the requested wait, the Server's task-queue diagnostics and the SDK workers report that at warning level, naming the observed cut-off and the configured wait.
A published-image test with a proxy shorter than the long-poll wait shows no lease-timeout delays and no capacity starvation.
Qualify abrupt Server-process restart and worker replacement separately with persistent/shared polling-cache storage. A restart must not leave dead processes occupying wait capacity while the API is serving again.
Preserve bounded global/namespace limits, exact-owner release and live waiters on other replicas. A capacity hint must not become task authority.

Add Server-process restart and shared-cache lifetime to this product fix's acceptance. The current LongPollWaitSlotStore source at 36d162736 still reserves a random-owner cache entry until the requested wait plus five seconds, and releases it when the request finishes. A process killed before release can leave that entry consuming capacity until TTL expiry even after the API is serving again.

Qualify abrupt Server restart, worker replacement and a disconnected client separately, with persistent/shared polling-cache storage. Preserve bounded global/namespace limits, exact owner checks and live waits on other replicas. A correction must not delete another live waiter's slot or claim that a capacity hint grants task authority. Also preserve the original lost-response/task-lease and warning/diagnostic acceptance in this issue.

General owns the PHP follow-through. Reproduce against the current immutable published image before a current-release attribution, then qualify the lifecycle correction and an actual bounded recovery. Raising capacity limits is not the product fix. The already merged Server 2.11 activity-lease/logging source is separate and does not claim to resolve this issue.

General reproduced the remaining lifecycle defects with current immutable published artifacts on 2026-10-24, 23:01–23:06 UTC. No application source overlay or native engine was used.

Artifact tuple:

Server 2.11.1, amd64 published manifest sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102, image config sha256:c53162134a94025a0b87b995e9b14896c1fdfffbeeace3b530d3fbd0f2b23c6c, OCI revision 5336a98dd256e046edfdb67c81764cde4babd360, PHP 8.3.36.
Normal PyPI durable-workflow 2.11.1 installation; downloaded wheel SHA-256 1e7f0210ef30ae33e2224f6857386a43e7793b725a298a0974b2bd6e53c0318f, negotiated worker protocol 1.20.
MySQL 8.0.46, Redis 7.4.11 shared polling cache, nginx 1.30.5. Requested worker wait 30 seconds, existing global worker-wait limit 2. All fixture containers ran as UID/GID 1000 with bounded resources and no published host ports.

Scenario	Observed result
Normal worker, 60-second proxy timeout	One-activity workflow completed in 1.332 s.
Same worker survives the actual 10-second proxy cutoff and retries	nginx recorded a workflow-poll `504` at exactly 10 seconds; a workflow started immediately afterward completed in 1.253 s. This current SDK recovery path does not reproduce the historical minute-long delay.
Abrupt worker replacement after the actual 10-second cutoff	Original SDK process was killed and exited 137 before workflow creation. Nevertheless its abandoned HTTP poll leased the first workflow task after the process was dead. The normal successor completed the workflow only after 26.728 s, with 13 workflow-capacity backpressure messages at DEBUG and no warning identifying the cutoff.
Abrupt Server-process restart, MySQL and Redis left running	Both stale worker-wait entries survived. Their TTL was 9 s immediately before the kill and still 6 s when `/api/ready` returned ready after restart. Live SDK polls repeatedly received `429` responses after readiness. A newly created workflow still completed in 2.127 s, because immediate task claims can bypass waiting-capacity admission; this is not evidence of complete task unavailability.

The replacement case has database evidence, not just elapsed time: its first task was leased to the dead original worker at 23:03:50.243596 UTC, with a nominal lease expiry at 23:04:50.243530. It became ready again at 23:04:15.808516 and the successor completed it on attempt 2. Thus current recovery occurred before the full 60-second task lease; the reproducible current failure is the abandoned claim and roughly 25.6-second ownership delay, not a claim that the historical full-minute timing persists unchanged.

For the restart case, the exact MySQL and Redis container IDs and start times were identical before and after the Server kill/start. The shared cache was not flushed, capacity limits were not raised, and no other replica's waiter was removed. An initial observer looked in Redis DB 0 instead of the configured cache DB 1 and timed out before performing any restart; that harness failure is excluded from product evidence. The corrected observation and actual restart are the results above.

Next: implement request/process-lifetime ownership and prompt abandoned-delivery recovery, then qualify disconnect, worker replacement and abrupt Server restart separately. Preserve global/namespace bounds, exact-owner release, live waits on other replicas and task-authority checks. Warning/diagnostic behavior, multi-replica controls and a fixed published image remain unqualified; the issue stays open with its full acceptance intact.

Transport constraint for #526, verified in an isolated fixture on 2026-10-24 at 23:17 UTC. This used the exact published PHP/Apache image from the baseline above, with a disposable probe vhost and nginx configuration. It is transport evidence, not application or fixed-image acceptance; product source was not modified.

With no response output during an eight-second wait, nginx returned 504 at 3.001 s. PHP's connection_aborted() remained 0 on every observation through 8.003 s, even though the client was already gone.
Flushing JSON whitespace once per second kept the same three-second upstream timeout alive, with nginx's default proxy buffering enabled. The client received HTTP 200 and parsed the unchanged final JSON object after 8.006 s.
When that client instead closed its connection at two seconds, the progress-writing PHP observer detected the abort 1.005 s after client close.

This agrees with PHP's documented output-dependent disconnect detection. A silent connection_aborted() check alone would not correct the published Apache failure. Response progress is a viable transport primitive in this fixture, but a real correction must preserve admission/status/error semantics, explicitly unwind exact request resources, and qualify actual first-party SDK/task behavior. Process-kill/shared-cache cleanup remains a separate required part of this issue. The observer used ignore_user_abort(true) solely to retain evidence after a disconnect; no product cleanup behavior was proven by that setting.

All transport fixture containers and its internal network were removed after the observations. No shared service, native source, provider route or credential was involved.

General verified an additional process-lifetime primitive for #526 on 2026-10-24 at 23:56–23:57 UTC. This is an isolated Redis/PHP experiment, not a product correction or application acceptance.

Using the same immutable published PHP image as the baseline and Redis 7.4.11, two separate bounded PHP containers each opened a private subscription through the existing PhpRedis extension. No message relay, heartbeat daemon or additional provider service was created. Both connections initially reported one subscriber. The first process was then killed with SIGKILL and exited 137; the second process remained running.

The next observation, less than one second after the killed process finished, showed zero subscribers for its connection and one for the live peer. A Redis Lua operation combining the exact cached owner comparison with that liveness observation:

reclaimed only the dead owner's synthetic slot;
refused a wrong expected owner;
refused to reclaim the live peer's slot;
refused a late release by the old owner after replacement, preserving the replacement and live-peer values.

No TTL expiry was needed. This gives a candidate way to distinguish process death from a slow live waiter without guessing from a shorter timeout. The experiment used synthetic raw cache values and a standalone Redis node. It does not yet qualify Laravel cache encoding/prefixes, TLS/ACL/client modes, Redis Cluster, rolling legacy owners, file/database cache backends, namespace/global reservation integration, or actual HTTP disconnect/delivery cleanup. Those remain implementation and qualification work; the full issue acceptance is unchanged.

The temporary Redis service, both process containers, all one-off containers and their internal network were removed after the experiment. No product source or production state was changed.

PHP process-lifetime source qualification, 2026-10-25 00:50 UTC. Draft PR #753, exact head 5fd88d0 / tree b6427986171e938554888714406a5d96d6f41d5b.

The existing wait limits are per API node, so the implementation uses kernel-owned local file locks under the existing writable polling directory, scoped by the unique Server ID and separate worker/query-task pools. A nonblocking admission mutex checks global and namespace budgets together; one slot file contains namespace metadata. No per-namespace file collection, Redis subscriber, heartbeat service, shorter guessed TTL, or task-authority mirror was added. Live locks do not expire; exact-owner release is idempotent and idle files are reused/compacted. Unavailable local storage refuses waits rather than exceeding limits. The removed cache prefix is retired from the active inventory; legacy values drain naturally and other replicas are not deleted.

Validation on normal installed Core 2.13.1 / PHP 8.3.36: Pint and 55 tests, 7,246 assertions pass. Coverage includes real process SIGKILL with live same-node and other-node peers, query-pool separation, namespace/global caps, unavailable cache/local storage, late duplicate release, 20 sequential namespaces and burst compaction.

Actual HTTP qualification mounted the candidate PHP source and stock Composer dependencies into immutable published PHP/Apache Server 2.11.1 (manifest sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102), with two API replicas, MySQL 8.0.46 and Redis 7.4.11. Both nodes shared polling storage and Redis with global cap 2 / namespace cap 1.

Killed the Apache process holding tenant A: only its slot released; a replacement occupied it in 0.231 seconds. The same-node tenant B and both other-node waits remained live.
Then killed and restarted the whole first API container: its two locks released in 0.300 seconds, before restart/readiness and without a cache-TTL wait. Replacement waits occupied both slots within 1.483 seconds of readiness.
Both replicas still refused excess idle polls with typed HTTP 429; a zero-wait immediate probe remained available. MySQL, Redis and the other API container retained exact IDs/start times, and a preexisting shared-cache sentinel retained its value.

The first fixture attempt assumed all simultaneous admissions would succeed and stopped before any kill when one got the intentionally nonblocking mutex's normal retryable 429. The corrected client honored the existing advertised one-second retry before holding a wait. That initial result is retained, not counted as process-recovery acceptance. The successful run and focused tests completed with exit 0. All fixture services, one-offs, network and volumes were removed and absence verified.

This proves the process-death capacity component on candidate source. It does not prove silent client-disconnect handling, prompt abandoned task-delivery recovery, cutoff diagnostics/warnings, or a fixed published image. Those remain required; this PR does not close #526. The earlier Redis PubSub experiment was not selected because the documented per-node capacity contract can use existing local storage without new Redis connections/ACL assumptions.

Additional published-SDK transport constraint for the remaining #526 work, 2026-10-25 01:11 UTC. This is a loopback response experiment, not a Server correction or a claim of fixed-image acceptance.

Used normal PyPI durable-workflow 2.11.1, wheel SHA256 1e7f0210ef30ae33e2224f6857386a43e7793b725a298a0974b2bd6e53c0318f, with its installed httpx 0.28.1 in a bounded nonroot container. The real SDK poll call requested timeout_seconds=0; its installed timeout helper reported a 1.0-second HTTP budget per attempt.

Silent-body control: six actual HTTP attempts completed in 6.674 seconds and returned the SDK's normal task=null / poll_status=timeout result. The nested transport retries plus outer poll retry are real behavior; this is not six duplicate successful tasks.
The identical valid JSON response with one newline every 0.2 seconds for three seconds: one actual HTTP attempt succeeded after 3.023 seconds despite that same 1.0-second per-attempt HTTP timeout.

Therefore progress bytes can keep the current read timeout alive beyond the intended per-attempt wall-clock budget. Any negotiated progress transport must add/preserve an absolute attempt deadline in SDKs rather than silently renewing it with each byte. It must also preserve typed late error status/retry semantics; emitting whitespace into an existing HTTP200 response alone cannot complete #526. This complements the earlier real PHP/Apache/nginx disconnect probe, while leaving all accepted delivery/diagnostic/published criteria open.

The initial observer assumed two silent attempts and asserted before recording its results; the actual six-attempt observation was retained and the observer corrected to record evidence before its checks. The successful observer completed exit0. No SDK or Server product source was changed. The SDK/dependencies lived only in disposable tmpfs; the container is removed, no host package/cache duplicate retained.

The next #526 candidate now has concrete HTTP progress and disconnect-cleanup evidence; it is still local, unpushed and unpublished.

The transport is strictly opt-in through Accept: application/vnd.durable-workflow.poll.v1+json. Legacy JSON remains the default. Progress consists of JSON whitespace during the wait, followed by a versioned envelope containing the original status, selected headers and unchanged encoded worker body. This preserves late errors after progress commits HTTP 200. Only the workflow/activity/query poll routes negotiate it.

Corrected candidate qualification: 129 PHP tests / 5,196 assertions, zero failures/errors/skips, plus Pint. Tests exercise the real Laravel routing pipeline, late 409/503/500 responses, exact wait cleanup, and all three duplicate-result waiters retaining their live leader's pending key. An initial run exposed a real Fiber-resume-after-termination bug when Laravel rendered the interruption; that was fixed before the passing cohort.

Actual PHP/Apache product requests through nginx with a 3-second idle timeout completed all three 4-second empty polls in 4.05–4.07 seconds, preserving the envelope and disabling mixed response compression. Early 422 and ordinary legacy JSON were independently checked. Closing a real 30-second client poll released its held wait slot after 2.017 seconds, with a Server warning naming observed duration and configured wait. All fixture services, networks and volumes were removed. An initial fixture-client gzip decoding error happened during legacy registration before polling; the observer was corrected and the fixture recreated.

This does not yet prove task-delivery recovery. A failed response can race with successful same-poll replay: blindly revoking its lease could revoke a task another response delivered. A completed socket write is also not SDK receipt. I am keeping authoritative recovery separate from wait/cache hints while qualifying that race. Server task-queue diagnostics, all first-party SDK integration/warnings and absolute attempt deadlines, and the fixed published-image/restart/replacement acceptance remain open. The published-SDK deadline evidence above still applies.

The Rust owner confirmed legacy-default negotiation and will own the matching shared parity case/port when this contract is ready for release. Existing frozen/shared fixtures remain unchanged. No claim of a released fix or completed #526 acceptance is being made.

Python transport integration is now implemented and locally qualified; it is still unpushed/unpublished (candidate e5f958a330b1bbeaf7a4d40f274a380b0dd93e08, based on current SDK main 351192443943d645f84b541b4e45f2b002e81193).

The client opts into the progress media type while accepting legacy JSON. It unwraps the terminal status and allowed retry/protocol headers before existing typed errors and retries. Each HTTP attempt has an absolute deadline, so progress bytes cannot renew it indefinitely. Early transport failures and HTTP 502/504 warn with observed duration and configured wait; intentional cancellation closes the request without that warning. The finite retry budget and original request body/poll ID remain intact. Coverage includes workflow, activity, query and dedicated update-validation client polls.

Qualification: 331 tests passed on Python 3.10 and 331 on Python 3.12, with zero failures/errors/skips; Ruff passed and strict mypy passed all 39 source files. Real chunked HTTP peers exercise progress, deadline, cancellation/socket closure and public retry identity. Replacing only the client with its unchanged parent makes all four absolute-deadline regressions fail, with zero test errors. The first cohort's incorrect generic-422 expectation was corrected to the existing InvalidArgument contract; the broader harness's missing telemetry dependency was installed before its successful run. No product behavior was relaxed to pass either check.

A locally built candidate SDK then exercised the actual candidate PHP Server through nginx's 3-second idle timeout: workflow/activity/query 4-second empty polls completed in 4.05–4.07 seconds using negotiated responses. Actual PHP 422 still became InvalidArgument, and ordinary PHP JSON remained available. That product HTTP run covers those three Server routes; the fourth client route is covered by client/real-peer tests, not claimed as a fourth PHP progress integration. All fixture services, networks, volumes, temporary dependency packages and inactive lock files are removed.

This advances the required SDK/transport component. Replay-safe abandoned-task recovery, Server task-queue diagnostics, PHP/Rust SDK integration and fixed published-image/restart/worker-replacement acceptance remain open. I have not published a Server image or changed the task-authority model, and #526 remains open through its full acceptance.

The latest Server candidate is now visible in draft PR #755, based on current main. It adds strictly opt-in JSON progress, exact disconnected-wait cleanup, and recent queue cutoff diagnostics. Authoritative task-delivery recovery remains unfinished; the PR does not close #526.

Latest local qualification: 153 tests / 8,454 assertions, zero failures/errors/skips, plus Pint. Actual PHP/MySQL/Redis through nginx's three-second idle timeout completed all four four-second poll types, including dedicated update validation, in 4.05–4.08 seconds. Closing a 30-second client freed its wait slot after 2.026 seconds; the real queue API reported 2.021 observed seconds / 30 configured seconds and repair.needs_attention=true. Early 422 and ordinary JSON controls passed. Diagnostic I/O follows wait release, and the last-observation cache has a bounded namespace/queue key with a 15-minute TTL. All fixture resources are removed.

Python SDK 2.11.2 is published and independently verified: normal registry installation, all 40 package files byte-identical to source, 33 installed transport tests passed, and actual published-PHP legacy/proxy-warning behavior checked with no source overlays. The measured SDK CI dependency improvement is recorded here.

Remaining work includes replay-safe abandoned-task recovery, PHP/Rust SDK integration, and the complete fixed published-image/restart/worker-replacement acceptance. A failed response must not revoke a task already confirmed through another response, and transport observations must not become execution authority. All original acceptance criteria remain open through verified delivery.

General — current task-delivery implementation and remaining acceptance.

The progress/diagnostic component merged in #755. Python SDK 2.11.2 and PHP SDK 3.5.1 are published and independently installed/verified for their transport components. The authoritative delivery candidate is now visible in draft PR #758, validated implementation 914b40fc1225a998bc4059c4e7075d3a43de761b (current combined head be4ea6c8418d56294f0b9e968a4cede91c77fa3e is requalifying); it remains unpublished.

The implementation uses stateless, sealed offers that never reserve or lease a task. Losing an offer response starts no activity attempt/history and leaves the task immediately claimable by a peer. Explicit /offers endpoints prevent an older Server from silently ignoring negotiation and claiming through its legacy poll handler. A worker's positive acceptance revalidates the exact task, attempt, namespace, queue, route, original poll/kinds and registration incarnation before the existing claim. SQL receipts are atomic with normal task claims in the existing rows; query receipts remain in the existing authoritative cache task record. The short acceptance budget is separate from authored scheduling, query deadlines, business retries and normal leases. No transaction spans a network write.

Lost accepted replies replay the existing live claim, including after offer expiry and ordinary worker heartbeats; a registration replacement or routing/process change remains fenced. Competing multiplexed SQL offers share an indexed binding and the worker lock. An unaccepted offer needs no expiry sweep because it grants no reservation or execution authority.

Current qualification: SQLite214/7448 at current head; MySQL43/1343 and PostgreSQL43/1343 for the SQL claim paths at9048c5a8; all passing without failures/errors/skips, plus Pint. Query cache authority is confirmed from the stored record, with discovery prepared before commit and no provisional lease hint. Six write/index failure cases and actual legacy-hint recovery preserve pending/accepted authority correctly. At the validated implementation, all 19 checks passed with 7 intentional skips, and Sonar has zero unresolved PR findings. The current combined head incorporates accepted main #757 CI changes and is requalifying. Full SDK and published acceptance remain unfinished; current evidence.

Required before full acceptance: harden alternative offers and cache failure points; integrate all first-party SDKs with unchanged-token/endpoint/poll retries for uncertain acceptance; qualify actual proxy disconnect, abrupt Server restart and worker replacement separately with persistent/shared polling storage; then repeat the original short-proxy case against an exact fixed published image. Bounded global/namespace capacity, exact-owner release, live peers and non-authoritative wait hints remain required. The draft does not complete any unproved original acceptance criterion.

Navigation Menu

Repository navigation

Abandoned worker polls can claim tasks and retain capacity after disconnect or restart #526

Problem

Reproduction

Acceptance criteria

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Abandoned worker polls can claim tasks and retain capacity after disconnect or restart #526

Description

Problem

Reproduction

Acceptance criteria

Activity

rmcdaniel commented on Oct 22, 2026

rmcdaniel commented on Oct 24, 2026

rmcdaniel commented on Oct 25, 2026

rmcdaniel commented on Oct 25, 2026

rmcdaniel commented on Oct 25, 2026

rmcdaniel commented on Oct 25, 2026

rmcdaniel commented on Oct 25, 2026

rmcdaniel commented on Oct 25, 2026

rmcdaniel commented on Oct 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions