Gate durable service scheduling by worker protocol, not HTTP transactions

Problem

Workflow 2.14.x activates the protocol 1.21 durable service-operation scheduler whenever a service call has caller_workflow_run_id and the model connection has any open transaction.

That predicate does not identify workflow-task completion. DefaultServiceControlPlane::execute() also opens a transaction for any request carrying service_call_id or idempotency_key, so an ordinary HTTP Nexus execute request with caller workflow context now returns admission-only (retry_scheduled=true, wait_for=accepted) instead of honoring wait_for=completed with an inline terminal result or typed failure.

This breaks current protocol 1.20 clients. In particular, Python SDK Client.execute_nexus_operation() sends caller workflow instance/run identity for workflow-originated HTTP execution, and its contract tests require a completed response or typed HTTP 409. PHP Client::executeServiceOperation() defaults to wait_for=completed and accepts caller identity in its options.

The protocol boundary is also not enforceable in the package API: DefaultWorkflowTaskBridge::complete(string $taskId, array $commands) does not receive the completing worker's protocol version, while every start_service_operation completed through it can now record ServiceOperationScheduled. The published contract says the command remains available from 1.13 and only the scheduled semantics require 1.21.

At diagnosis, PHP, Python, and Rust SDK main branches all declared worker protocol 1.20.

Required behavior

Direct/control-plane HTTP execute requests must not enter the durable workflow-task scheduler merely because caller workflow identity and an idempotency transaction are present. wait_for=completed must retain its terminal result / typed failure contract.
Workflow-task completion must activate ServiceOperationScheduled only when the completing worker protocol is at least DURABLE_SERVICE_OPERATION_SCHEDULER_MINIMUM_PROTOCOL_VERSION (1.21).
Workers below 1.21 must retain the pre-1.21 inline start_service_operation path; they must not be rejected solely because the package supports the new scheduler.
Extend the stable completion surface compatibly so hosts can provide the authenticated worker protocol version. Existing embedded/default calls should retain the current package protocol default.
Add regression coverage for caller-context HTTP execution, protocol 1.20 inline completion/failure, protocol 1.21 scheduled admission, and no-write refusal of invalid protocol input.
Publish a patch release for Server adoption; Server must remain at protocol 1.20 until it consumes the corrected package and explicitly passes worker protocol context.

Evidence

Introduced by 2a7511501 (Schedule workflow service calls outside completion locks).
Reported release at diagnosis: af9dbadb41bf96f51ad78047022c392887e3c7a6 (2.14.1).
Server adoption tests correctly observe three regressions: sync completed and typed-failure HTTP calls with caller workflow identity now return 200 admission-only.

Current state and next action

General owns delivery. Workflow 2.14.2 is published at d60ce5f60e70b834c5d4ef142c3d5dc415cd39fe, tree 675082c4bc86810467f9abaf852868b8144c95fb. The release and Packagist source/dist references identify that exact commit.

PRs #943, #944 and the forward qualification correction #945 are merged. The correction preserves the nested non-waiting workflow-update guard inside idempotency transactions. Exact-source Security acceptance and full corrected-main build 40361556659 are successful: 5,013 unit cases / 71,655 assertions, all applicable database, quality, corpus and upgrade jobs, and combined line coverage 93.63%. The prior failed builds remain historical evidence and were not treated as release qualification.

Published release verification 40365636740 passed all 16 Laravel/PHP upgrade cells. The installed embedded Nexus receipt passed 12/12 scenarios with real worker processes, cold readers, typed failures, cancellation, interruption and external payloads, using the normally installed 2.14.2 package. The separate #899 installed acceptance passed on SQLite, MySQL and PostgreSQL, 6 tests / 7,152 assertions.

Remaining: published Server consumer acceptance. Server #791, exact head 9d7cd8fcba3dda8bcec984e7048d0ffa27566d00, implements #784 alongside the accepted PHP stream/payload repairs. It pins published2.14.2, preserves server protocol1.20, passes the authenticated version through the optional protocol-aware bridge, retains request/replay authority and the60-second bound. Product source has429 focused cases/5,123 assertions with four-engine races; final test corrections are qualified locally and hosted CI is running.

General owns the protected merge, independent main qualification, prepared PHPServer2.11.3 publication and actual installed PHP/Python Nexus callers, registry-installed Rust Nexus profiles and nested non-waiting workflow-update acceptance. The existing published Nexus workflow's current-artifact pins are prepared on test/php791-installed-acceptance; it has not been dispatched against the unpublished Server tag and will use the exact image digest after registry verification. No version-only SDK release is needed. Native2.14 schema takeover recognition remains with the Rust owner. Keep this issue open until the published Server acceptance is complete; no operator action is required.

Fix service scheduler protocol gating (#942)

Preserve completion ABI and qualify protocol scheduling (#942)

General's consumer inventory for the coordinated 2.14.2 patch, checked against GitHub main on October 26.

Consumer	Current dependency	Required follow-through
Server	Runtime Workflow pin 2.13.1; #784's isolated adoption branch was qualifying 2.14.1	#784 owns the 2.14.2 pin, optional protocol-aware completion integration, regression qualification, and published Server artifact receipt. Keep protocol 1.20 until its separate SDK/fencing upgrade.
Waterline	Workflow 2.5.12 is development-only; embedded applications supply their runtime Workflow package	#899's owner will qualify normally installed Workflow 2.14.2 with published Waterline on SQLite/MySQL/PostgreSQL. A new Waterline tag is not needed to change a development-only pin.
Sample App	Runtime Workflow constraint ^2.0; committed lock holds 2.11.3, Waterline 2.12.1, PHP SDK 3.1.0	The existing lock does not consume the 2.14.x regression. Its constraint allows 2.14.2 on update; no automatic lockfile or deployed application update is claimed. Published Laravel upgrade verification qualifies the package independently.
PHP/Python/Rust SDKs and CLI	Service clients do not install the Laravel Workflow package	No version-only SDK/CLI release. Server adoption must preserve protocol 1.20 terminal HTTP execution and inline completion; a later 1.21 capability rollout remains separate.

Publication, installed Laravel qualification, Server adoption, and #899's consumer qualification remain pending. This inventory is not an artifact acceptance receipt.

Workflow 2.14.2 is published and the package-level installed acceptance passes.

Immutable tag and Packagist source/dist reference: d60ce5f60e70b834c5d4ef142c3d5dc415cd39fe, tree 675082c4bc86810467f9abaf852868b8144c95fb. Release, registry metadata.
Full corrected-main build 40361556659 passes every applicable job. Unit suite: 5,013 cases / 71,655 assertions, zero failures, 43 skips and 15 notices. MySQL/PostgreSQL feature shards, MariaDB, source Laravel journeys, quality, corpus, CodeQL, public boundary, combined coverage and Sonar pass. Combined v2 line coverage is 93.63% (71,917 / 76,806), above the accepted 93.08% baseline.
Published verification 40365636740 passes package registry/platform identity and all 16 supported Laravel/PHP installed-upgrade cells.
Normal Composer installation of Workflow 2.14.2 with Laravel 12.70.1 passes the embedded Nexus profile below. No package source override or package edits were used.

The normal installed embedded run executed October 26, 17:15:49.319670–17:16:47.688743 UTC, on PHP 8.5.12 with real SQLite database queues. The runtime image was composer@sha256:da624c2c705d54e2f5a9bb77a5dc6c7d091711433ebefd7035672e84c2010fee, under UID/GID 1000, 2 CPUs and 2 GiB, with networking disabled during execution. The installed package's exact version, reference and vendor path are asserted by the probe.

All 12 originals pass: completion, typed failure, target cancellation, waiting-caller cancellation, backend interruption, and external payloads, each in the same namespace and across namespaces. There are 24 original runs, 12 calls, 93 distinct real queue-worker processes, a separate cold reader for each original, zero remaining runnable tasks and zero failed Laravel jobs. Each original retains its 30-second observer budget; call/execution/cleanup deadlines and identities are checked unchanged. The largest value is 114,688 bytes, above the real 1,024-byte external-storage threshold. int64, null, Unicode, external references and typed results survive cold reads without changing database bytes.

The existing embedded fixture was copied into a disposable consumer. Its historical version/reference pins were advanced, the CLI dependency/provider preparation followed its README, and the observer now checks the current two-phase durable scheduling contract: scheduled admission with no inline target, a separate dispatch worker, one ServiceOperationScheduled, no inline ServiceCallStarted, and one terminal service event. The fixture host authorizer permits only its caller class, the two fixture namespaces, and its fixed target service/operation. All package authorization checks remain unchanged.

The first attempt deliberately remains separate: the historical fixture lacked the now-required host binding and the installed default policy refused with service_operation_authority_invalid, producing one failed run and zero service calls. After adding the documented fixture-scoped host binding, all 12 cases ran on a fresh database. The observer also now exits nonzero on a failed assertion. Neither the original incomplete attempt nor its exit code is counted as acceptance.

Identities:

Consumer lock SHA-256: 1fa968bd9c27a97722c74665dff0bc955a86c27542b65b3ee3037d33983b3997.
Adapted probe SHA-256: 3e3451cd2013f5fb52450d4355ea38e6f82922848349309325bee433f5997242.
Adapted workflow/provider fixture SHA-256: eddc35bcd504e9bbc778deb0130d9786e430379ae4fe5af7a4e548d5d921dce1.

The patch below records the fixture adaptations against the runner revision linked above. The original checked-in historical profile remains unchanged. Use the profile's CLI preparation, the published package and framework versions above, and a fresh disposable database.

Server #784 retains the published Server artifact and actual PHP/Python/Rust SDK runtime qualification. Workflow #899's owner retains its normally installed Workflow + published Waterline acceptance on SQLite/MySQL/PostgreSQL. These are separate pending receipts; #942 remains open.

--- a/composer.json
+++ b/composer.json
@@ -5,11 +5,8 @@
     "license": "MIT",
     "require": {
         "php": "^8.3",
-        "durable-workflow/workflow": "2.5.19",
-        "laravel/framework": "12.69.3",
-        "laravel/octane": "2.21.0",
-        "spiral/roadrunner-cli": "^2.6.0",
-        "spiral/roadrunner-http": "^3.3.0"
+        "durable-workflow/workflow": "2.14.2",
+        "laravel/framework": "^12.69.3"
     },
     "autoload": {
         "psr-4": {"App\\": "app/"}
--- a/qualification/nexus.php
+++ b/qualification/nexus.php
@@ -36,8 +36,8 @@
     public function run(): void
     {
         $this->check(PHP_INT_SIZE === 8, 'The int64 profile requires 64-bit PHP');
-        $this->check(InstalledVersions::getPrettyVersion(self::PACKAGE_NAME) === '2.6.1', 'Wrong published Core version');
-        $this->check(InstalledVersions::getReference(self::PACKAGE_NAME) === '1f72c41fa527685d9bcc75ea86f02ebe3cc9adfb', 'Wrong published Core source');
+        $this->check(InstalledVersions::getPrettyVersion(self::PACKAGE_NAME) === '2.14.2', 'Wrong published Core version');
+        $this->check(InstalledVersions::getReference(self::PACKAGE_NAME) === 'd60ce5f60e70b834c5d4ef142c3d5dc415cd39fe', 'Wrong published Core source');
         $package = realpath(InstalledVersions::getInstallPath(self::PACKAGE_NAME));
         $this->check(is_string($package) && str_starts_with($package, base_path('vendor') . '/'), 'Product must be normally installed under vendor');
         $this->check(config('queue.default') === 'database', 'A real database queue is required');
@@ -55,7 +55,7 @@
         }
         $this->check(WorkflowRun::query()->count() === 24, 'Every original must retain exactly caller and target runs');
         $this->check(DB::table('failed_jobs')->count() === 0, 'Unexpected failed Laravel jobs');
-        echo json_encode(['outcome' => 'pass', 'version' => '2.6.1', 'source' => InstalledVersions::getReference(self::PACKAGE_NAME),
+        echo json_encode(['outcome' => 'pass', 'version' => '2.14.2', 'source' => InstalledVersions::getReference(self::PACKAGE_NAME),
             'php' => PHP_VERSION, 'framework' => app()->version(), 'composer_lock_sha256' => hash_file('sha256', base_path('composer.lock')),
             'cases' => count($cases), 'queue_processes' => count($this->processes), 'distinct_queue_processes' => count(array_unique($this->processes)),
             'runs' => WorkflowRun::query()->count(), 'calls' => WorkflowServiceCall::query()->count()], JSON_THROW_ON_ERROR), "\n";
@@ -88,6 +88,13 @@
         $processStart = count($this->processes);
         $this->worker($deadline);
         $call = WorkflowServiceCall::query()->where('caller_workflow_run_id', $originalRun)->sole();
+        $this->check($call->status === 'accepted' && $caller->refresh()->run()->status === RunStatus::Waiting && $caller->output() === null,
+            'A scheduled completion-waiting caller must remain waiting at admission');
+        $this->check(\Workflow\V2\Models\WorkflowServiceCallDispatch::query()->where('service_call_id', $call->id)->where('state', 'scheduled')->count() === 1,
+            'Admission must persist one deferred service dispatch');
+        $this->check($call->linked_workflow_run_id === null, 'Admission must not start the target inline');
+        $this->worker($deadline);
+        $call->refresh();
         $this->check($call->status === 'started' && $caller->refresh()->run()->status === RunStatus::Waiting && $caller->output() === null,
             'A completion-waiting caller must not complete at admission');
         $callDeadline = $call->getRawOriginal('call_deadline_at');
@@ -127,7 +134,8 @@
                 && $cleanup['cleanup_deadline_at'] === $root->deadline()->toISOString(), 'Target cleanup lost the original root/deadline');
         }
         $events = $caller->run()->historyEvents()->get();
-        $this->check($events->filter(static fn ($event) => $event->event_type === HistoryEventType::ServiceCallStarted)->count() === 1, 'Expected one original service start');
+        $this->check($events->filter(static fn ($event) => $event->event_type === HistoryEventType::ServiceOperationScheduled)->count() === 1, 'Expected one original durable service schedule');
+        $this->check($events->filter(static fn ($event) => $event->event_type === HistoryEventType::ServiceCallStarted)->count() === 0, 'Scheduled dispatch must not also record an inline service start');
         $terminal = $events->filter(static fn ($event) => in_array($event->event_type, [HistoryEventType::ServiceCallCompleted, HistoryEventType::ServiceCallFailed, HistoryEventType::ServiceCallCancelled], true));
         $this->check($terminal->count() === 1, 'Expected exactly one terminal service event');
         $this->check(WorkflowTask::query()->whereIn('workflow_run_id', [$originalRun, $target->runId()])->whereIn('status', ['ready', 'leased'])->count() === 0,
@@ -227,6 +235,7 @@
     }
 }
 
+try {
 if (($argv[1] ?? null) === '--inspect') {
     $workflow = WorkflowStub::loadRun($argv[2]);
     echo json_encode(['run_id' => $workflow->runId(), 'status' => $workflow->run()->status->value,
@@ -234,3 +243,7 @@
 } else {
     (new PublishedEmbeddedNexusProbe())->run();
 }
+} catch (Throwable $failure) {
+    fwrite(STDERR, $failure::class.': '.$failure->getMessage()."\n");
+    exit(1);
+}
--- a/qualification/nexus-workflows.php
+++ b/qualification/nexus-workflows.php
@@ -23,6 +23,21 @@
 {
     public function register(): void
     {
+        $this->app->bind(\Workflow\V2\Contracts\WorkflowServiceOperationAuthorizer::class, static fn () => new class implements \Workflow\V2\Contracts\WorkflowServiceOperationAuthorizer {
+            public function authorize(\Workflow\V2\Models\WorkflowRun $run, array $command): \Workflow\V2\Support\ServiceCallPrincipal
+            {
+                if ($run->workflow_class !== PublishedNexusCaller::class
+                    || ! in_array($run->namespace, ['shared', 'tenant-a'], true)
+                    || ($command['namespace'] ?? $run->namespace) !== 'shared'
+                    || ($command['endpoint_name'] ?? null) !== 'embedded-preview'
+                    || ($command['service_name'] ?? null) !== 'greeter'
+                    || ($command['operation_name'] ?? null) !== 'run') {
+                    throw new \Workflow\V2\Exceptions\ServiceOperationAuthorizationException();
+                }
+
+                return new \Workflow\V2\Support\ServiceCallPrincipal(subject: 'workflow:'.$run->id, method: 'workflow');
+            }
+        });
         $this->app->singleton(ExternalPayloadStoragePolicy::class, static fn () => new class implements ExternalPayloadStoragePolicy {
             public function driverFor(?string $namespace): ?ExternalPayloadStorageDriver
             {

The first exact-published-tuple Nexus acceptance run 40418210934 finished failed. Server 2.11.3 / Core 2.14.2 / PHP SDK 3.5.1 / Python SDK 2.11.2 / CLI 2.3.0 / Waterline 2.13.0 were used; the image was pinned to published index sha256:923a8924ecad13485562d7af30dc87d7fd6d9d7d4758ec8113bac0766fa8603e.

The hosted check annotations establish:

All eight uncaught typed-failure caller/service/namespace observations qualify.
All four Python-caller authorization observations qualify. Four PHP-caller authorization observations time out waiting for the caller workflow after 30 seconds; the direct admin refusal remains HTTP 403 in every case.
All eight external recovery observations fail to observe the original call plus waiting linked service before the kill checkpoint. They do not prove recovery or reference access. A missing WorkflowFailed summary is insufficient to diagnose why.

The acceptance assertions and original budgets remain intact. Diagnostic run 40420956902 repeats the same published tuple at fixture head 1e12cdbc33d4d1ec0a32362088e5b9c8a53b3c8f, adding only bounded last-observed event types and worker diagnostics. The existing 13 evidence-reader checks and PHP/Python syntax checks pass. This distinguishes an obsolete fixture event expectation from an actual delivery defect before changing either behavior or the qualification contract.

Installed consumer acceptance remains open. The package/source lifecycle fixes in Server #794 / Workflow #947 remain unreleased and are not claimed by this published-image result. Release/image/chart publication itself passed; this result does not establish completion of the remaining acceptance work.

Navigation Menu

Repository navigation

Gate durable service scheduling by worker protocol, not HTTP transactions #942

Problem

Required behavior

Evidence

Current state and next action

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Gate durable service scheduling by worker protocol, not HTTP transactions #942

Description

Problem

Required behavior

Evidence

Current state and next action

Activity

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions