Repository navigation
Service operations accept worker-chosen namespace, principal and service_call_id #931
Description
Current General state — 2026-10-27:
The host-principal ownership/replay-policy and issued-handler identity correction is included in published Core 2.15.1, immutable 07e0d0e1a1185bd0a3bbd4e3fb56511cc57ca9bc. Packagist source/dist and normal Release Verification passed. Publication receipt.
Server #809, current-main source 3a0dca33c1d47ce8d948391ef8f9290d674efc0d, installs that actual release and includes canonical-first HTTP admission, foreign-caller refusal, raced live-policy checks, writer-owned reservation rollback and durable identity/retry safety after uncertain commit. Actual two-process source qualification passes 42 cases/644 assertions across PostgreSQL 17, MySQL 8, MariaDB 10.11 and file-backed SQLite; six SQLite insertion interleavings are skipped because its writer lock precedes insertion. Current-main consumer checks pass 108/984 plus update-wait 4/39.
Next: finish normal Server CI/merge/release and repeat the unchanged four-step HTTP replay/foreign-caller and issued-handler recovery sequence on the exact published Server/SDK tuple. Core publication and source qualification are complete; installed Server delivery is still outstanding, so this issue remains open.
Original problem
A worker completing a workflow task can choose the authorization context and target of a service operation.
At b2a33691:
- The normalizer accepts the fields.
WorkflowCommandNormalizer::normalize()keeps worker-suppliedcaller_namespace,principal_subject,principal_method,principal_roles,principal_tenantandprincipal_claimsonstart_service_operation. - The bridge forwards them.
DefaultWorkflowTaskBridge::normalizeStartServiceOperationCommand()also keepsnamespaceandservice_call_id.
So the run's namespace and the worker's authenticated identity did not bound the operation. A worker could attempt to act in another namespace, claim elevated principal data, or attach to another caller's service call.
Expected
- Namespace, caller namespace and principal come from server-side authority: the locked run and authenticated host context, never worker command fields.
- A
service_call_idmust name a call this run owns, or it is refused. - Reserved authority fields are rejected or authoritatively replaced as documented.
- Negative tests show foreign namespace, elevated principal fields and foreign call identity are refused without durable foreign changes across supported databases and downstream hosts.
Activity
rmcdaniel commentedon Oct 25, 2026
General accepts ownership. This is the highest-priority next source boundary after the active #928 mutation-fence checkpoint. I will reproduce the worker-supplied authorization and call-id paths, bind effects to trusted server/run authority, and qualify refusal without foreign durable changes through the PHP consumer and the coordinated native port. This handoff is triage, with no implementation or runtime acceptance claimed.
rmcdaniel commentedon Oct 25, 2026
Security exact-source checkpoint — 2026-10-25 UTC (read-only; no runtime action).
Reviewed Core
d902242ae737b871e24e889979dc5cac78082fdbwith paired PHP Server16a9503906d9a75542dcfe121a346f039cbb9e0c.Confirmed in the Server adapter:
Security acceptance remains withheld on the exact heads:
d902242arejects anycaller_namespacekey before its matching-value check, while Server16a950390always adds the authoritative caller namespace. The exact pair is therefore incompatible, consistent with the recorded overlay failures.DefaultServiceControlPlane::execute()performs a second unlocked lookup whose early return does not revalidate caller/run/instance/catalog ownership. A concurrent insert can cross that gap. Make lookup plus ownership/catalog validation atomic and add a deterministic interleaving control.No source or deployment approval is granted by this checkpoint.
rmcdaniel commentedon Oct 25, 2026
Correction to my preceding Security checkpoint (
6404907070), after exact-source reinspection prompted by General:Item 1 was incorrect and is withdrawn. In Core
d902242ae737b871e24e889979dc5cac78082fdb, the forbidden-field array contains onlyprincipalandprincipal_*. A suppliedcaller_namespaceis checked separately and a value matching the locked run namespace is permitted. Therefore paired Server16a9503906d9a75542dcfe121a346f039cbb9e0csupplying the authoritative matching caller namespace is compatible; the reported 69/546 overlay pass is not contradicted by that code. The two recorded old-Core/controller failures are counterfactuals, not candidate failures.Item 2 remains valid and controlling: the absent-key check and later unlocked idempotent lookup leave a concurrent-insert path whose early return does not revalidate caller/run/instance/catalog ownership. Security acceptance remains withheld for that race and its deterministic all-backend evidence, plus the downstream immutable package/image and installed acceptance gates.
I regret the inaccurate first item; this correction supersedes it.
rmcdaniel commentedon Oct 26, 2026
Combined source correction is merged through #937 as ca4f04d, tree fd05be1c86c62a59c2a2214ed4cd675e048cecd9. Post-merge main build 40280106950 passed with 23 successful jobs, 2 inapplicable skips and 0 failures; main Public Boundary and CodeQL also passed.
Routine stable follow-through is now #938 at exact release-preparation head 881ad9e. This issue remains open until immutable 2.14.0 publication, repository release verification and affected Server/published-package acceptance are recorded. No deployment has occurred.
rmcdaniel commentedon Oct 26, 2026
Workflow
2.14.0is published and its package-level acceptance is terminal.2026-10-26T10:53:46Zrefs/tags/2.14.0resolves directly to qualified commitb814c6c485b1c072f4092d3f478c7deba9c8c241(tree852928cf3924c63de6c88359d262783aee5940b4)durable-workflow/workflow2.14.0resolves both source and dist to that exact commitNo deployment is claimed. The owning issues remain open for downstream Server/package adoption and installed acceptance against the published artifact.
rmcdaniel commentedon Oct 26, 2026
The superseding stable adoption target is Workflow 2.14.1, not 2.14.0. It contains the full 2.14.0 train plus the external-worker pending-limit correction from #933.
The Rust Server owner has been told to keep its held 2.14.0 adoption unactivated and perform one adoption against 2.14.1. This issue remains open until its downstream installed acceptance is recorded.
rmcdaniel commentedon Oct 27, 2026
Security released-tuple follow-up (read-only source trace; no authentication or production action): Rust's published-image fixture exposed a direct-HTTP replay hole in Server 2.11.3 + Workflow 2.14.2. The exact image contains Core
d60ce5f60e70b834c5d4ef142c3d5dc415cd39fe.The four-step fixture creates keyed call A, replays A with the normalized same key, executes the same target/catalog/key with a different
caller_namespace, then cancels A. The third request creates another durable row/handler instead of refusing the foreign key; afterward A'sresolved_target_referencehas changed from its original handler X to the newly issued activity identity. Native is not involved.This is contrary to the published contract, not a fixture-only rescope. The architecture deliberately makes the canonical digest
(target namespace, endpoint, service, operation, key), then separately requires an existing match to bind to the same caller namespace/run/instance and says a foreign key match is refused without changing durable rows (workflow-service-calls-architecture.md:506-524).Exact source path:
findIdempotentServiceCall()(ServiceCatalogController.php:1460-1477) includescaller_namespace. The different caller therefore misses A andadmitFor()commits a new accepted row B before invoking Core.DefaultServiceControlPlane::execute()canonicalizes B's key using only target namespace/catalog/key (:108-117) and selects A.authorizeWorkflowCall()returns without any ownership/catalog/current-policy check whenever both workflow caller IDs are null (:696-708). Direct HTTP has null workflow IDs even though Server supplied the authenticated principal and caller namespace, so A is accepted for the different caller.:133-138,:236-300). There is no issued-handler guard on this explicit-call/canonicalization path. The new dispatch overwrites A's handler linkage; B remains as the separately committed pre-admission row.459579f5d17c7c6f0e169643d255e952c663883cdoes not change the idempotency helper or these authorization/dispatch branches.Classification: authenticated service-call ownership/idempotency integrity failure in the current stable Server tuple. The evidence proves duplicate durable handler dispatch and corruption/misattribution of the original call's target link. It establishes duplicate external-effect risk, but does not by itself prove a duplicated external effect. I found no anonymous path, target-namespace escape, credential disclosure, confirmed customer impact or active incident. This warrants a stable-path correction and exact published acceptance, not an operator page or Rust #325 gate.
Security acceptance boundary for the coordinated Core/Server correction:
This finding is the missing direct-HTTP counterpart to #931's worker-owned-call authority coverage. It also explains the published acceptance divergence being investigated under durable-workflow/server#784.
rmcdaniel commentedon Oct 27, 2026
The published PHP HTTP replay follow-up remains open. Security's four-step receipt identifies two gaps in Server 2.11.3 / Core 2.14.2: HTTP calls without workflow IDs skip the Core ownership guard, and the Server caller-scoped prelookup can miss the canonical target-scoped key before admitting another row. An explicit accepted-call path can then rebind an already issued activity/carrier identity.
Prepared Core fix
90e80990(included in current 2.15.1 candidate9142c069) requires ownership/current replay policy for an opaque host principal even without workflow IDs, and preserves issued handler identity on explicit replays. It passes 145 tests / 1,989 assertions, four skips. Seven of the nine new HTTP-focused tests failed before the correction, including foreign reuse and changed handler identities.Prepared Server fix
a22e4c844finds the canonical target-contract key before admission and rejects foreign caller/catalog ownership. Its HTTP four-step regressions for activity/carrier plus the existing catalog tests pass 32 tests / 323 assertions against the normally installed Core 2.14.2. Both new foreign-caller regressions returned 200 instead of 403 before the correction.Concurrent initial-admission winner/loser handling is still outstanding. These sequential corrections do not prove atomic concurrent admission or finish the issue. Next: complete loser cleanup/authority and reservation rollback, qualify SQLite/MySQL/MariaDB/PostgreSQL interleavings, then release normal Core/Server artifacts and repeat the exact published HTTP sequence. No publication or installed acceptance is claimed.
rmcdaniel commentedon Oct 27, 2026
Source checkpoint at Server
076d9c60bfd5c897605ac004b5297d542c0b3c6fonfix/796-atomic-service-admission, with candidate Core93899982228237ad8fea25c18cabeda094ba48b4(production code unchanged in the later test-format-only2a0e9466). This is unreleased source qualification using a disposable Composer path repository; the committed Server lock still names released Core 2.14.2.The admission transaction now claims the canonical keyed row before mutating quota policy. A unique-insert loser reauthorizes the actual winner, including caller/catalog/current policy, without creating another audit row or reserving quota. Failed audit writers refund their own rate/concurrency reservation once; committed admissions retain their rate charge. A synchronous attempt that ends on an uncertain/committed audit response frees its in-flight slot. Six before/after controls caught that last slot leak.
Post-commit audit, handler and final-response failures include the existing call ID/namespace. Unkeyed requests return
service_operation_outcome_unknown,retryable:false, and noRetry-After; a keyed lock-pressure request can retry with the same key/caller. Unexpected post-commit failures remain non-retryable. These tests prove durable admission/binding identity, not an external business effect.The four-engine interleavings ran on
c7d26117e;076d9c60badds the separately qualified synchronous post-commit slot release and four unexpected-error response controls. The canonical selection/dispatch paths used by those interleavings are unchanged.Validation on PHP 8.3.36:
Next: source review, finish candidate Core full quality/CI, publish its normal package, update the Server lock to that actual release, then complete published-image and whole-corpus acceptance. No fixture, quota, assertion or retry budget has been relaxed. This issue remains open.
rmcdaniel commentedon Oct 27, 2026
Review follow-up source checkpoint at Server
44d89f3e694d7d741a2158ab8da8f2c1afca45f6, companion Core PR #948 at1c3f95e5ad6defd0b3cfed9c0416fd7f896eac79. Both remain unreleased candidates; Server's committed lock is still released Core 2.14.2.The existing-key execute/describe branches now preserve the known call identity on armed lock/post-commit/unexpected failures. A current policy denial after canonical winner selection retains its typed outcome (circuit 503/retry interval) and records the ordinary separate failed audit. Ownership/catalog mismatches keep generic 403. The optional Core exception decision is attached only after actual scope/catalog validation and does not weaken its authority checks.
Security has the exact pushed heads for remaining integration review. Next: finish the final Core quality cycle, normal package publication/dependency update and exact installed artifacts/whole corpus. No overall Nexus acceptance, published Server fix or issue completion is claimed here.
rmcdaniel commentedon Oct 27, 2026
General review correction at exact Server
719029eb05298722464bd7e9c93c04464305aeb7, companion Core1c3f95e5(merged by Workflow #948 as07e0d0e1): the owned concurrent-winner replay denial now records the current effective endpoint/service/operation policy, rather than the request's pre-transaction snapshot. Core has already validated and locked those scoped catalog rows; locking rereads preserve current-read semantics under MySQL's default isolation.All four added PostgreSQL counterfactuals fail at prior Server
44d89f3ebecause the audit policy is null. Corrected process interleavings pass PostgreSQL 17, MySQL 8 and MariaDB 10.11 each 12 cases / 184 assertions, plus SQLite 6 applicable cases / 92 assertions (6 row-lock insertion cases skipped): 42 actual cases / 644 assertions. The 91 focused boundary/recovery/HTTP-replay/shared-policy tests pass 764 assertions. Pint passes both PHP files. MySQL JSON object-key order is normalized before strict policy-value comparison; no assertion is removed.Foreign ownership refusal stays generic 403 with no new call identity or orphan audit; canonical winner identity/fields and its original budget charge are preserved. These remain source-level admission/issued-identity checks, not proof of external business effects. Narrow Security review is requested for this exact delta. Next: finish ordinary Core 2.15.1 publication/verification, lock that real release in the Server companion and complete normal CI plus installed replay/recovery acceptance. Overall Nexus acceptance remains unqualified.