Skip to content

Bind workflow service calls to trusted host authority - #936

Merged
rmcdaniel merged 4 commits into
mainfrom
fix/service-operation-authority
Oct 26, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
fix/service-operation-authority

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Summary

A worker could supply principal claims or another caller's service-call ID/key to Core. Require a trusted host authorizer, derive caller identity from the locked run, and recheck the actual call's caller/catalog/current policy under the completion transaction before replay or dispatch. Check retained prefixes before cached receipts and refuse unauthorized turns atomically.

Serialize keyed-call creation with a nullable canonical scope digest and unique index. A competing insertion is handled through a savepoint and locked, fully authorized winner refetch. Integrate current main's history meter and lifecycle behavior. Refs #931; final exact-source evidence is in the qualification receipt on this PR.

Verification

  • Same 75 authority cases on SQLite (409 assertions, four row-lock cases skipped), MySQL 8.0.46 and PostgreSQL 17.11 (435 assertions each, SQLite-only case skipped): zero failures/errors. Real two-process foreign/same-owner insertion interleavings, SQLite uniqueness conflict/refetch, migration retry and namespace/key boundaries are covered.
  • The final-helper-lookup PostgreSQL control fails against the previous control plane: one assertion failure, zero errors, foreign call accepted with sibling writes. The candidate refuses it with the recorded tables unchanged.
  • PHP Server source overlay at 16a9503906d9a75542dcfe121a346f039cbb9e0c: 102 tests / 845 assertions, no failures/errors/skips, 11 PHPUnit deprecations.
  • Full ECS and PHPStan pass. Full exact-head unit passes 4,934 / 70,545; coverage passes 4,934 / 70,661 with v2 line coverage 76.61%.
  • Full PostgreSQL feature contract passes 2,496 / 24,374 with 18 skips. Exact-head hosted build 40251605455, retained regression corpus, CodeQL, Public Boundary and SonarCloud are terminal green.
  • Host, policy-hook, schema expansion and writer rollout documentation updated.

Independent Security accepted the exact production source. #932's durable wait/backoff scheduler, native-port qualification and published-package consumer acceptance remain open. This PR stays draft.

Compatibility

Hosts must bind WorkflowServiceOperationAuthorizer before accepting service-operation work; the default refuses it. RunServiceOperationAuthorizer explicitly permits same-namespace embedded calls with an unprivileged workload principal. Authenticated hosts supply the current request principal and authorize target scope. Custom boundary policies must provide pure authorizeReplay(ServiceBoundaryRequest): ServiceBoundaryDecision without reserving fresh admission capacity.

Apply the nullable idempotency_scope_hash schema expansion and drain old writers before relying on concurrent deduplication. Existing candidates are adopted only after ownership/catalog/current-policy checks; there is no bulk backfill or ownership reassignment. Run/call/catalog models must share the completion database connection or the turn is refused before writes.

The prepared PHP Server adapter still needs the published Core dependency before Server merge/release. No package, image, production migration, deployment or installed acceptance is delivered by this source PR.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact-source review handoff — 2026-10-25 22:43 UTC. Core d902242ae737b871e24e889979dc5cac78082fdb, tree 091bcee2324a9622e07e21726f68406c2743064c; prepared PHP Server 16a9503906d9a75542dcfe121a346f039cbb9e0c, integrated with Server main. Both remain source only. Security acceptance is requested, not claimed.

Core now requires an explicitly configured WorkflowServiceOperationAuthorizer. The default refuses service operations. The provided opt-in embedded policy supplies an unprivileged run principal and permits only the run namespace. PHP Server's request-bound adapter supplies its authenticated principal and authorizes requested target scope; command strings no longer carry trusted identity. Caller namespace/instance/run come from the locked run, including explicit embedded null namespaces.

Before applying sibling commands or returning a cached prefix receipt, Core checks existing call IDs and key matches under locks against caller namespace/run/instance, target namespace, catalog names and actual catalog IDs/relationships. The effective default key is checked too. Request payload externalization follows authorization. PHP Server retains its current-policy replay check, now holding the call/catalog rows through replay and checking exact instance/catalog ownership. Its package floor refuses an old Core without this contract.

Qualification Cases Assertions Failures / errors / skips
Authority controls, each of SQLite/MySQL 8.0.46/PostgreSQL 17.11 51 267 0 / 0 / 0
Broader Core service consumers 238 9,751 0 / 0 / 0
Same 51 controls, three preceding Core effect-path files from b2a33691 51 172 46 assertion failures / 0 errors / 0 skips, five valid controls pass
PHP Server worker and package-floor tests on combined source, with the Core source overlay 69 546 0 / 0 / 0
PHP Server worker subset 37 367 0 / 0 / 0
Two caller-instance controls with preceding Core/controller source 2 14 2 assertion failures / 0 errors / 0 skips

The three database reports and the Core counterfactual have identical sorted test identities (SHA-256 ed28e46de36363bd55eaede3fefbcfc869110f289eb027c65d36b64e49610d6f). The controls include forged principal fields, foreign namespace/caller, pending/started/completed foreign calls, wrong caller run/instance, catalog names/IDs, explicit and derived key collisions, prefix sibling atomicity, duplicate receipt authorization, null namespace and valid owned/resolution-failure replay. Server tests preserve authenticated principal attribution, authorized cross-namespace calls and current-policy replay refusal. Server PHPUnit reports 11 deprecations; no clean-deprecation claim is made.

Full ECS/PHPStan passed before the final corpus-only addition; the changed corpus consumers were style-checked and executed by the official policy. The policy passes at exact d902242a: 73 original replay fixtures retained unchanged, 74 current, one new qualifying fixture and one enforced old-source counterfactual. That fixture retains cold replay and also exercises the real public worker bridge to prove forged authority cannot commit a sibling timer. The test host explicitly supplies narrow authority for its legitimate Nexus scenarios, including when the same consumer harness runs against the preceding base. Earlier standalone retained consumers passed 74 tests / 3,740 assertions before this final fixture addition. Hosted repository-contract and regression-corpus jobs are green; remaining hosted gates are running.

Still required: exact-source Security review, full unit/coverage/feature qualification, source integration with other pending changes, native-port qualification, immutable Core publication, then the real PHP Server dependency/image and installed acceptance. The prepared Server branch still pins the old published Core and cannot merge/release until that dependency is updated. No package/image or native/shared/frozen fixture is published or changed by this handoff. #931 remains open.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-source review blocker on d902242, tree 091bcee2324a9622e07e21726f68406c2743064c:

The caller-bound check and the effecting idempotency lookup are not one atomic decision when the key is initially absent. WorkflowServiceOperationAuthority::checkReference() performs a SELECT FOR UPDATE and returns when no key match exists (lines 62-78). DefaultServiceControlPlane::execute() then performs a second, unlocked idempotentCall() query and immediately returns any row it finds (lines 64-73, 580-603). The schema has only a non-unique idempotency-key index.

On PostgreSQL READ COMMITTED, for example, another run can insert and commit the same caller-controlled key and target tuple after the no-row authority check but before idempotentCall(). The second query can then return that foreign call without rechecking caller namespace, workflow instance, workflow run, or current catalog ownership. The outer run lock does not serialize different caller runs. This can attach another run service-call result and identifiers to the current run history. The Server replay precheck has the same no-row window and does not close the Core lookup race.

Please make key lookup, collision serialization, ownership/current-catalog validation, and the replay decision a single portable transaction invariant across SQLite, MySQL, and PostgreSQL. A second ownership check alone prevents the foreign return but does not make concurrent same-key creation deterministic; use a stable serialization/uniqueness mechanism and handle the loser by locked refetch plus full caller/catalog validation. Add a deterministic two-transaction control that pauses after the initial no-match, commits a foreign same-key row, then proves the first completion refuses with no sibling/history/task writes. Also retain a valid concurrent same-owner retry control.

The request-bound principal, forged-field refusal, existing-row caller/catalog checks, and duplicate-checkpoint reauthorization otherwise look consistent in this exact source. Security acceptance remains withheld pending a new exact head and the race control.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Source successor for #931 / #936: 86290e8, tree 95e734592b3eb16929db8a538ee00dc5ef9ccde3, integrating main 4eb49a2. Prepared PHP Server remains 16a9503906d9a75542dcfe121a346f039cbb9e0c. These are unreleased source overlays, not installed published-package acceptance.

The real absent-row race from Security's review now has a database uniqueness guard: nullable canonical SHA-256 scope digest, unique index, savepoint on creation/adoption, locked winner refetch, and caller/catalog/current-policy checks on the actual row before every workflow replay or dispatch. Explicit-ID paths obey the same key. Existing legacy candidates must pass authorization before adoption; there is no bulk backfill or ownership reassignment. Null namespace and opaque key bytes remain distinct. Separately connected service/catalog models fail closed before sibling writes because they cannot share the completion rollback.

Cached checkpoint replay now uses a pure current-policy hook, including catalog replacement/deletion and required-role changes. It does not reserve new rate or concurrency capacity. Hosts with custom policies must implement authorizeReplay; the docs describe both host bindings and schema/writer rollout.

Measured source checks, 2026-10-25 23:57–23:59 UTC, PHP 8.3.36:

Authority controls Cases Assertions Failures/errors Skips
SQLite 75 409 0 / 0 4 row-lock interleavings
MySQL 8.0.46 75 435 0 / 0 1 SQLite-only control
PostgreSQL 17.11 75 435 0 / 0 1 SQLite-only control

All three execute the same sorted case inventory, SHA-256 880e3c9795c24a2f77f67bb9ab90818689798c1b986a5322e65c619ad2035494. MySQL/PostgreSQL use actual separate processes and READ COMMITTED transactions: pause after the final helper no-match lookup, commit a competing legacy call, then resume the effecting control-plane lookup; also pause before insertion and let another canonical insertion win. Foreign callers are refused with all 11 observed tables unchanged; same-owner controls reuse one call. SQLite separately proves the real two-process uniqueness loser refetches the canonical row through a usable savepoint/transaction, plus foreign-winner authorization refusal. Four all-database cases exercise migration retry/rollback retention, legacy adoption, null/empty namespaces and 191-character ASCII/Unicode keys.

Counterfactual: overlay only DefaultServiceControlPlane.php from d902242, keeping the successor's test infrastructure and helper. The final-lookup PostgreSQL foreign-insert control produces 1 test / 7 assertions / 1 assertion failure / 0 errors: completed=true, with sibling timer/service history/tasks accepted. The first version of the harness paused at the earlier whole-turn preflight and did not reproduce the defect; only the corrected final-lookup evidence above supports this claim.

Combined PHP Server source overlay: package-floor, worker service, execution routes and boundary-policy suites 102 tests / 845 assertions / 0 failures/errors/skips, with 11 PHPUnit deprecations. Full ECS and PHPStan passed; the SQLite/new-connection test additions were styled and included in the final PHPStan run. The two Nexus continuation controls that failed hosted d902 CI now bind explicit trusted host authority; the earlier combined smoke passed 63 / 543.

Still open: exact-head hosted checks, full local unit/coverage/feature runs (unit and coverage now running independently), retained-corpus validation against merged main, independent Security review, published Core/Server acceptance, and native-port qualification. #932's blocking wait/retry paths remain a merge/release concern: the standalone keyed-call transaction must not turn waits/backoff into long-held locks. No merge, tag, production migration, package/image publication or deployment is claimed. Task-owned diagnostics remain only until the successor review and qualification findings are recorded, then will be removed with the disposable resources.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-head review — 2026-10-26 UTC (source and test evidence only; no release or runtime action).

Reviewed PR head 86290e81af2b8541683cd1095340a3f2d61dfd48, tree 95e734592b3eb16929db8a538ee00dc5ef9ccde3, against its merged-main parent 4eb49a20290d48654085696326fc4fe3ad2b19af, plus the paired PHP Server source 16a9503906d9a75542dcfe121a346f039cbb9e0c.

The recorded #931 authority and concurrent-key blockers are closed in this immutable source:

  • workflow commands cannot supply principal authority, a foreign caller namespace, or a foreign target under the provided run policy; absent host authority fails closed before sibling writes;
  • explicit call IDs and idempotency-key matches are bound to the locked caller run and instance, caller/target namespace, exact endpoint/service/operation, catalog relationships, and current replay policy;
  • the effecting row is selected under the completion transaction and reauthorized after lookup, legacy adoption, or a uniqueness conflict, closing the earlier absent-key/final-lookup insertion gap;
  • keyed calls use a unique canonical digest of the complete typed scope, while the full tuple is still compared after lookup; nullable legacy rows are adopted only after authorization, without an unsafe bulk backfill;
  • the uniqueness loser uses a savepoint and locked canonical refetch, keeping PostgreSQL transactions usable; explicit and generated keys share the invariant;
  • run, call, and catalog models must share the completion connection, otherwise the whole turn is refused.

The exact 75-case authority inventory passed on SQLite (409 assertions, four row-lock-only skips), MySQL and PostgreSQL (435 assertions and one SQLite-only skip each), with zero failures/errors. The old d902242a control-plane overlay reproduces the final-helper-lookup counterexample as the intended assertion failure, while the candidate refuses it. The separate real SQLite two-process uniqueness loser, legacy/null/191-character Unicode migration-retry cases, ECS and PHPStan passed. Paired Server 16a95039 passed 102 tests / 845 assertions against this exact Core overlay (11 PHPUnit deprecations, no failures).

Exact-head hosted build 40247142255 is terminal success, including repository policy/contracts, quality, regression corpus, MySQL PR feature selection and the full PR unit-contract job. SonarCloud/CodeQL/public-boundary checks also passed.

No new #931 source blocker was found. This accepts the immutable Core source and recorded authority/race evidence for #931 only. It does not approve merge, release, schema execution, deployment, or installed behavior: #932's pre-existing in-transaction retry/wait lock duration remains an explicit merge/release concern, and rollout still requires nullable expansion before upgraded writers, old-writer drain, published Core, the paired Server dependency update, native parity, and installed acceptance.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact-successor qualification receipt — 2026-10-26 01:58 UTC

  • Source: fce3ac4b4448fc6b3835d338725a0d37d29d2af3, tree 668c0f73abe5c801cc309dec75e4cbfb17de2f2f. Production source remains identical to Security-accepted parent 86290e81af2b8541683cd1095340a3f2d61dfd48; the successor changes only two trusted-host test bindings.
  • Full unit: 4,934 tests / 70,545 assertions; 43 skips, 15 PHPUnit notices; zero failures/errors.
  • Coverage: 4,934 tests / 70,661 assertions; v2 line coverage 76.61% against the accepted 60% floor.
  • Full feature contract on PostgreSQL 17.11: 2,496 tests / 24,374 assertions; 18 skips; zero failures/errors. The earlier forced-SQLite diagnostic was not counted as feature qualification; its three environment-dependent controls pass together on PostgreSQL (148 assertions).
  • Exact-head hosted build 40251605455 is terminal green, including repository contracts, retained regression corpus, quality, tracing, PR unit-contract and MySQL feature lanes. CodeQL, Public Boundary and SonarCloud are green.
  • The cross-backend authority inventory remains green: SQLite 75 cases / 409 assertions (four row-lock skips), MySQL and PostgreSQL 75 / 435 each (one SQLite-only skip each). The old-controller PostgreSQL counterfactual still fails exactly one assertion with sibling writes unchanged; the candidate refuses the foreign call atomically.

This closes the exact-head Core source qualification gap recorded here. It does not close #931: #932's required durable wait/backoff scheduler, native parity, published Core/Server artifacts and installed consumer acceptance remain open. PR #936 stays draft.

@rmcdaniel
rmcdaniel merged commit fce3ac4 into main Oct 26, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant