Skip to content

[Epic]: Durable Functions on LMI #798

Open
14 of 23 issues completed
Open

Description

@zhongkechen

Objective

Complete the agreed SDK changes, resolve ownership and supported-lifecycle decisions, and validate Durable Functions execution and observability on Lambda Managed Instances (LMI). Keep work and telemetry attributable to the correct invocation while preserving checkpoint/replay behavior and healthy concurrent invocations.

Status checked 2026-11-12 using linked issues, PRs, reviews, and existing CI records, with main-branch code review for Java, JavaScript/TypeScript, Python, and shared conformance. The Rust entry summarizes its linked tracker; it is not a new code or deployment audit. Implementation merges, closed issues, local tests, and deployed LMI acceptance are recorded separately below. No new LMI deployment was run for this status review.

The epic has 24 direct sub-issues: 14 open and 10 closed. This includes active implementation and validation work, plus closed lifecycle and documentation tasks. Java #839 and the JS disposal/deadline/guidance issues are directly attached, along with the Rust work below.

Runtime boundaries

LMI environments remain active between invocations, and an invocation timeout does not forcibly terminate the function's code. Already-started work may retain capacity or produce effects after the timeout.

  • JavaScript/TypeScript: asynchronous invocations can overlap inside a Node.js worker; continuations and shared resources must retain invocation ownership.
  • Java: runtime threads share a JVM and handler object; task lifetime, thread-local scopes, and executor capacity require coordination.
  • Python: the runtime uses separate worker processes, each serving one request at a time; blocked invocation-owned threads can still retain a worker.

Native invocation concurrency is distinct from SDK map/parallel concurrency. An invocation is one handler attempt; a durable execution may span multiple invocations. Parent completion, handler result, wrapper return, PENDING, and platform timeout are distinct boundaries. Remaining designs must establish which layer owns a behavior and what guarantee is possible before prescribing a cross-SDK cancellation mechanism.

Remaining implementation and design work

Area Tracker Current scope and implementation status
JS checkpoint disposal and step-signal cleanup JS #927; #1101 Both remain open. Current main still admits checkpoints after disposal and does not abort unawaited step work on normal handler return/failure. #1114 remains open; its current proposal closes checkpoint admission, abandons queued work on disposal, and adds disposal-time step-signal cleanup while preserving in-flight request completion.
JS deadline termination and operation admission JS #927; #1102 Both remain open. #1202 now proposes deadline-driven termination, cooperative step abort, and operation/serialization admission gates. It reports real-LMI probe results, but is not merged. It does not establish a bound on in-flight checkpoint waiting or queued checkpoint draining.
JS telemetry after stop or execution timeout JS #1127 The issue remains open. #1202 proposes documentation clarifying that RETRYING does not guarantee a later invocation and that plugins must end remaining operations in onInvocationEnd. Its deadline probe does not validate the separate stop/execution-timeout report.
Java deadline admission and cooperative cancellation Java #838 #733 remains open. Current main records a deadline but does not enforce it at operation/checkpoint admission. Admission fencing, cancellation, and result/failure correctness belong here; this does not establish hard caller-return or worker-reclamation bounds.
Java cleanup bounds and worker recovery Java #726 Current main can wait indefinitely for actual task exit or checkpoint transport shutdown. Agree SDK/runtime ownership, achievable budgets, and a supported recovery policy for work that outlives cancellation, then implement that contract while preserving actual owner/task cleanup and owner End ordering. Safe forced termination of arbitrary non-cooperative Java code is not an established SDK guarantee.
Java bounded-pool progress and capacity Java #839 Shared and nested bounded pools can starve queued descendants. Define supported nesting, admission, and resource limits; per-invocation fixed pools alone do not solve arbitrary nested blocking.
Java invocation-owned executors Java #840 #735 remains open and stacked on #733. Its proposed opt-in executor isolation is not in main; root and child work still share the configured executor. Broader nested-pool progress stays in #839; hard cleanup and recovery bounds stay in #726.
Python lifecycle ownership, deadlines, and recovery Python #741 Current main does not enforce the invocation deadline and retains unbounded wrapper/checkpoint waits and branch-pool cleanup. Keep these gaps separate from the unresolved SDK/runtime ownership and worker-recovery contract. #783 remains open, but its review requests closure and reconsideration of the architecture; its partial controller is not an accepted complete resolution.
Python new operations under completed/orphaned parents Python #741 — consolidated scope #641 is closed as a duplicate of the linked Python tracker. Implementation and validation remain open: current main's guard checks the operation's own ID and does not reliably reject newly created descendants of completed ancestors. The duplicate closure does not mean the bug is fixed. The consolidated scope retains nested map, invoke/callback/wait, NestingType.FLAT, atomic enqueue-race, and normal/ReplayChildren replay requirements.
Rust concurrent runtime integration and lifecycle Rust #145 Track the supported concurrent entry point/service adapter, invocation deadline and admission behavior, cancellation-safe cleanup, and deployed LMI isolation/recovery evidence. The runtime-specific implementation and acceptance work remains open.

The JS lifecycle proposal #1014 was closed without merging. Since its replacement decision, #1193 has merged cooperative cancellation through StepContext.signal, including early-completing map/parallel work. #1195 merged the function/plugin/transport guidance and closed #1103.

Keep JS #927 focused on remaining gaps and decisions: disposal/deadline changes in #1114 and #1202, unfinished branches of context.promise.race / context.promise.any, automatic invocation cancellation of SDK-owned checkpoint/get-state requests, and checkpoint draining after the deadline. Application I/O can already use StepContext.signal; that does not automatically cancel the SDK's own transport. #1202 explicitly still waits for in-flight checkpoints and permits queued updates to drain; its default client request timeout is 55 seconds, not a worker-recovery bound. The smaller issues do not yet account for every remaining requirement in JS #927. Before closing JS #927 as superseded, assign each residual behavior to a linked implementation or an explicitly accepted supported limit.

Arbitrary non-cooperative I/O, event-loop-blocking synchronous code, and suspension leaving the original business stack/finally unresolved are documented JS limits. Do not continue to track forced cancellation or artificial stack unwinding as promised SDK capabilities.

For Java, #733 targets main and #735 currently targets the #733 branch. If those proposals proceed, merge #733 before retargeting #735. The task-registry and original harness prerequisites are already incorporated through #782; they are not a pending #728 → #731 merge stack.

Remaining LMI validation and shared conformance

Tracker Remaining work and current status
Java #727 The base harness is merged, but lifecycle/recovery and both OTel views still need final deployed evidence with proven same-JVM overlap and genuine runtime X-Ray headers. #823 remains open; its latest LMI cloud check is skipped. Main's cloud workflow is manually triggered, so the requested every-main-push validation is not implemented. The incorporated harness does not establish acceptance.
JS #1017 Lifecycle and deployed OTel/X-Ray validation remain open. #928 was closed without merging. Follow the review decision: design the remaining tests around the final disposal/deadline changes and state what deployed tests prove beyond unit/composed tests before deciding whether to close or rescope this tracker. #1202 reports a real-LMI main/branch deadline probe with signal-aware and signal-ignoring steps; this is partial evidence, not complete JS #1017 acceptance. Same-worker overlap, both OTel views, genuine runtime carriers, and the remaining lifecycle/replay cases still need appropriate evidence. Historical Core 2 results do not validate current Core 3 / OTel 2 artifacts.
Python #742 Runtime-specific lifecycle, worker recovery, replay, and observability acceptance remain open. #743 remains open with review and coverage gaps; its latest LMI cloud check is skipped. The proposal now has 76 cases, but no current full deployed acceptance. The historical 28-case run (16 passed, 12 failed) does not validate the expanded suite or current artifacts. Align recovery assertions with the contract to be agreed in Python #741.
Conformance #161 Implement the optional language-neutral LMI suite, runtime profiles, shared scenario coordinator and versioned evidence, resource leases/quarantine, strict acceptance that fails when applicable mandatory coverage is missing, and SDK adoption with evidence from at least two distinct runtime models. No shared-suite implementation is present in main. Independent SDK harnesses do not replace this work.

Work already landed

Additional closed JS lifecycle tasks are attached: shared-logger invocation attribution #1027 and polling lifetime/stale-timer cleanup #794. Their closed status does not establish acceptance for the final SDK artifacts.

These completed prerequisites do not establish bounded recovery for arbitrary non-cooperative work or complete this epic's LMI acceptance.

Invariants for remaining changes and validation

  • Preserve invocation ownership, healthy peers, and caller-owned executors/clients/providers/exporters. Define cooperative boundaries and finite SDK wait/I/O budgets where supported. Do not claim that cancellation, process freezing, or unsafe thread termination can safely stop arbitrary user work or undo accepted external side effects.
  • For Java, preserve actual task/owner cleanup and matching owner-thread End ordering. Across SDKs, distinguish logical cancellation, actual exit, caller completion, and restored capacity; an early response or exception is not proof of cleanup or recovery.
  • Preserve service-acknowledged terminal outcomes when later cleanup or metadata-pagination deadlines expire. Retain original failure meaning, checkpoint ordering, stored successes/failures, operation identity, and appropriate side-effect counts across replay/retry. Do not fabricate success or PENDING for unacknowledged critical checkpoints.
  • Preserve each language's suspension model. Java owner cleanup must not be imposed on JS by artificially unwinding a suspended business stack.
  • Validate both OTel views, late callbacks, cleanup/flush failures, and shared-provider lifetime. Record expected timeout/export losses without promising lossless or acknowledged export. Preserve Workflow Insight attribution and delivery semantics under overlap.
  • Capture invocation-local X-Ray headers at entry, before Java worker dispatch. The runtime carrier is authoritative even when empty, malformed, or unreadable; use ordinary-Lambda fallback only when unavailable. Preserve Root, Parent, and Sampled, custom/legacy extractors, and context restoration without changing shared global carriers.

Epic completion criteria

  • Each remaining behavior has an implemented, accepted resolution or an explicitly agreed supported limit with linked follow-up ownership. Record the applicable SDK version and configuration; resolve pending design decisions rather than inferring agreement from a proposed implementation.
  • Intended implementation combinations are merged and pass required review, compatibility, and regression checks. Preserve negative controls and fixed-behavior assertions.
  • Public API tests cover initial invocation, timeout, early branch completion, in-flight work, normal/failing return, suspension/resume, and retry while retaining the invariants above.
  • Final deployed LMI evidence establishes the required native sharing boundary, same-environment overlap, deadline/admission behavior, supported worker-capacity recovery, and sustained task/resource behavior. Record commits/artifacts, deployed configuration, environment/worker/process and invocation identities, lifecycle timing, service histories, logs, JUnit, and telemetry evidence.
  • Tests distinguish setup/placement/collection failures from SDK failures, and attempted requests from service acceptance. Emergency releases, unrelated retries, replacement environments, eventual cleanup, skipped checks, or incomplete evidence do not satisfy a recovery bound. Agree test budgets explicitly without presenting them as product SLAs.
  • Required OTel/X-Ray and Workflow Insight scenarios pass with the final compatible artifacts, including sampling, replay, overlapping invocations, present-but-empty/malformed/unreadable carriers versus unavailable-carrier fallback, ownership, and failure paths. Label controlled compatibility fixtures separately from genuine runtime cases.
  • Shared conformance/profile/coordinator requirements and SDK adoption in conformance #161 are complete, with required runtime-model evidence and remaining per-SDK gaps visible.
  • Compatibility/migration guidance, release validation, and release notes are complete. Link final acceptance runs and released core/plugin/provider/layer versions from the relevant trackers.

Runtime references

Activity

  1. self-assigned this
    on Oct 16, 2026
  2. changed the title [-][Bug]: Durable Functions on LMI[/-] [+][Epic]: Durable Functions on LMI[/+] on Oct 16, 2026
  3. 12 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions