From 8927f7a4219e5b1a84003b0c3a69442ae5c8d6ed Mon Sep 17 00:00:00 2001 From: Frank Chen <65260095+zhongkechen@users.noreply.github.com> Date: Wed, 28 Oct 2026 14:51:07 -0700 Subject: [PATCH 01/31] docs: simplify and update OpenTelemetry guide --- .../observability/opentelemetry.md | 1028 +++++------------ .../opentelemetry/execution-plugin.java | 4 +- .../opentelemetry/invocation-plugin.java | 4 +- .../opentelemetry/execution-plugin.py | 4 +- .../opentelemetry/invocation-plugin.py | 4 +- .../opentelemetry/execution-plugin.ts | 4 +- .../opentelemetry/invocation-plugin.ts | 4 +- 7 files changed, 319 insertions(+), 733 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 21998b5f..5e6841fd 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -1,101 +1,54 @@ # OpenTelemetry -The OpenTelemetry plugin instruments a durable execution and emits distributed -traces to any OpenTelemetry backend, such as Amazon CloudWatch. It builds on the -[plugin interface](plugins.md): you register it in your handler configuration, -and it opens spans at the workflow, invocation, operation, and attempt -boundaries. - -A durable execution can run across many Lambda invocations. The durable backend -propagates a stable server span through `_X_AMZN_TRACE_ID`. When the header -contains valid `Root` and `Parent` fields, every plugin span for one execution -uses that canonical trace ID. The execution-scoped `Workflow` span is a direct -child of the remote backend span and keeps a deterministic span ID across -reinvocations. - -Each Lambda invocation also produces an `Invocation` span. It uses the active -ambient Lambda span as its parent only when that span belongs to the canonical -trace. Otherwise, it is a direct child of the remote backend span. This places -the workflow, invocations, operations, and attempts in one trace. - -The two plugins provide different operation views within that trace: - -- **Workflow view**: `ExecutionOtelPlugin` parents operations to `Workflow`. - Each operation or attempt span links to the current `Invocation` span when - that span is created. -- **Invocation view**: `InvocationOtelPlugin` parents each operation segment to - the current `Invocation`. Operations and attempts link to `Workflow`. - Continuation and replay segments also link to the initial logical operation - span. - -Links provide correlation and do not replace parent-child relationships. The -deterministic ID overrides are scoped to spans created by the plugin, so -unrelated OpenTelemetry instrumentation continues to use the provider's normal -ID generator. - -Provider and export-pipeline configuration are external to the plugins. By -default, each plugin uses the global provider. For an application-owned -pipeline, supply the provider, provider factory, or provider builder and -configure its processors, exporter, sampling, resources, propagators, and -library instrumentation. - -Some names on this page keep the X-Ray label, such as the X-Ray trace header and -the `awsxray` collector exporter, but the resulting traces are available in -CloudWatch. +The OpenTelemetry plugins trace durable operations across Lambda invocations. +Choose a workflow or invocation view, register its factory, and configure an +OpenTelemetry provider through a telemetry layer or application code. The +provider owns exporters, resources, library instrumentation, and sampling policy. !!! note "Experimental feature" - The OpenTelemetry plugin is experimental and may change in a future - release. It is not recommended for production workloads yet. Feedback is - welcome through our - [GitHub Discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206). - -!!! note "C# support" - - The OpenTelemetry plugin for C# is a work in progress. Until it ships, this - page covers TypeScript, Python, and Java. - -!!! note "Go support" - - The OpenTelemetry plugin for Go is a work in progress. Until it ships, this - page covers TypeScript, Python, and Java. + The plugins are experimental and are not yet recommended for production workloads. + They are available for TypeScript, + Python, and Java; Go and C# do not yet provide this integration. Feedback is welcome + in the [GitHub Discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206). ## Install the plugin +These examples use the current factory API: core SDK 3.x with OTel plugin 2.x +for TypeScript and Python, and SDK/plugin 3.x for Java. Upgrade the core and +plugin together, including copies supplied by Lambda layers. Older plugin +instances cannot be registered with the 3.x factory API. + === "TypeScript" + Requires Node.js 22 or later: + ```bash - npm install @aws/durable-execution-sdk-js-otel \ - @opentelemetry/api \ - @opentelemetry/core \ - @opentelemetry/sdk-trace-node + npm install @aws/durable-execution-sdk-js@^3 \ + @aws/durable-execution-sdk-js-otel@^2 \ + @opentelemetry/api @opentelemetry/core @opentelemetry/sdk-trace-node ``` - The TypeScript plugin requires Node.js 22 or later. The OpenTelemetry - packages are peer dependencies used by the configured provider or telemetry - layer. - === "Python" - When a Lambda telemetry layer supplies the OpenTelemetry libraries: + When a telemetry layer supplies the OpenTelemetry libraries: ```bash - pip install aws-durable-execution-sdk-python-otel + pip install "aws-durable-execution-sdk-python-otel>=2,<3" ``` - The base package intentionally does not install OpenTelemetry libraries, - which prevents function dependencies from shadowing the version-aligned - libraries in the layer. - - When the application configures its own provider, install the `standalone` - extra: + The package installs a compatible core SDK but leaves OpenTelemetry libraries + to the layer. When the application supplies its own provider, install the + `standalone` extra instead: ```bash - pip install "aws-durable-execution-sdk-python-otel[standalone]" + pip install "aws-durable-execution-sdk-python-otel[standalone]>=2,<3" ``` === "Java" + Use the same 3.x version as the core SDK: + ```xml software.amazon.lambda.durable @@ -104,78 +57,31 @@ CloudWatch. ``` - Add an exporter dependency when the application supplies its own tracer - provider. The ADOT Java agent supplies the export pipeline for the no-arg - plugin constructors. +## Choose and register a view -## The two plugins +Register one OTel view factory. The SDK creates a separate plugin instance for +each invocation, while the provider can be shared across invocations. -Both plugins emit the same span types in one canonical trace. They differ in the -parent of operation spans, the links on those spans, and when the spans become -visible. Register exactly one of them. +| Aspect | Execution view | Invocation view | +| ------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------- | +| Top-level operation parent | `Workflow` | Current `Invocation` | +| Operation and attempt links | Current `Invocation` | `Workflow` | +| Operation spanning a suspension | One logical span, exported on completion | Separate segments across invocations; later segments link to the initial operation span | +| Best suited to | Inspecting the workflow hierarchy | Inspecting progress during long executions | -### ExecutionOtelPlugin +Nested operations remain under their child-context operation span; attempts +remain under their operation. Both views export `Invocation` spans as invocations +end and `Workflow` only when the SDK observes execution success or failure. +Workflow spans can cover long durations, so backend ingestion limits still apply +to both views. Export and flush are best effort; a timeout, stop, or terminated +runtime can leave spans missing. -`ExecutionOtelPlugin` provides a workflow-centered view. Operation spans are -children of the `Workflow` span and link to the current `Invocation` span when -they are created. The `Workflow` span is exported only when the execution -reaches a terminal status. An operation that suspends is also left unexported -until it completes. +### Invocation view -Choose this plugin for shorter executions or when the durable workflow hierarchy -is more important than seeing each invocation as it completes. Long-open -workflow spans can exceed the ingestion limits of some observability platforms. - -=== "TypeScript" - - ```typescript - --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts" - ``` - -=== "Python" - - ```python - --8<-- "examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py" - ``` - -=== "Java" - - ```java - --8<-- "examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java" - ``` - -For an execution that suspends at a wait, the trace resembles: - -```text -Remote backend server span -|-- Workflow (exported on terminal status) -| |-- Operation: fetch-data (STEP) -> link to Invocation #1 -| | `-- Attempt: fetch-data attempt 1 -> link to Invocation #1 -| |-- Operation: cooldown (WAIT) -> link to Invocation #2 -| `-- Operation: process (STEP) -> link to Invocation #2 -| `-- Attempt: process attempt 1 -> link to Invocation #2 -|-- Ambient Lambda span #1 -| `-- Invocation #1 -`-- Ambient Lambda span #2 - `-- Invocation #2 -``` - -Each operation or attempt span links only to the current invocation when that -span is created. If an operation starts in one invocation and completes in -another, its exported operation span links to the invocation that completes it; -the plugin does not retain the earlier invocation span ID. These links do not -make the invocation spans children of `Workflow`. - -### InvocationOtelPlugin - -`InvocationOtelPlugin` provides an invocation-centered view. Each `Invocation` -span parents the operation and attempt spans emitted during that Lambda -invocation. Those operation and attempt spans link to the deterministic -`Workflow` span for execution-scoped correlation. - -The plugin exports an invocation and its child spans when that invocation ends. -Choose it for long-running executions or when you want traces to appear as the -execution progresses. +Use this view to see each invocation's work as the execution progresses. +Completed work uses checkpointed results on replay. A resumed operation or +replayed context can produce another span segment without rerunning a completed +step's body. === "TypeScript" @@ -195,452 +101,142 @@ execution progresses. --8<-- "examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java" ``` -For an execution that suspends at a wait, the trace resembles: - -```text -Remote backend server span -|-- Workflow -|-- Ambient Lambda span #1 -| `-- Invocation #1 -| |-- Operation: fetch-data (STEP) -> link to Workflow -| | `-- Attempt: fetch-data attempt 1 -| | `-> link to Workflow -| `-- Operation: cooldown (WAIT) -> link to Workflow -`-- Ambient Lambda span #2 - `-- Invocation #2 - |-- Operation: cooldown (WAIT) -> links to initial cooldown - | span and Workflow - `-- Operation: process (STEP) -> link to Workflow - `-- Attempt: process attempt 1 -> link to Workflow -``` +### Execution view -An operation that crosses an invocation boundary can produce more than one -span. A continuation or replay segment uses a new span ID and links to the -initial logical operation span, which uses the deterministic operation span ID. -It also retains the `Workflow` link. CloudWatch does not yet visualize span -links. - -### Choosing a plugin - -For both plugins, your observability platform's quotas and limits apply. - -| Consideration | ExecutionOtelPlugin | InvocationOtelPlugin | -| ---------------------- | --------------------------------------------------------- | ----------------------------------------------------- | -| Operation parent | `Workflow` | Current `Invocation` | -| Correlation link | Operation or attempt to current `Invocation` | Operation or attempt to `Workflow` | -| Continuation link | Reconstructed span links only to current `Invocation` | Later segment links to initial logical operation span | -| Workflow export | On terminal execution status | On terminal execution status | -| Operation visibility | When the operation completes | At each invocation boundary | -| In-progress visibility | Limited until operations and the execution complete | Each invocation appears as it completes | -| Better for | Short executions and a workflow-centered hierarchy | Long executions and an invocation-centered hierarchy | -| Platform compatibility | Long-open spans can exceed platform ingestion time limits | Spans stay within the Lambda invocation time limit | - -## Trace parent and sampling - -The plugins resolve the execution parent and the sampling decision -independently. A missing or unusable `Sampled` field does not replace a valid -remote parent with a synthetic root. - -| `_X_AMZN_TRACE_ID` state | Canonical trace ID | Common execution ancestor | Sampling | -| --------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------- | ----------------------------------------------------------------------- | -| Valid `Root`, `Parent`, `Sampled=1` | Reuse `Root` | Remote `Parent` | Preserve sampled | -| Valid `Root`, `Parent`, `Sampled=0` | Reuse `Root` | Remote `Parent` | Preserve not sampled | -| Valid `Root`, `Parent`, no valid `Sampled` | Reuse `Root` | Remote `Parent` | Leave the sampled trace flag unset; configured sampler behavior applies | -| Valid `Root`, missing or invalid `Parent`, `Sampled=1` or `Sampled=0` | Reuse `Root` | Synthetic execution root | Preserve the explicit decision | -| Valid `Root`, missing or invalid `Parent`, no valid `Sampled` | Reuse `Root` | Synthetic execution root | Configured root sampler decides | -| Missing or invalid `Root` | Derive from the execution ARN and stable execution start time | Synthetic execution root | Configured root sampler decides | - -Only `Sampled=0` and `Sampled=1` carry authoritative upstream decisions. An -absent or unusable value does not mean that the upstream system explicitly -chose not to sample. Its OpenTelemetry span context still represents the value -with an unset sampled bit. - -A `ParentBased` sampler treats a remote parent with an unset sampled bit as not -sampled. A directly configured non-parent-based `TraceIdRatioBased` sampler can -decide from the canonical trace ID instead. The reused or derived trace ID -remains stable across reinvocations, so trace-ID-ratio decisions also remain -stable. Other sampler decisions can change between reinvocations unless the -application persists them. - -When `Root` and `Parent` are valid, the normal hierarchy is: - -```text -Remote backend server span (`Root` / `Parent`) -|-- Workflow -|-- Ambient Lambda span #1 -| `-- Invocation #1 -|-- Ambient Lambda span #2 -| `-- Invocation #2 -`-- Invocation #N (when no valid same-trace ambient span exists) -``` - -`Workflow` inherits the canonical trace ID and keeps a deterministic, -execution-scoped span ID. An ambient span can parent `Invocation` only when it -uses the same trace ID. A valid remote parent remains the real ancestor when -`Sampled` is missing. The plugins do not create a synthetic-root link to it. - -When the plugins cannot construct a valid remote parent, they use: - -```text -Synthetic execution root -|-- Workflow -|-- Invocation #1 -|-- Invocation #2 -`-- Invocation #N -``` - -The synthetic execution root has a deterministic span ID that remains stable -across reinvocations and differs from Workflow and operation span IDs. A valid -`Root` still supplies the canonical trace ID. If `Root` is also invalid, the -plugins derive the trace ID from the execution ARN and stable execution start -time. Fallback mode ignores unrelated ambient context. TypeScript, Python, and -Java use this same topology with global and application-owned providers. The -selected sampling decision applies consistently to every descendant. - -## Span structure and attributes - -Both plugins emit four span types. An invocation span covers one Lambda -invocation. Operation spans cover durable operations such as steps, waits, child -contexts, callbacks, and chained invokes. Attempt spans cover user function -attempts, so retries appear as sibling spans. - -| Span | Attributes | -| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Workflow | `durable.execution.arn`, `durable.execution.status` | -| Invocation | `durable.execution.arn`, `durable.invocation.status`, `durable.invocation.first` | -| Operation | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status` | -| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.attempt.number`, `durable.attempt.outcome` | - -`durable.operation.type` includes `STEP`, `WAIT`, `CONTEXT`, `CHAINED_INVOKE`, -and `CALLBACK`. A `CONTEXT` operation gets an operation span but not a separate -attempt span. - -## Deploy with the ADOT layer - -The AWS Distro for OpenTelemetry (ADOT) Lambda layer supplies a global provider, -auto-instrumentation, and a collector extension. Add the language-specific -layer, set `AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument`, enable active tracing -so the runtime supplies `_X_AMZN_TRACE_ID`, and grant the function role the -`AWSXRayDaemonWriteAccess` managed policy. Auto-instrumentation can then create -an ambient Lambda span in the same trace as the durable backend parent. - -The no-arg plugin constructors use the global provider. You do not need a -provider option. +Use this view to keep the durable operation hierarchy beneath `Workflow`. +An operation that suspends is exported when it completes, using its durable +start and end times. Its span links to the invocation that exports the completion; +it does not accumulate links to every earlier invocation. === "TypeScript" - Find the current ADOT JavaScript layer ARN for your region and architecture - in the - [ADOT Lambda layer documentation](https://aws-otel.github.io/docs/getting-started/lambda#adot-lambda-layer-arns). - - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: nodejs24.x - Handler: index.handler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` - ```typescript - new ExecutionOtelPlugin(); - // or - new InvocationOtelPlugin(); + --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts" ``` === "Python" - Find the current ADOT Python layer ARN for your region and architecture in - the - [ADOT Lambda layer documentation](https://aws-otel.github.io/docs/getting-started/lambda#adot-lambda-layer-arns). - - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: python3.14 - Handler: index.handler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` - ```python - ExecutionOtelPlugin() - # or - InvocationOtelPlugin() + --8<-- "examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py" ``` === "Java" - The Java agent must also load the plugin JAR as an agent extension so its - auto-configuration SPI can scope deterministic ID generation to durable - spans. Package the JAR in a layer under `java/lib`, then point - `OTEL_JAVAAGENT_EXTENSIONS` to the deployed path. - - Find the current ADOT Java layer ARN in the - [ADOT Java instrumentation releases](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest). - - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: java25 - Handler: com.example.ExampleHandler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - - - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument - OTEL_JAVAAGENT_EXTENSIONS: /opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` - ```java - new ExecutionOtelPlugin(); - // or - new InvocationOtelPlugin(); + --8<-- "examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java" ``` - To load the plugin from the layer without registering it in handler code, - also set `DURABLE_EXECUTION_PLUGINS` to `otel-execution` or - `otel-invocation`. +## Configure a telemetry layer -When the plugins use the global provider, they resolve it at invocation start. -If a usable SDK provider or the integration required for scoped deterministic -IDs is unavailable, all three SDKs log a warning and disable telemetry for that -invocation. They retry provider resolution on the next invocation instead of -emitting spans without deterministic durable IDs. +A language auto-instrumentation layer registers the global provider used by the +default factories. Select a layer compatible with your runtime, architecture, +and region: -## Deploy with the community auto-instrumentation layer +| Layer | `AWS_LAMBDA_EXEC_WRAPPER` | Setup | +| -------------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------- | +| AWS Distro for OpenTelemetry (ADOT) | `/opt/otel-instrument` | [ADOT Lambda layers](https://aws-otel.github.io/docs/getting-started/lambda) | +| OpenTelemetry community language layer | `/opt/otel-handler` | [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) | -The OpenTelemetry community language layer initializes auto-instrumentation, -registers a global OpenTelemetry provider, and exports spans using OTLP. Set -`AWS_LAMBDA_EXEC_WRAPPER` to `/opt/otel-handler` and configure the OTLP endpoint -for your observability backend. Enable active tracing so the plugin can reuse -the durable backend parent and the layer can create a same-trace ambient Lambda -span. +Enable active tracing to receive the durable backend's X-Ray trace context. +For ADOT, add these properties to your existing SAM function and grant its role +`AWSXRayDaemonWriteAccess` to export traces to CloudWatch through X-Ray: -The no-arg plugin constructors use the global provider. You do not need to -configure or pass a provider to the plugin. +```yaml +Layers: + - +Environment: + Variables: + AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument +Tracing: Active +``` + +For a community language layer exporting directly to an OTLP backend, use: -The examples below use OTLP HTTP and disable metrics and log export. If the -backend requires authentication, also configure `OTEL_EXPORTER_OTLP_HEADERS`. +```yaml +Layers: + - +Environment: + Variables: + AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler + OTEL_TRACES_EXPORTER: otlp + OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf + OTEL_EXPORTER_OTLP_ENDPOINT: https:// + OTEL_METRICS_EXPORTER: none + OTEL_LOGS_EXPORTER: none +Tracing: Active +``` -Follow the -[OpenTelemetry Lambda auto-instrumentation documentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) -to obtain the language layer. +Configure `OTEL_EXPORTER_OTLP_HEADERS` if your backend requires authentication. +A collector extension is optional for local processing or additional exporters; +follow the [collector setup guide](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector) +for its layer, receiver, and `OPENTELEMETRY_COLLECTOR_CONFIG_URI` configuration. +A collector alone does not initialize a language provider. For that setup, leave +`AWS_LAMBDA_EXEC_WRAPPER` unset and configure the provider in application code as +shown below. For collector export to CloudWatch, enable Transaction Search in +your account. === "TypeScript" - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: nodejs24.x - Handler: index.handler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler - OTEL_TRACES_EXPORTER: otlp - OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf - OTEL_EXPORTER_OTLP_ENDPOINT: https:// - OTEL_METRICS_EXPORTER: none - OTEL_LOGS_EXPORTER: none - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` - - ```typescript - new ExecutionOtelPlugin(); - // or - new InvocationOtelPlugin(); - ``` + The default factory resolves the global provider at invocation start. === "Python" - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: python3.14 - Handler: index.handler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler - OTEL_TRACES_EXPORTER: otlp - OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf - OTEL_EXPORTER_OTLP_ENDPOINT: https:// - OTEL_METRICS_EXPORTER: none - OTEL_LOGS_EXPORTER: none - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` - - ```python - ExecutionOtelPlugin() - # or - InvocationOtelPlugin() - ``` + Use the base plugin package with a telemetry layer so function dependencies + do not shadow the layer's OpenTelemetry libraries. === "Java" - Use the community Java agent layer for auto-instrumentation. The agent must - also load the plugin JAR as an agent extension so its auto-configuration SPI - can scope deterministic ID generation to durable spans. Package the JAR in - a layer under `java/lib`, then point `OTEL_JAVAAGENT_EXTENSIONS` to the - deployed path. - - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: java25 - Handler: com.example.ExampleHandler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - - - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler - OTEL_TRACES_EXPORTER: otlp - OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf - OTEL_EXPORTER_OTLP_ENDPOINT: https:// - OTEL_METRICS_EXPORTER: none - OTEL_LOGS_EXPORTER: none - OTEL_JAVAAGENT_EXTENSIONS: /opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` + The Java agent must also load the plugin JAR as an extension to install + durable ID generation and sampling. Package the JAR in a layer under + `java/lib`, add that layer to the function, and set: - ```java - new ExecutionOtelPlugin(); - // or - new InvocationOtelPlugin(); + ```text + OTEL_JAVAAGENT_EXTENSIONS=/opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar ``` - To load the plugin from the layer without registering it in handler code, - also set `DURABLE_EXECUTION_PLUGINS` to `otel-execution` or - `otel-invocation`. - -### Add a collector extension (optional) - -Add the community collector extension layer when you need local processing, -routing, or an exporter that is not available in the language layer. Keep -`AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler` so the language layer continues to -initialize the provider and the ambient Lambda span. + Obtain the ADOT Java layer from the + [Java instrumentation releases](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest). -Append the collector layer, point the OTLP exporter at its local receiver, and -set the collector configuration path: +If the global SDK provider or required durable ID integration is unavailable, +the plugin logs a warning and disables its telemetry for that invocation. +The next invocation retries initialization. -```yaml -Layers: - - - - !Sub arn:aws:lambda:${AWS::Region}::layer:: -Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler - OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost:4318 - OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml -``` +### Load the factory without handler registration -For example, include the following `collector.yaml` in the function bundle to -export traces to X-Ray: +`DURABLE_EXECUTION_PLUGINS` selects a default-configured factory from an installed +package or layer. It still requires a global provider. Choose one view through +either code or environment configuration. -```yaml -receivers: - otlp: - protocols: - http: - endpoint: "localhost:4318" -exporters: - awsxray: - region: "${AWS_REGION}" -service: - pipelines: - traces: - receivers: [otlp] - exporters: [awsxray] -``` +| Language | Invocation view | Execution view | +| ---------- | ---------------------------------------------------- | --------------------------------------------------- | +| TypeScript | `@aws/durable-execution-sdk-js-otel/otel-invocation` | `@aws/durable-execution-sdk-js-otel/otel-execution` | +| Python | `otel-invocation` | `otel-execution` | +| Java | `otel-invocation` | `otel-execution` | -Change the collector exporter to route spans to another platform. See the -[collector extension documentation](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector) -for available components and layer deployment instructions. +## Configure an application-owned provider -### Collector-only setup (advanced) - -Use the collector extension without a language auto-instrumentation layer when -you need to configure the OpenTelemetry provider in application code. Add only -the collector layer, leave `AWS_LAMBDA_EXEC_WRAPPER` unset, keep active tracing -enabled, and pass the application-owned provider to the plugin. Without an -ambient Lambda span, `Invocation` uses the remote backend span directly. - -```yaml -Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: -Environment: - Variables: - OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml -Tracing: Active -``` +Configure the exporter and processors once, then register the resulting factory +as in the handler examples above. The plugin flushes at invocation end but does +not take over provider shutdown. These examples send OTLP HTTP to a collector +listening on loopback port 4318. === "TypeScript" - `tracerProviderFactory` receives a function that creates the plugin's scoped - deterministic ID generator. Use it when constructing the provider. The - application owns provider registration and shutdown. - - ```bash - npm install @opentelemetry/exporter-trace-otlp-http - ``` + Install `@opentelemetry/exporter-trace-otlp-http`. The provider factory receives + the durable ID-generator factory; pass its result to the provider constructor. ```typescript - import { ExecutionOtelPlugin } from "@aws/durable-execution-sdk-js-otel"; + import { createInvocationOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; - import { - BatchSpanProcessor, - NodeTracerProvider, - } from "@opentelemetry/sdk-trace-node"; + import { BatchSpanProcessor, NodeTracerProvider } from "@opentelemetry/sdk-trace-node"; - const plugin = new ExecutionOtelPlugin({ + const pluginFactory = createInvocationOtelPluginFactory({ tracerProviderFactory: (createIdGenerator) => { const provider = new NodeTracerProvider({ idGenerator: createIdGenerator(), - spanProcessors: [ - new BatchSpanProcessor( - new OTLPTraceExporter({ - url: "http://localhost:4318/v1/traces", - }), - ), - ], + spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter({ + url: "http://localhost:4318/v1/traces", + }))], }); provider.register(); return provider; @@ -648,270 +244,260 @@ Tracing: Active }); ``` - Substitute `InvocationOtelPlugin` to use the invocation-centered view. To - preserve a custom application ID generator for unrelated spans, pass it to - `createIdGenerator(applicationIdGenerator)`. + Use `createExecutionOtelPluginFactory` for the execution view. === "Python" - Install the standalone dependencies and an OTLP HTTP exporter: - - ```bash - pip install "aws-durable-execution-sdk-python-otel[standalone]" \ - opentelemetry-exporter-otlp-proto-http - ``` - - Pass the application-owned provider through the shared configuration - object: + Install the `standalone` extra and `opentelemetry-exporter-otlp-proto-http`. ```python + from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor - from aws_durable_execution_sdk_python_otel import ( - ExecutionOtelPlugin, + InvocationOtelPluginFactory, OtelPluginConfig, ) provider = TracerProvider() - provider.add_span_processor( - BatchSpanProcessor( - OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces") - ) - ) - - plugin = ExecutionOtelPlugin( + provider.add_span_processor(BatchSpanProcessor( + OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces") + )) + trace.set_tracer_provider(provider) + plugin_factory = InvocationOtelPluginFactory( OtelPluginConfig(tracer_provider=provider) ) ``` - Substitute `InvocationOtelPlugin` to use the invocation-centered view. + Use `ExecutionOtelPluginFactory` for the execution view. === "Java" - Add the OpenTelemetry OTLP exporter dependency, then pass a provider builder - to either plugin. The plugin wraps the builder's ID generator and builds the - application-owned provider. - - ```xml - - io.opentelemetry - opentelemetry-exporter-otlp - ${opentelemetry.version} - - ``` + Add `io.opentelemetry:opentelemetry-exporter-otlp` to your dependencies. The + factory wraps the provider builder's ID generator and sampler, then builds + the provider once. ```java import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter; import io.opentelemetry.sdk.trace.SdkTracerProvider; import io.opentelemetry.sdk.trace.export.BatchSpanProcessor; - import software.amazon.lambda.durable.otel.ExecutionOtelPlugin; + import software.amazon.lambda.durable.otel.InvocationOtelPlugin; var exporter = OtlpHttpSpanExporter.builder() .setEndpoint("http://localhost:4318/v1/traces") .build(); - - var providerBuilder = SdkTracerProvider.builder() - .addSpanProcessor(BatchSpanProcessor.builder(exporter).build()); - - var plugin = new ExecutionOtelPlugin(providerBuilder); + var pluginFactory = InvocationOtelPlugin.factory( + SdkTracerProvider.builder() + .addSpanProcessor(BatchSpanProcessor.builder(exporter).build())); ``` - Substitute `InvocationOtelPlugin` to use the invocation-centered view. + Use `ExecutionOtelPlugin.factory` for the execution view. -## Configuration +### Plugin options === "TypeScript" - Both plugins accept an `OtelPluginConfig` object: + Pass `OtelPluginConfig` to either `createInvocationOtelPluginFactory(config)` + or `createExecutionOtelPluginFactory(config)`. - - **tracerProviderFactory** Creates an application-owned provider. Omit it - to use the global provider. - - **contextExtractor** Extracts upstream trace context. Defaults to - `xRayContextExtractor`. Use `w3cClientContextExtractor` for W3C - `traceparent` data in Lambda client context. - - **instrumentationName** Instrumentation scope name. Defaults to - `aws-durable-execution-sdk-js`. - - **workflowSpanName** Name of the execution-scoped `Workflow` span. - Defaults to `Workflow`. - - **enrichLogger** Adds `traceId`, `spanId`, and `otelTraceSampled` to each - durable log record. Defaults to `true`. + | Option | Default | Purpose | + | ----------------------- | ------------------------------ | ----------------------------------------------------------------- | + | `tracerProviderFactory` | Global provider | Build an application-owned provider | + | `contextExtractor` | `xRayContextExtractor` | Extract this execution's upstream context | + | `instrumentationName` | `aws-durable-execution-sdk-js` | Instrumentation scope | + | `workflowSpanName` | `Workflow` | Workflow span name | + | `enrichLogger` | `true` | Add trace fields to durable logs | + | `flushTimeoutMillis` | `1000` | Total time budget for awaiting provider flushes at invocation end | - The application or telemetry layer owns exporters, propagators, resources, - sampling, and HTTP or AWS SDK instrumentation. + `w3cClientContextExtractor` reads `traceparent` from Lambda client context. + The default X-Ray extractor reads invocation-local `xRayTraceId` when + present, otherwise `_X_AMZN_TRACE_ID`. + A flush timeout stops waiting; it does not cancel exports or synchronous + provider work. === "Python" - Both plugins accept the same `OtelPluginConfig` dataclass: + Pass `OtelPluginConfig` to `InvocationOtelPluginFactory(config)` or + `ExecutionOtelPluginFactory(config)`. - - **tracer_provider** Application-owned provider. Omit it to use the global - provider. - - **context_extractor** Defaults to `xray_context_extractor`. Use - `w3c_client_context_extractor` for W3C `traceparent` data in Lambda - client context. - - **instrument_name** Instrumentation scope name. Defaults to - `aws-durable-execution-sdk-python`. - - **workflow_span_name** Name of the execution-scoped `Workflow` span. - Defaults to `Workflow`. - - **enrich_logger** Installs a root-logger filter that stamps trace context - onto log records. Defaults to `True`. + | Option | Default | Purpose | + | -------------------- | ---------------------------------- | ------------------------------------------------ | + | `tracer_provider` | Global provider | Use an application-owned SDK provider | + | `context_extractor` | `xray_context_extractor` | Extract upstream context from `_X_AMZN_TRACE_ID` | + | `instrument_name` | `aws-durable-execution-sdk-python` | Instrumentation scope | + | `workflow_span_name` | `Workflow` | Workflow span name | + | `enrich_logger` | `True` | Add trace fields through root logger handlers | - ```python - plugin = InvocationOtelPlugin( - OtelPluginConfig( - instrument_name="my-service", - enrich_logger=False, - ) - ) - ``` + `w3c_client_context_extractor` is a placeholder that returns `None`. + Use the default X-Ray extractor or implement a custom extractor. === "Java" - Each plugin supports the following constructor forms: + Both plugins expose `factory()`, `factory(config)`, `factory(providerBuilder)`, + and `factory(providerBuilder, config)`. Build `OtelPluginConfig` with: - ```java - new InvocationOtelPlugin(); - new InvocationOtelPlugin(otelPluginConfig); - new InvocationOtelPlugin(tracerProviderBuilder); - new InvocationOtelPlugin(tracerProviderBuilder, otelPluginConfig); - ``` + | Builder method | Default | Purpose | + | -------------------------- | -------------------------------- | ------------------------------ | + | `contextExtractor(...)` | `new XRayContextExtractor()` | Extract upstream trace context | + | `instrumentationName(...)` | `aws-durable-execution-sdk-java` | Instrumentation scope | + | `workflowSpanName(...)` | `Workflow` | Workflow span name | + | `enableMdc(...)` | `true` | Add trace fields to SLF4J MDC | - `ExecutionOtelPlugin` provides the same forms. The no-arg and config-only - constructors use the global provider. The builder constructors create an - application-owned provider. +A custom extractor must return the durable execution's own context consistently +across resumes. An unrelated or per-invocation trace ID splits the execution trace. - Build `OtelPluginConfig` with: +## Trace identity and sampling - - **contextExtractor(...)** Defaults to `new XRayContextExtractor()`. - - **enableMdc(...)** Adds `traceId`, `spanId`, and `otelTraceSampled` to the - SLF4J MDC. Defaults to `true`. - - **workflowSpanName(...)** Defaults to `Workflow`. - - **instrumentationName(...)** Defaults to - `aws-durable-execution-sdk-java`. +Both views keep one canonical trace ID per execution. A valid propagated X-Ray +`Root` supplies that ID, and a valid `Parent` becomes the remote execution +ancestor. Missing `Sampled` does not invalidate a complete remote parent. +`Workflow` has a stable execution-scoped span ID and parents to that ancestor. +`Invocation` uses an active ambient span, such as the Lambda span, only when it +belongs to the same trace; otherwise it parents to the execution ancestor. - ```java - var config = OtelPluginConfig.builder() - .instrumentationName("my-service") - .enableMdc(false) - .build(); +```text +Execution ancestor +|-- Workflow +|-- Ambient Lambda span #1 +| `-- Invocation #1 +`-- Ambient Lambda span #2 + `-- Invocation #2 +``` - var plugin = new InvocationOtelPlugin(config); - ``` +When no complete remote parent is available, the SDK uses a synthetic +`DurableExecutionRoot`. It retains a valid propagated trace ID, or derives one +from the execution ARN and stable execution start time. With a supported provider +and that start time, sampled invocations export a zero-duration root with stable +identity to preserve ancestry before the execution completes. Resumes can export +another copy. `Workflow` carries duration and outcome; the fallback root carries +neither. Remote backend spans are owned by the backend and are never exported +by the plugin. -Configure sampling through the provider. With ADOT, use the standard -`OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` environment variables. +Configure sampling through the provider, or through `OTEL_TRACES_SAMPLER` and +`OTEL_TRACES_SAMPLER_ARG` when using auto-configuration. Keep the plugin's sampler +integration installed so durable spans share the resolved decision while other +instrumentation retains its own policy. -## Load plugins dynamically +=== "TypeScript" -The SDK can discover a plugin from a Lambda layer or installed package without -handler registration code. Dynamic loading uses default configuration and -therefore requires a global OpenTelemetry provider. + An explicit `Sampled=1` or `Sampled=0` on a valid extracted trace takes + precedence. Otherwise, the configured sampler evaluates its root policy + using the canonical trace ID. Missing `Sampled` therefore does not make a + `ParentBased` sampler automatically drop the trace. -=== "TypeScript" + A tracer that hides its sampler cannot enforce a shared durable sampling + decision and retains terminal-only fallback-root export. - ```text - DURABLE_EXECUTION_PLUGINS=@aws/durable-execution-sdk-js-otel/otel-invocation - ``` +=== "Python" + + Sampling follows the explicit backend decision, then the recording/sampled + state of a same-trace ambient span, then the configured sampler's root policy. + The plugin resolves the result once per invocation for its durable spans. - Use `@aws/durable-execution-sdk-js-otel/otel-execution` for the - workflow-centered plugin. +=== "Java" + + Sampling follows the explicit backend decision, then a same-trace ambient + span, then the configured sampler. Retain `DurableSampler`, installed by the + provider-builder and agent-extension paths, as the final sampler. Replacing + it with a plain sampler removes the shared-decision guarantee. + +Sampling decisions are not checkpointed across invocations. An explicit upstream +decision or deterministic trace-ID-based policy keeps them consistent across +resumes; a stateful custom sampler can choose differently on a later invocation. + +## Span attributes and failures + +| Span | Main attributes | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Workflow | `durable.execution.arn`, `durable.execution.status` | +| Invocation | `durable.execution.arn`, `durable.invocation.status`, `durable.invocation.first` | +| Operation | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status` | +| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.attempt.number`, `durable.attempt.outcome` | + +Operation types include `STEP`, `WAIT`, `CONTEXT`, `CHAINED_INVOKE`, and `CALLBACK`. +A context has an operation span without a separate attempt span. An invocation +that will retry reports `durable.invocation.status=RETRYING` with OTel `UNSET`. +A caught operation failure can have an `ERROR` span while the enclosing invocation +and workflow succeed. Inspect operation status as well as exception events; +a backend can report failure without diagnostic details. + +=== "TypeScript" + + `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without + diagnostics. The plugins export visible completion updates even when replay skips the + operation's body. They need the operation metadata to do so. When completed + child contexts would otherwise prune that metadata, configure + `pluginsConfig.childOperationsDepth` as described in + [operation detail](telemetry/exporter.md#operation-detail). === "Python" - ```text - DURABLE_EXECUTION_PLUGINS=otel-invocation - ``` + Two open fixes refine this behavior: - Use `otel-execution` for the workflow-centered plugin. + - [PR #865](https://github.com/aws/aws-durable-execution-sdk-python/pull/865) + marks `FAILED`, `TIMED_OUT`, and `STOPPED` operations as `ERROR` even when + diagnostics are absent, and places terminal exception events at the span's + normalized end time. Without this fix, these outcomes can remain `UNSET` + when no error object is supplied. + - [PR #869](https://github.com/aws/aws-durable-execution-sdk-python/pull/869) + exports visible terminal updates from invocation input and checkpoint + notifications when cached-parent replay skips the completed child. It + deduplicates successful observations within an invocation, without promising + exactly-once delivery across retries or exporter failures. -=== "Java" + These changes require builds containing the PRs until they are merged and released. + The plugins cannot reconstruct operation metadata that the service has pruned. - ```text - DURABLE_EXECUTION_PLUGINS=otel-invocation - ``` +=== "Java" - Use `otel-execution` for the workflow-centered plugin. Put the plugin JAR - under `java/lib` in the layer so it is on the Lambda class path. + `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without + diagnostics. Missing details do not produce an exception event. Execution-view + operation exceptions use the known backend end timestamp when available; + invocation-view exceptions retain normal OpenTelemetry event timing. -## Correlate logs with traces +## Correlate logs and verify traces -The plugin stamps active trace and span IDs onto log records. Logs from every -reinvocation of one execution use the same canonical `traceId`. The `spanId` -identifies the active invocation, operation, or attempt, and -`otelTraceSampled` reflects the effective sampled flag. See -[Logging](logging.md) for the SDK logger. +Log correlation is enabled by default. `traceId` identifies the execution trace, +`spanId` identifies the active span, and `otelTraceSampled` carries its sampled +flag. Application-created spans on that trace can become the active context too. === "TypeScript" - With `enrichLogger` enabled, the plugin adds `traceId`, `spanId`, and - `otelTraceSampled` to each durable log record through the - `enrichLogContext()` hook described in - [Logging from a plugin](plugins.md#logging-from-a-plugin). + `enrichLogger` adds fields through the durable logger's `enrichLogContext` + hook. In execution view, handler code between operations uses `Workflow`; + invocation view uses `Invocation`. Step bodies use their attempt span. + Custom logger adapters must forward the enriched context. === "Python" - With `enrich_logger=True`, the plugin installs a filter on root logger - handlers. It adds `traceId`, `spanId`, and `otelTraceSampled` when a valid - span is active. Treat these fields as optional in your formatter. + `enrich_logger` installs a filter on root logger handlers. Handler code uses + `Invocation`, step bodies use their attempt, and child-context bodies use + their context span. Fields are optional: initialization logs and threads + without invocation context are left uncorrelated. === "Java" - With `enableMdc(true)`, the plugin puts `traceId`, `spanId`, and - `otelTraceSampled` into the SLF4J MDC. Configure your logging framework to - include MDC fields in its output. - -## Verify - -Invoke a durable function that includes a wait or resume so it runs across more -than one Lambda invocation. In CloudWatch Traces, expect one canonical trace ID -for every span carrying the execution ARN: - -- The durable backend server span is the common ancestor when CloudWatch - returns it. `Workflow` is its direct child. -- Each `Invocation` is a child of a same-trace ambient Lambda span, or a direct - child of the backend server span when no valid ambient span exists. -- `ExecutionOtelPlugin` operations are children of `Workflow` and link only to - the current invocation when each span is created. -- `InvocationOtelPlugin` operation segments are children of `Invocation` and - link to `Workflow`. A resumed wait or replayed operation also links to its - initial logical operation span. -- Log records retain the canonical `traceId`. Their `spanId` values match the - active invocation, operation, or attempt. - -Some backend queries omit the remote backend span. In that case, `Workflow` and -directly parented `Invocation` spans still contain its non-null parent span ID. -Do not interpret the unresolved parent as a parentless Workflow span. - -For Java, use the CloudWatch **Group by nodes** view to inspect the hierarchy. -The ungrouped X-Ray Segments Timeline cannot attach OTLP-exported spans beneath -the Lambda platform segment. - -When you use the community collector layer, enable CloudWatch Transaction -Search in your account for traces to appear. - -If no plugin spans appear: - -- Confirm a provider with a span processor and exporter is registered. -- Check for a warning that the global SDK provider or its deterministic ID - integration was unavailable at invocation start. -- With ADOT, confirm `AWS_LAMBDA_EXEC_WRAPPER` and active tracing are enabled. -- With the community collector, confirm the layer and - `OPENTELEMETRY_COLLECTOR_CONFIG_URI` are configured. -- For Java ADOT, confirm the plugin JAR is listed in - `OTEL_JAVAAGENT_EXTENSIONS`. -- Check `_X_AMZN_TRACE_ID` and provider sampling configuration. An explicit - `Sampled=0` suppresses export, and `ParentBased` treats a missing sampled bit - as not sampled. - -If spans carrying one execution ARN use different trace IDs, confirm active -tracing is enabled and that every invocation uses the same current plugin and -context extractor. Missing or invalid remote parent data should still produce -one canonical fallback trace, not one trace per invocation. + `enableMdc` adds the fields to SLF4J MDC. Configure your logging framework + to include MDC fields in its output. In CloudWatch, use **Group by nodes** + to inspect the hierarchy; the ungrouped X-Ray Segments Timeline cannot nest + OTLP spans under the Lambda platform segment. + +Test with an execution that waits and resumes. Confirm that its spans share a +trace ID, invocations appear as they finish, completed steps do not run again, +and logs identify the active span. `Workflow` appears after terminal completion. +Span links correlate the two views but do not change parent-child relationships; +CloudWatch does not currently visualize these links. + +If plugin spans are missing, check provider initialization warnings, exporter +configuration, the layer's execution wrapper, and sampling. For the Java agent, +also check `OTEL_JAVAAGENT_EXTENSIONS`. A missing `Workflow` can mean the execution +has not finished or no terminal invocation exported it. A parent ID without a +visible span can belong to the remote backend or an undelivered fallback root. ## See also - [Plugins](plugins.md) -- [Durable Execution Telemetry](telemetry/index.md) - [Logging](logging.md) +- [Durable Execution Telemetry](telemetry/index.md) - [Steps](../operations/step.md) diff --git a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java index c1541cbf..162c1651 100644 --- a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java +++ b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java @@ -8,11 +8,11 @@ public class ExecutionOtelExample extends DurableHandler, String> { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new ExecutionOtelPlugin()).build(); + return DurableConfig.builder().withPlugins(ExecutionOtelPlugin.factory()).build(); } @Override - protected String handleRequest(Map event, DurableContext context) { + public String handleRequest(Map event, DurableContext context) { String data = context.step("fetch-data", String.class, ctx -> event.get("id")); context.wait("cooldown", Duration.ofSeconds(5)); return context.step("process", String.class, ctx -> "processed " + data); diff --git a/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java b/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java index 6ab7ec2c..b00e4e15 100644 --- a/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java +++ b/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java @@ -6,11 +6,11 @@ public class InvocationOtelExample extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new InvocationOtelPlugin()).build(); + return DurableConfig.builder().withPlugins(InvocationOtelPlugin.factory()).build(); } @Override - protected String handleRequest(Object event, DurableContext context) { + public String handleRequest(Object event, DurableContext context) { return context.step("process", String.class, ctx -> "done"); } } diff --git a/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py b/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py index 0a1b7327..151ac2af 100644 --- a/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py +++ b/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py @@ -1,9 +1,9 @@ from aws_durable_execution_sdk_python import DurableContext, durable_execution from aws_durable_execution_sdk_python.config import Duration -from aws_durable_execution_sdk_python_otel import ExecutionOtelPlugin +from aws_durable_execution_sdk_python_otel import ExecutionOtelPluginFactory -@durable_execution(plugins=[ExecutionOtelPlugin()]) +@durable_execution(plugins=[ExecutionOtelPluginFactory()]) def handler(event: dict, context: DurableContext) -> str: data = context.step(lambda ctx: event["id"], name="fetch-data") context.wait(Duration.from_seconds(5), name="cooldown") diff --git a/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py b/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py index 0d6d5f34..24b80ee9 100644 --- a/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py +++ b/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py @@ -1,7 +1,7 @@ from aws_durable_execution_sdk_python import DurableContext, durable_execution -from aws_durable_execution_sdk_python_otel import InvocationOtelPlugin +from aws_durable_execution_sdk_python_otel import InvocationOtelPluginFactory -@durable_execution(plugins=[InvocationOtelPlugin()]) +@durable_execution(plugins=[InvocationOtelPluginFactory()]) def handler(event: dict, context: DurableContext) -> str: return context.step(lambda ctx: "done", name="process") diff --git a/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts b/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts index 07aef2fd..ac28adb9 100644 --- a/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts +++ b/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts @@ -1,5 +1,5 @@ import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; -import { ExecutionOtelPlugin } from "@aws/durable-execution-sdk-js-otel"; +import { createExecutionOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; export const handler = withDurableExecution( async (event: { id: string }, context: DurableContext) => { @@ -7,5 +7,5 @@ export const handler = withDurableExecution( await context.wait("cooldown", { seconds: 5 }); return await context.step("process", async () => `processed ${data}`); }, - { plugins: [new ExecutionOtelPlugin()] }, + { plugins: [createExecutionOtelPluginFactory()] }, ); diff --git a/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts b/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts index 3d4a8927..5dbdd413 100644 --- a/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts +++ b/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts @@ -1,9 +1,9 @@ import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; -import { InvocationOtelPlugin } from "@aws/durable-execution-sdk-js-otel"; +import { createInvocationOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; export const handler = withDurableExecution( async (event, context: DurableContext) => { return await context.step("process", async () => "done"); }, - { plugins: [new InvocationOtelPlugin()] }, + { plugins: [createInvocationOtelPluginFactory()] }, ); From ba690dd109e010924120c83ffa7fb93af07c6e48 Mon Sep 17 00:00:00 2001 From: Frank Chen <65260095+zhongkechen@users.noreply.github.com> Date: Wed, 28 Oct 2026 15:03:14 -0700 Subject: [PATCH 02/31] docs: remove stale experimental OTel warning --- docs/sdk-reference/observability/opentelemetry.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 5e6841fd..143125f2 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -5,12 +5,8 @@ Choose a workflow or invocation view, register its factory, and configure an OpenTelemetry provider through a telemetry layer or application code. The provider owns exporters, resources, library instrumentation, and sampling policy. -!!! note "Experimental feature" - - The plugins are experimental and are not yet recommended for production workloads. - They are available for TypeScript, - Python, and Java; Go and C# do not yet provide this integration. Feedback is welcome - in the [GitHub Discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206). +The plugins are available for TypeScript, Python, and Java. Go and C# do not yet +provide this integration. ## Install the plugin @@ -501,3 +497,4 @@ visible span can belong to the remote backend or an undelivered fallback root. - [Logging](logging.md) - [Durable Execution Telemetry](telemetry/index.md) - [Steps](../operations/step.md) +- [Feedback and discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206) From 05ed660a7f6c7d19146877ec37b06bab8b205f3f Mon Sep 17 00:00:00 2001 From: Frank Chen <65260095+zhongkechen@users.noreply.github.com> Date: Wed, 28 Oct 2026 15:08:05 -0700 Subject: [PATCH 03/31] docs: explain OTel plugin discovery values --- .../observability/opentelemetry.md | 22 +++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 143125f2..b2344d3a 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -199,15 +199,19 @@ The next invocation retries initialization. ### Load the factory without handler registration -`DURABLE_EXECUTION_PLUGINS` selects a default-configured factory from an installed -package or layer. It still requires a global provider. Choose one view through -either code or environment configuration. - -| Language | Invocation view | Execution view | -| ---------- | ---------------------------------------------------- | --------------------------------------------------- | -| TypeScript | `@aws/durable-execution-sdk-js-otel/otel-invocation` | `@aws/durable-execution-sdk-js-otel/otel-execution` | -| Python | `otel-invocation` | `otel-execution` | -| Java | `otel-invocation` | `otel-execution` | +`DURABLE_EXECUTION_PLUGINS` selects a default-configured OTel factory from an installed +package or layer. These factories use the global OpenTelemetry tracer provider +initialized by your telemetry layer or application. Choose one view through either +code or environment configuration. + +The value follows each SDK's discovery mechanism: an importable module specifier +or a registered provider name. + +| Language and lookup | Invocation view | Execution view | +| ------------------------------------ | ---------------------------------------------------- | --------------------------------------------------- | +| TypeScript: Node.js module specifier | `@aws/durable-execution-sdk-js-otel/otel-invocation` | `@aws/durable-execution-sdk-js-otel/otel-execution` | +| Python: package entry-point name | `otel-invocation` | `otel-execution` | +| Java: `ServiceLoader` provider name | `otel-invocation` | `otel-execution` | ## Configure an application-owned provider From c3cc75526c4628c9f2cc8a1b3b2c951e85b11bc5 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 3 Nov 2026 12:41:10 -0700 Subject: [PATCH 04/31] docs: refresh OTel status and export guidance --- .../observability/opentelemetry.md | 66 ++++++++++++++++--- 1 file changed, 56 insertions(+), 10 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index b2344d3a..f40ea13c 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -134,6 +134,9 @@ and region: | OpenTelemetry community language layer | `/opt/otel-handler` | [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) | Enable active tracing to receive the durable backend's X-Ray trace context. +An OTLP-only backend might not receive the native Lambda spans referenced by +that context. See [missing parents in an OTLP backend](#missing-parents-in-an-otlp-backend) +when choosing your export path. For ADOT, add these properties to your existing SAM function and grant its role `AWSXRayDaemonWriteAccess` to export traces to CloudWatch through X-Ray: @@ -433,20 +436,26 @@ a backend can report failure without diagnostic details. === "Python" - Two open fixes refine this behavior: + Released OTel 2.0.0 can leave a failed operation's status as `UNSET` when no + error object is supplied. Two fixes are merged on `main` but are not included + in that release: - [PR #865](https://github.com/aws/aws-durable-execution-sdk-python/pull/865) marks `FAILED`, `TIMED_OUT`, and `STOPPED` operations as `ERROR` even when diagnostics are absent, and places terminal exception events at the span's - normalized end time. Without this fix, these outcomes can remain `UNSET` - when no error object is supplied. - - [PR #869](https://github.com/aws/aws-durable-execution-sdk-python/pull/869) - exports visible terminal updates from invocation input and checkpoint - notifications when cached-parent replay skips the completed child. It - deduplicates successful observations within an invocation, without promising - exactly-once delivery across retries or exporter failures. - - These changes require builds containing the PRs until they are merged and released. + normalized end time. + - [PR #901](https://github.com/aws/aws-durable-execution-sdk-python/pull/901) + emits only the checkpointed exception fields, without inventing a missing + type, message, or stack trace. It omits the exception event when neither a + type nor a message was stored; the span status still records the failure. + + [PR #869](https://github.com/aws/aws-durable-execution-sdk-python/pull/869) + remains open. It exports visible terminal updates from invocation input and + checkpoint notifications when cached-parent replay skips the completed child. + It deduplicates successful observations within an invocation, without promising + exactly-once delivery across retries or exporter failures. + + Use builds containing the respective changes until a release includes them. The plugins cannot reconstruct operation metadata that the service has pruned. === "Java" @@ -455,6 +464,9 @@ a backend can report failure without diagnostic details. diagnostics. Missing details do not produce an exception event. Execution-view operation exceptions use the known backend end timestamp when available; invocation-view exceptions retain normal OpenTelemetry event timing. + Since SDK/plugin 3.0.1, operation exception events report the stored original + error type when available, including errors from another language, while + retaining the throwable's message and stack diagnostics. ## Correlate logs and verify traces @@ -494,6 +506,40 @@ configuration, the layer's execution wrapper, and sampling. For the Java agent, also check `OTEL_JAVAAGENT_EXTENSIONS`. A missing `Workflow` can mean the execution has not finished or no terminal invocation exported it. A parent ID without a visible span can belong to the remote backend or an undelivered fallback root. +Check the full trace and its export path before treating it as missing SDK telemetry. + +### Missing parents in an OTLP backend + +With Lambda active tracing, a valid remote parent can identify a native Lambda +platform segment exported to X-Ray. The durable plugin preserves that externally +owned parent; it does not synthesize or re-export the segment. An OTLP-only backend +can therefore receive the durable spans and their shared trace ID without receiving +all of their ancestors. + +First query all spans for the trace ID, including vendor and application spans +without `durable.execution.arn`. An ARN-only filter can hide a parent that the +backend already stores. If a parent remains absent, compare its ID with the native +X-Ray segments and the raw OTLP export. A parent present only in X-Ray identifies +an export boundary. A parent present in the raw OTLP export but absent from a +backend query calls for checking ingestion, filters, and the UI separately. + +`DASH0_XRAY_TRACES_ENABLED=true` alone does not import native X-Ray platform +segments into the OTLP stream. The [validation record in issue #297](https://github.com/aws/aws-durable-execution-docs/issues/297) +covers JavaScript source `721217ae`, Dash0 Node layer v26, both plugin views, and +multiple resumes with that setting off and on. It establishes the export boundary +from raw captures and X-Ray data; it does not qualify Dash0 backend/UI behavior or +Python and Java combinations. + +If native Lambda platform spans are required, use an integration that receives both +application OTLP spans and their native X-Ray ancestors. Issue #297 also tracks a +proposed alternative: export a real caller span to the same OTLP backend, propagate +its actual trace context and sampling decision, and use Lambda `PassThrough` tracing +while retaining application instrumentation. That combination has not been validated +end to end and is not a supported workaround documented here. It also gives up native +Lambda X-Ray platform tracing. An arbitrary parent ID or matching trace IDs alone +cannot establish a complete parent tree. Qualification must verify the actual caller +and parents in the same backend across both views, resumes, chained invokes, and +sampled and unsampled executions. ## See also From 0a7966bd4387bc4e039136a85501f4c0e41b4a4d Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 3 Nov 2026 12:49:44 -0700 Subject: [PATCH 05/31] docs: split OTLP ancestor guidance for review --- .../observability/opentelemetry.md | 37 ------------------- 1 file changed, 37 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index f40ea13c..ccfb31c1 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -134,9 +134,6 @@ and region: | OpenTelemetry community language layer | `/opt/otel-handler` | [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) | Enable active tracing to receive the durable backend's X-Ray trace context. -An OTLP-only backend might not receive the native Lambda spans referenced by -that context. See [missing parents in an OTLP backend](#missing-parents-in-an-otlp-backend) -when choosing your export path. For ADOT, add these properties to your existing SAM function and grant its role `AWSXRayDaemonWriteAccess` to export traces to CloudWatch through X-Ray: @@ -506,40 +503,6 @@ configuration, the layer's execution wrapper, and sampling. For the Java agent, also check `OTEL_JAVAAGENT_EXTENSIONS`. A missing `Workflow` can mean the execution has not finished or no terminal invocation exported it. A parent ID without a visible span can belong to the remote backend or an undelivered fallback root. -Check the full trace and its export path before treating it as missing SDK telemetry. - -### Missing parents in an OTLP backend - -With Lambda active tracing, a valid remote parent can identify a native Lambda -platform segment exported to X-Ray. The durable plugin preserves that externally -owned parent; it does not synthesize or re-export the segment. An OTLP-only backend -can therefore receive the durable spans and their shared trace ID without receiving -all of their ancestors. - -First query all spans for the trace ID, including vendor and application spans -without `durable.execution.arn`. An ARN-only filter can hide a parent that the -backend already stores. If a parent remains absent, compare its ID with the native -X-Ray segments and the raw OTLP export. A parent present only in X-Ray identifies -an export boundary. A parent present in the raw OTLP export but absent from a -backend query calls for checking ingestion, filters, and the UI separately. - -`DASH0_XRAY_TRACES_ENABLED=true` alone does not import native X-Ray platform -segments into the OTLP stream. The [validation record in issue #297](https://github.com/aws/aws-durable-execution-docs/issues/297) -covers JavaScript source `721217ae`, Dash0 Node layer v26, both plugin views, and -multiple resumes with that setting off and on. It establishes the export boundary -from raw captures and X-Ray data; it does not qualify Dash0 backend/UI behavior or -Python and Java combinations. - -If native Lambda platform spans are required, use an integration that receives both -application OTLP spans and their native X-Ray ancestors. Issue #297 also tracks a -proposed alternative: export a real caller span to the same OTLP backend, propagate -its actual trace context and sampling decision, and use Lambda `PassThrough` tracing -while retaining application instrumentation. That combination has not been validated -end to end and is not a supported workaround documented here. It also gives up native -Lambda X-Ray platform tracing. An arbitrary parent ID or matching trace IDs alone -cannot establish a complete parent tree. Qualification must verify the actual caller -and parents in the same backend across both views, resumes, chained invokes, and -sampled and unsampled executions. ## See also From 8a80579e59ff1356bc0d36472860290fe0330d1d Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:00:03 -0700 Subject: [PATCH 06/31] docs: shorten OTel release and replay guidance --- .../observability/opentelemetry.md | 40 +++++++------------ 1 file changed, 14 insertions(+), 26 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index ccfb31c1..aa7798bb 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -74,7 +74,6 @@ runtime can leave spans missing. ### Invocation view -Use this view to see each invocation's work as the execution progresses. Completed work uses checkpointed results on replay. A resumed operation or replayed context can produce another span segment without rerunning a completed step's body. @@ -99,7 +98,6 @@ step's body. ### Execution view -Use this view to keep the durable operation hierarchy beneath `Workflow`. An operation that suspends is exported when it completes, using its durable start and end times. Its span links to the invocation that exports the completion; it does not accumulate links to every earlier invocation. @@ -433,27 +431,18 @@ a backend can report failure without diagnostic details. === "Python" - Released OTel 2.0.0 can leave a failed operation's status as `UNSET` when no - error object is supplied. Two fixes are merged on `main` but are not included - in that release: - - - [PR #865](https://github.com/aws/aws-durable-execution-sdk-python/pull/865) - marks `FAILED`, `TIMED_OUT`, and `STOPPED` operations as `ERROR` even when - diagnostics are absent, and places terminal exception events at the span's - normalized end time. - - [PR #901](https://github.com/aws/aws-durable-execution-sdk-python/pull/901) - emits only the checkpointed exception fields, without inventing a missing - type, message, or stack trace. It omits the exception event when neither a - type nor a message was stored; the span status still records the failure. + In OTel 2.0.0, a failed operation without an error object can remain `UNSET`. + [PR #865](https://github.com/aws/aws-durable-execution-sdk-python/pull/865) + fixes that status and aligns terminal exception timestamps with span ends. + [PR #901](https://github.com/aws/aws-durable-execution-sdk-python/pull/901) + preserves only checkpointed exception fields and omits events lacking both a + type and a message. Both fixes are merged on `main` but absent from OTel 2.0.0. [PR #869](https://github.com/aws/aws-durable-execution-sdk-python/pull/869) - remains open. It exports visible terminal updates from invocation input and - checkpoint notifications when cached-parent replay skips the completed child. - It deduplicates successful observations within an invocation, without promising - exactly-once delivery across retries or exporter failures. - - Use builds containing the respective changes until a release includes them. - The plugins cannot reconstruct operation metadata that the service has pruned. + remains open. It exports visible completion updates when cached-parent replay + skips a completed child, with deduplication within an invocation. It does not + guarantee exactly-once export. Until these fixes ship, use builds containing + the changes. Pruned operation metadata cannot be reconstructed. === "Java" @@ -498,11 +487,10 @@ and logs identify the active span. `Workflow` appears after terminal completion. Span links correlate the two views but do not change parent-child relationships; CloudWatch does not currently visualize these links. -If plugin spans are missing, check provider initialization warnings, exporter -configuration, the layer's execution wrapper, and sampling. For the Java agent, -also check `OTEL_JAVAAGENT_EXTENSIONS`. A missing `Workflow` can mean the execution -has not finished or no terminal invocation exported it. A parent ID without a -visible span can belong to the remote backend or an undelivered fallback root. +If spans are missing, check provider warnings, exporter configuration, the layer's +execution wrapper, and sampling. For Java agents, also check `OTEL_JAVAAGENT_EXTENSIONS`. +`Workflow` requires a terminal invocation to export it. An unresolved parent can +belong to the remote backend or an undelivered fallback root. ## See also From ab347b4c2afa2a832eb0b3a53588b8eb848d0194 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:19:53 -0700 Subject: [PATCH 07/31] docs: retain the plugin selection comparison --- .../observability/opentelemetry.md | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index aa7798bb..1fefbd69 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -58,12 +58,20 @@ instances cannot be registered with the 3.x factory API. Register one OTel view factory. The SDK creates a separate plugin instance for each invocation, while the provider can be shared across invocations. -| Aspect | Execution view | Invocation view | -| ------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------- | -| Top-level operation parent | `Workflow` | Current `Invocation` | -| Operation and attempt links | Current `Invocation` | `Workflow` | -| Operation spanning a suspension | One logical span, exported on completion | Separate segments across invocations; later segments link to the initial operation span | -| Best suited to | Inspecting the workflow hierarchy | Inspecting progress during long executions | +### Choosing a plugin + +For both views, your observability platform's quotas and limits apply. + +| Consideration | Execution view | Invocation view | +| -------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| Top-level operation parent | `Workflow` | Current `Invocation` | +| Correlation link | Operation or attempt to current `Invocation` | Operation or attempt to `Workflow` | +| Continuation link | Completed operation links only to the invocation that exports it | Later segment links to the initial logical operation span | +| Workflow export | When the SDK observes execution success or failure | When the SDK observes execution success or failure | +| Operation visibility | When the operation completes | At each invocation boundary | +| In-progress visibility | Limited until operations and the execution complete | Each invocation appears as it completes | +| Better for | Shorter executions and a workflow-centered hierarchy | Long executions and progress through individual invocations | +| Platform compatibility | Long-open spans can exceed ingestion time limits | Operation segments stay within an invocation; long Workflow spans still face ingestion limits | Nested operations remain under their child-context operation span; attempts remain under their operation. Both views export `Invocation` spans as invocations From e7ae89a0bb459963c8502a0be7debacb952a3a9b Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:20:30 -0700 Subject: [PATCH 08/31] docs: explain X-Ray-only ancestors in OTLP (#306) --- .../observability/opentelemetry.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 1fefbd69..10bafac0 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -140,6 +140,8 @@ and region: | OpenTelemetry community language layer | `/opt/otel-handler` | [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) | Enable active tracing to receive the durable backend's X-Ray trace context. +OTLP-only backends may lack the referenced native Lambda spans; see +[missing parents in an OTLP backend](#missing-parents-in-an-otlp-backend). For ADOT, add these properties to your existing SAM function and grant its role `AWSXRayDaemonWriteAccess` to export traces to CloudWatch through X-Ray: @@ -500,6 +502,34 @@ execution wrapper, and sampling. For Java agents, also check `OTEL_JAVAAGENT_EXT `Workflow` requires a terminal invocation to export it. An unresolved parent can belong to the remote backend or an undelivered fallback root. +### Missing parents in an OTLP backend + +With Lambda active tracing, a remote parent can be a native Lambda segment exported +only to X-Ray. The durable plugin preserves that external parent without re-exporting +it, so a shared trace ID does not guarantee a complete parent tree in an OTLP backend. + +To locate a missing parent: + +1. Query all spans for the trace ID. Filtering by `durable.execution.arn` can hide + vendor or application ancestors already stored in the backend. +1. Compare unresolved parent IDs with raw OTLP exports and native X-Ray segments. + A parent found only in X-Ray identifies the export boundary. +1. If the parent is in the OTLP export, check backend ingestion, query filters, + and UI rendering. + +`DASH0_XRAY_TRACES_ENABLED=true` does not import native X-Ray segments into OTLP. +[Issue #297](https://github.com/aws/aws-durable-execution-docs/issues/297) records +this limitation with JavaScript source `721217ae` and Dash0 Node layer v26 across +both views and multiple resumes, with X-Ray mode off and on. Those captures do not +qualify Dash0 backend/UI behavior or Python and Java combinations. + +For native Lambda platform spans, use an integration that receives both OTLP spans +and their X-Ray ancestors. The alternative tracked in #297 uses a real caller span +exported to the same backend, propagates its context and sampling decision, and +selects `PassThrough` while retaining application instrumentation. It gives up native +Lambda X-Ray platform tracing and remains unvalidated end to end. Do not treat it as +a supported workaround. + ## See also - [Plugins](plugins.md) From 546679c2f634083f33fcf063eb513916dfd7a45b Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 3 Nov 2026 13:23:02 -0700 Subject: [PATCH 09/31] docs: include sampling guidance in combined PR --- .../observability/opentelemetry.md | 43 ++++++++++++++++--- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 10bafac0..94fa6a70 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -382,7 +382,8 @@ another copy. `Workflow` carries duration and outcome; the fallback root carries neither. Remote backend spans are owned by the backend and are never exported by the plugin. -Configure sampling through the provider, or through `OTEL_TRACES_SAMPLER` and +Configure a [supported sampling mode](#supported-sampling-modes) through the provider, +or through `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` when using auto-configuration. Keep the plugin's sampler integration installed so durable spans share the resolved decision while other instrumentation retains its own policy. @@ -410,9 +411,40 @@ instrumentation retains its own policy. provider-builder and agent-extension paths, as the final sampler. Replacing it with a plain sampler removes the shared-decision guarantee. -Sampling decisions are not checkpointed across invocations. An explicit upstream -decision or deterministic trace-ID-based policy keeps them consistent across -resumes; a stateful custom sampler can choose differently on a later invocation. +### Supported sampling modes + +These rules apply to TypeScript core 3.0.0 / OTel 2.0.0, Python core 3.0.1 / OTel 2.0.0, +and Java SDK/plugin 3.0.1, in both views: + +- **Explicit upstream decision:** a valid `Sampled=0` or `Sampled=1` overrides the + local sampler for durable spans. Thus `Sampled=1` still exports durable spans + with local `always_off`, and `Sampled=0` drops them with local `always_on`. + Active tracing alone does not imply `Sampled=1`. +- **Otherwise:** use a deterministic policy based on the canonical trace ID, such + as `always_on`, `always_off`, or `traceidratio` with a fixed ratio. For + `traceidratio`, `OTEL_TRACES_SAMPLER_ARG=0.5` selects whole execution traces when + the local policy is consulted. + +Keep the policy, ratio, and any inherited upstream or ambient decision stable across +retries, resumes, and processes. Random, time-, load-, or mutable-state-dependent +policies are unsupported and can produce partial traces. The plugins do not +checkpoint sampling decisions; caching or application persistence does not make +another policy supported. + +The provider or auto-configuration layer resolves the environment settings, defaults, +and invalid values. They do not override an explicitly configured application +provider. Keep the plugin's sampler integration installed; other application and +ADOT spans retain their own policy. + +[Issue #295](https://github.com/aws/aws-durable-execution-docs/issues/295) proposes +letting the configured sampler decide whether to inherit upstream sampling, using +standard direct and `parentbased_*` semantics. That change is not available in the +versions above; follow the issue for runtime support and migration guidance. + +Sampling does not guarantee export or a complete parent tree: a durable span may be +sampled while its application parent is dropped, or its native X-Ray parent may be +absent from OTLP. See [missing parents](#missing-parents-in-an-otlp-backend) to distinguish +sampling, raw export, and backend/UI results. ## Span attributes and failures @@ -498,7 +530,8 @@ Span links correlate the two views but do not change parent-child relationships; CloudWatch does not currently visualize these links. If spans are missing, check provider warnings, exporter configuration, the layer's -execution wrapper, and sampling. For Java agents, also check `OTEL_JAVAAGENT_EXTENSIONS`. +execution wrapper, and [sampling](#supported-sampling-modes). For Java agents, also check +`OTEL_JAVAAGENT_EXTENSIONS`. `Workflow` requires a terminal invocation to export it. An unresolved parent can belong to the remote backend or an undelivered fallback root. From 56c6e7240a7060b707c8d48023732b9c56257d8a Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Wed, 4 Nov 2026 17:04:57 +0000 Subject: [PATCH 10/31] docs: use unified OTel factory initialization --- docs/sdk-reference/languages/java/index.md | 11 +- .../observability/opentelemetry.md | 103 ++++++++++-------- .../opentelemetry/execution-plugin.java | 4 +- .../opentelemetry/invocation-plugin.java | 4 +- .../opentelemetry/execution-plugin.py | 4 +- .../opentelemetry/invocation-plugin.py | 4 +- .../opentelemetry/execution-plugin.ts | 4 +- .../opentelemetry/invocation-plugin.ts | 4 +- 8 files changed, 77 insertions(+), 61 deletions(-) diff --git a/docs/sdk-reference/languages/java/index.md b/docs/sdk-reference/languages/java/index.md index dfdbc4e8..f753d1b4 100644 --- a/docs/sdk-reference/languages/java/index.md +++ b/docs/sdk-reference/languages/java/index.md @@ -208,18 +208,19 @@ return DurableConfig.builder() ### Load plugins dynamically -`withPlugins(...)` registers plugin instances in application code. Java can also load +`withPlugins(...)` registers plugin factories in application code. Java can also load plugin providers from the application class path. Set `DURABLE_EXECUTION_PLUGINS` to an ordered, comma-separated list of provider names: ```text -DURABLE_EXECUTION_PLUGINS=otel-invocation,com.example.audit +DURABLE_EXECUTION_PLUGINS=otel,com.example.audit ``` The SDK discovers `DurableExecutionPluginProvider` implementations with -`ServiceLoader`, but creates only the providers named in the environment variable. -Dynamically loaded plugins run first in the listed order, followed by plugins supplied -through `withPlugins(...)`. +`ServiceLoader` and selects the factories named in the environment variable. +It creates a fresh plugin for each invocation. Dynamically loaded plugins run first +in the listed order, followed by factories supplied through `withPlugins(...)`. +The `otel` provider selects the default [OpenTelemetry execution view](../../observability/opentelemetry.md#execution-view). For a Lambda layer, place the provider JAR under `java/lib`. The JAR must contain this service-provider configuration file: diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 94fa6a70..bffe144c 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -1,8 +1,9 @@ # OpenTelemetry The OpenTelemetry plugins trace durable operations across Lambda invocations. -Choose a workflow or invocation view, register its factory, and configure an -OpenTelemetry provider through a telemetry layer or application code. The +Create a factory for the execution view (the default) or select the invocation +view explicitly. Configure an OpenTelemetry provider through a telemetry layer +or application code. The provider owns exporters, resources, library instrumentation, and sampling policy. The plugins are available for TypeScript, Python, and Java. Go and C# do not yet @@ -10,14 +11,16 @@ provide this integration. ## Install the plugin -These examples use the current factory API: core SDK 3.x with OTel plugin 2.x -for TypeScript and Python, and SDK/plugin 3.x for Java. Upgrade the core and -plugin together, including copies supplied by Lambda layers. Older plugin -instances cannot be registered with the 3.x factory API. +These examples use the unified factory APIs. The version ranges below identify +compatible SDK families; they do not guarantee that a release contains the new +entry point. Use a build containing the change linked in your language's tab. +Upgrade the core and plugin together, including copies supplied by Lambda layers. === "TypeScript" - Requires Node.js 22 or later: + Requires Node.js 22 or later and a plugin build containing + [`createOtelPluginFactory`](https://github.com/aws/aws-durable-execution-sdk-js/pull/1171). + OTel 2.0.0 does not include this entry point. ```bash npm install @aws/durable-execution-sdk-js@^3 \ @@ -27,6 +30,10 @@ instances cannot be registered with the 3.x factory API. === "Python" + Use a plugin build containing + [`create_otel_plugin_factory`](https://github.com/aws/aws-durable-execution-sdk-python/pull/905). + OTel 2.0.0 does not include this entry point. + When a telemetry layer supplies the OpenTelemetry libraries: ```bash @@ -43,7 +50,9 @@ instances cannot be registered with the 3.x factory API. === "Java" - Use the same 3.x version as the core SDK: + Use the same 3.x version as the core SDK, containing + [`OtelPlugin.factory`](https://github.com/aws/aws-durable-execution-sdk-java/pull/873). + SDK/plugin 3.0.1 does not include this entry point. ```xml @@ -55,8 +64,9 @@ instances cannot be registered with the 3.x factory API. ## Choose and register a view -Register one OTel view factory. The SDK creates a separate plugin instance for -each invocation, while the provider can be shared across invocations. +Register one factory using the unified entry point for your language. Omitting +the mode selects the execution view. The SDK creates a separate plugin instance +for each invocation, while the provider can be shared across invocations. ### Choosing a plugin @@ -80,52 +90,53 @@ Workflow spans can cover long durations, so backend ingestion limits still apply to both views. Export and flush are best effort; a timeout, stop, or terminated runtime can leave spans missing. -### Invocation view +### Execution view -Completed work uses checkpointed results on replay. A resumed operation or -replayed context can produce another span segment without rerunning a completed -step's body. +The factory selects this view by default. An operation that suspends is exported +when it completes, using its durable +start and end times. Its span links to the invocation that exports the completion; +it does not accumulate links to every earlier invocation. === "TypeScript" ```typescript - --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts" + --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts" ``` === "Python" ```python - --8<-- "examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py" + --8<-- "examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py" ``` === "Java" ```java - --8<-- "examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java" + --8<-- "examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java" ``` -### Execution view +### Invocation view -An operation that suspends is exported when it completes, using its durable -start and end times. Its span links to the invocation that exports the completion; -it does not accumulate links to every earlier invocation. +Select invocation mode explicitly. Completed work uses checkpointed results on replay. A resumed operation or +replayed context can produce another span segment without rerunning a completed +step's body. === "TypeScript" ```typescript - --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts" + --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts" ``` === "Python" ```python - --8<-- "examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py" + --8<-- "examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py" ``` === "Java" ```java - --8<-- "examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java" + --8<-- "examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java" ``` ## Configure a telemetry layer @@ -215,11 +226,11 @@ code or environment configuration. The value follows each SDK's discovery mechanism: an importable module specifier or a registered provider name. -| Language and lookup | Invocation view | Execution view | -| ------------------------------------ | ---------------------------------------------------- | --------------------------------------------------- | -| TypeScript: Node.js module specifier | `@aws/durable-execution-sdk-js-otel/otel-invocation` | `@aws/durable-execution-sdk-js-otel/otel-execution` | -| Python: package entry-point name | `otel-invocation` | `otel-execution` | -| Java: `ServiceLoader` provider name | `otel-invocation` | `otel-execution` | +| Language and lookup | Execution view (default) | Invocation view | +| ------------------------------------ | ------------------------------------ | ---------------------------------------------------- | +| TypeScript: Node.js module specifier | `@aws/durable-execution-sdk-js-otel` | `@aws/durable-execution-sdk-js-otel/otel-invocation` | +| Python: package entry-point name | `otel` | `otel-invocation` | +| Java: `ServiceLoader` provider name | `otel` | `otel-invocation` | ## Configure an application-owned provider @@ -234,11 +245,11 @@ listening on loopback port 4318. the durable ID-generator factory; pass its result to the provider constructor. ```typescript - import { createInvocationOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; + import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { BatchSpanProcessor, NodeTracerProvider } from "@opentelemetry/sdk-trace-node"; - const pluginFactory = createInvocationOtelPluginFactory({ + const pluginFactory = createOtelPluginFactory({ tracerProviderFactory: (createIdGenerator) => { const provider = new NodeTracerProvider({ idGenerator: createIdGenerator(), @@ -252,7 +263,7 @@ listening on loopback port 4318. }); ``` - Use `createExecutionOtelPluginFactory` for the execution view. + Set `mode: "invocation"` in the same options object to select the invocation view. === "Python" @@ -264,7 +275,7 @@ listening on loopback port 4318. from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from aws_durable_execution_sdk_python_otel import ( - InvocationOtelPluginFactory, + create_otel_plugin_factory, OtelPluginConfig, ) @@ -273,12 +284,12 @@ listening on loopback port 4318. OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces") )) trace.set_tracer_provider(provider) - plugin_factory = InvocationOtelPluginFactory( + plugin_factory = create_otel_plugin_factory( OtelPluginConfig(tracer_provider=provider) ) ``` - Use `ExecutionOtelPluginFactory` for the execution view. + Pass `mode="invocation"` after the configuration argument to select the invocation view. === "Java" @@ -290,27 +301,28 @@ listening on loopback port 4318. import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter; import io.opentelemetry.sdk.trace.SdkTracerProvider; import io.opentelemetry.sdk.trace.export.BatchSpanProcessor; - import software.amazon.lambda.durable.otel.InvocationOtelPlugin; + import software.amazon.lambda.durable.otel.OtelPlugin; var exporter = OtlpHttpSpanExporter.builder() .setEndpoint("http://localhost:4318/v1/traces") .build(); - var pluginFactory = InvocationOtelPlugin.factory( + var pluginFactory = OtelPlugin.factory( SdkTracerProvider.builder() .addSpanProcessor(BatchSpanProcessor.builder(exporter).build())); ``` - Use `ExecutionOtelPlugin.factory` for the execution view. + Pass `OtelPlugin.Mode.INVOCATION` as the first argument to select the invocation view. ### Plugin options === "TypeScript" - Pass `OtelPluginConfig` to either `createInvocationOtelPluginFactory(config)` - or `createExecutionOtelPluginFactory(config)`. + Pass `OtelPluginFactoryOptions` to `createOtelPluginFactory(options)`. These + options extend `OtelPluginConfig` with the view selection. | Option | Default | Purpose | | ----------------------- | ------------------------------ | ----------------------------------------------------------------- | + | `mode` | `"execution"` | Select `"execution"` or `"invocation"` | | `tracerProviderFactory` | Global provider | Build an application-owned provider | | `contextExtractor` | `xRayContextExtractor` | Extract this execution's upstream context | | `instrumentationName` | `aws-durable-execution-sdk-js` | Instrumentation scope | @@ -326,8 +338,9 @@ listening on loopback port 4318. === "Python" - Pass `OtelPluginConfig` to `InvocationOtelPluginFactory(config)` or - `ExecutionOtelPluginFactory(config)`. + Pass `OtelPluginConfig` to `create_otel_plugin_factory(config)`. Select the + invocation view with `create_otel_plugin_factory(config, mode="invocation")`; + `mode` is a keyword argument, separate from `OtelPluginConfig`. | Option | Default | Purpose | | -------------------- | ---------------------------------- | ------------------------------------------------ | @@ -342,8 +355,10 @@ listening on loopback port 4318. === "Java" - Both plugins expose `factory()`, `factory(config)`, `factory(providerBuilder)`, - and `factory(providerBuilder, config)`. Build `OtelPluginConfig` with: + `OtelPlugin` exposes `factory()`, `factory(config)`, `factory(providerBuilder)`, + and `factory(providerBuilder, config)` for the execution view. Each overload + also accepts `OtelPlugin.Mode.INVOCATION` as its first argument. Build + `OtelPluginConfig` with: | Builder method | Default | Purpose | | -------------------------- | -------------------------------- | ------------------------------ | diff --git a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java index 162c1651..6ee6922e 100644 --- a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java +++ b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java @@ -3,12 +3,12 @@ import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.DurableContext; import software.amazon.lambda.durable.DurableHandler; -import software.amazon.lambda.durable.otel.ExecutionOtelPlugin; +import software.amazon.lambda.durable.otel.OtelPlugin; public class ExecutionOtelExample extends DurableHandler, String> { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(ExecutionOtelPlugin.factory()).build(); + return DurableConfig.builder().withPlugins(OtelPlugin.factory()).build(); } @Override diff --git a/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java b/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java index b00e4e15..2949cb3c 100644 --- a/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java +++ b/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java @@ -1,12 +1,12 @@ import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.DurableContext; import software.amazon.lambda.durable.DurableHandler; -import software.amazon.lambda.durable.otel.InvocationOtelPlugin; +import software.amazon.lambda.durable.otel.OtelPlugin; public class InvocationOtelExample extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(InvocationOtelPlugin.factory()).build(); + return DurableConfig.builder().withPlugins(OtelPlugin.factory(OtelPlugin.Mode.INVOCATION)).build(); } @Override diff --git a/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py b/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py index 151ac2af..33329994 100644 --- a/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py +++ b/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py @@ -1,9 +1,9 @@ from aws_durable_execution_sdk_python import DurableContext, durable_execution from aws_durable_execution_sdk_python.config import Duration -from aws_durable_execution_sdk_python_otel import ExecutionOtelPluginFactory +from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory -@durable_execution(plugins=[ExecutionOtelPluginFactory()]) +@durable_execution(plugins=[create_otel_plugin_factory()]) def handler(event: dict, context: DurableContext) -> str: data = context.step(lambda ctx: event["id"], name="fetch-data") context.wait(Duration.from_seconds(5), name="cooldown") diff --git a/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py b/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py index 24b80ee9..502fabfc 100644 --- a/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py +++ b/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py @@ -1,7 +1,7 @@ from aws_durable_execution_sdk_python import DurableContext, durable_execution -from aws_durable_execution_sdk_python_otel import InvocationOtelPluginFactory +from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory -@durable_execution(plugins=[InvocationOtelPluginFactory()]) +@durable_execution(plugins=[create_otel_plugin_factory(mode="invocation")]) def handler(event: dict, context: DurableContext) -> str: return context.step(lambda ctx: "done", name="process") diff --git a/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts b/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts index ac28adb9..90d25c0f 100644 --- a/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts +++ b/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts @@ -1,5 +1,5 @@ import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; -import { createExecutionOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; +import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; export const handler = withDurableExecution( async (event: { id: string }, context: DurableContext) => { @@ -7,5 +7,5 @@ export const handler = withDurableExecution( await context.wait("cooldown", { seconds: 5 }); return await context.step("process", async () => `processed ${data}`); }, - { plugins: [createExecutionOtelPluginFactory()] }, + { plugins: [createOtelPluginFactory()] }, ); diff --git a/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts b/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts index 5dbdd413..336d2f85 100644 --- a/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts +++ b/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts @@ -1,9 +1,9 @@ import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; -import { createInvocationOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; +import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; export const handler = withDurableExecution( async (event, context: DurableContext) => { return await context.step("process", async () => "done"); }, - { plugins: [createInvocationOtelPluginFactory()] }, + { plugins: [createOtelPluginFactory({ mode: "invocation" })] }, ); From d5fae5304745dacde01feb225fe3c3bc351e3c24 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Wed, 4 Nov 2026 17:23:11 +0000 Subject: [PATCH 11/31] docs: fix configured OTel factory examples --- docs/sdk-reference/observability/opentelemetry.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index bffe144c..94514dcf 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -307,11 +307,12 @@ listening on loopback port 4318. .setEndpoint("http://localhost:4318/v1/traces") .build(); var pluginFactory = OtelPlugin.factory( + OtelPlugin.Mode.EXECUTION, SdkTracerProvider.builder() .addSpanProcessor(BatchSpanProcessor.builder(exporter).build())); ``` - Pass `OtelPlugin.Mode.INVOCATION` as the first argument to select the invocation view. + Use `OtelPlugin.Mode.INVOCATION` as the first argument to select the invocation view. ### Plugin options @@ -355,9 +356,10 @@ listening on loopback port 4318. === "Java" - `OtelPlugin` exposes `factory()`, `factory(config)`, `factory(providerBuilder)`, - and `factory(providerBuilder, config)` for the execution view. Each overload - also accepts `OtelPlugin.Mode.INVOCATION` as its first argument. Build + `OtelPlugin` exposes `factory()`, `factory(mode)`, `factory(mode, config)`, + `factory(mode, providerBuilder)`, and `factory(mode, providerBuilder, config)`. + Use `OtelPlugin.Mode.EXECUTION` or `OtelPlugin.Mode.INVOCATION` for `mode`. + Only `factory()` omits the mode and defaults to execution. Build `OtelPluginConfig` with: | Builder method | Default | Purpose | From cfef0e41892cd1801f6d97123c9d7b84fd3be0f1 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Wed, 4 Nov 2026 10:50:03 -0700 Subject: [PATCH 12/31] docs: correct attempt attributes and the one-view contract Review feedback on #302. The attempt row of the span-attribute table dropped `durable.operation.type` and `durable.operation.name`, which both SDKs set on attempt spans: Java's `ExecutionOtelPlugin` writes them on the attempt span builder, and Python builds attempt attributes from the shared `_operation_attributes`. Restore both. Warn that exactly one OTel view may be configured, covering two views in `DURABLE_EXECUTION_PLUGINS` and one there alongside a factory in handler code. Two names for the same view are not portable: Java builds a factory per provider and rejects `otel,otel-execution` at startup, while Python resolves both names to one object and logs that it skipped the duplicate. Also note the older `otel-execution` name alongside `otel`. Show `AWSXRayDaemonWriteAccess` in the telemetry-layer YAML rather than only in prose, and keep `AWSLambdaBasicDurableExecutionRolePolicy` visible so the additive `Policies` block is not read as a complete one. Open each view section with the parenting rule that names it, rather than leaving that only in the comparison table. Explain why the Python application-owned provider example calls `set_tracer_provider` beside an explicit `tracer_provider`. Drop the `Workflow export` comparison row, which had become identical in both columns and is stated in the paragraph below. --- .../observability/opentelemetry.md | 42 ++++++++++++++++--- 1 file changed, 37 insertions(+), 5 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 94514dcf..8653ee2a 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -77,7 +77,6 @@ For both views, your observability platform's quotas and limits apply. | Top-level operation parent | `Workflow` | Current `Invocation` | | Correlation link | Operation or attempt to current `Invocation` | Operation or attempt to `Workflow` | | Continuation link | Completed operation links only to the invocation that exports it | Later segment links to the initial logical operation span | -| Workflow export | When the SDK observes execution success or failure | When the SDK observes execution success or failure | | Operation visibility | When the operation completes | At each invocation boundary | | In-progress visibility | Limited until operations and the execution complete | Each invocation appears as it completes | | Better for | Shorter executions and a workflow-centered hierarchy | Long executions and progress through individual invocations | @@ -92,6 +91,7 @@ runtime can leave spans missing. ### Execution view +Operation spans are children of `Workflow` and link to the current `Invocation`. The factory selects this view by default. An operation that suspends is exported when it completes, using its durable start and end times. Its span links to the invocation that exports the completion; @@ -117,7 +117,9 @@ it does not accumulate links to every earlier invocation. ### Invocation view -Select invocation mode explicitly. Completed work uses checkpointed results on replay. A resumed operation or +Operation spans are children of the `Invocation` that emitted them and link to +`Workflow`, so each invocation's spans export as it ends. Select invocation mode +explicitly. Completed work uses checkpointed results on replay. A resumed operation or replayed context can produce another span segment without rerunning a completed step's body. @@ -153,8 +155,10 @@ and region: Enable active tracing to receive the durable backend's X-Ray trace context. OTLP-only backends may lack the referenced native Lambda spans; see [missing parents in an OTLP backend](#missing-parents-in-an-otlp-backend). -For ADOT, add these properties to your existing SAM function and grant its role -`AWSXRayDaemonWriteAccess` to export traces to CloudWatch through X-Ray: +For ADOT, add these properties to your existing SAM function. `Policies` is +additive: keep the `AWSLambdaBasicDurableExecutionRolePolicy` the function +already needs, and add `AWSXRayDaemonWriteAccess` so active tracing and X-Ray +export can write segments. ```yaml Layers: @@ -163,6 +167,9 @@ Environment: Variables: AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument Tracing: Active +Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy + - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess ``` For a community language layer exporting directly to an OTLP backend, use: @@ -179,6 +186,9 @@ Environment: OTEL_METRICS_EXPORTER: none OTEL_LOGS_EXPORTER: none Tracing: Active +Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy + - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess ``` Configure `OTEL_EXPORTER_OTLP_HEADERS` if your backend requires authentication. @@ -232,6 +242,23 @@ or a registered provider name. | Python: package entry-point name | `otel` | `otel-invocation` | | Java: `ServiceLoader` provider name | `otel` | `otel-invocation` | +Python and Java also accept the older `otel-execution` name for the execution +view. + +!!! warning "Configure exactly one OTel view" + + Registering two OTel factories fails validation, because both belong to the + `durable-otel-view` exclusive group. This applies to naming two views in + `DURABLE_EXECUTION_PLUGINS`, and to naming one there while also registering a + factory in handler code. + + Two names for the *same* view are not portable. Java builds a separate + factory per provider, so `DURABLE_EXECUTION_PLUGINS=otel,otel-execution` + fails at startup with `Conflicting plugin factories otel-execution + (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin)`. Python resolves both + names to one object, so it loads a single factory and logs that it skipped + the duplicate. Set one name in either language. + ## Configure an application-owned provider Configure the exporter and processors once, then register the resulting factory @@ -291,6 +318,11 @@ listening on loopback port 4318. Pass `mode="invocation"` after the configuration argument to select the invocation view. + `tracer_provider` is what the plugin uses, so `set_tracer_provider` is only + needed to point *other* instrumentation at the same provider. It takes effect + once per process and is ignored with a warning if a telemetry layer already + registered a global provider; omit it in that case. + === "Java" Add `io.opentelemetry:opentelemetry-exporter-otlp` to your dependencies. The @@ -470,7 +502,7 @@ sampling, raw export, and backend/UI results. | Workflow | `durable.execution.arn`, `durable.execution.status` | | Invocation | `durable.execution.arn`, `durable.invocation.status`, `durable.invocation.first` | | Operation | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status` | -| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.attempt.number`, `durable.attempt.outcome` | +| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.attempt.number`, `durable.attempt.outcome` | Operation types include `STEP`, `WAIT`, `CONTEXT`, `CHAINED_INVOKE`, and `CALLBACK`. A context has an operation span without a separate attempt span. An invocation From 269b991b4970be64943a9981d0ec6099181fec4f Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Wed, 4 Nov 2026 18:01:24 +0000 Subject: [PATCH 13/31] docs: clarify OTel permissions and exclusivity --- .../observability/opentelemetry.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 8653ee2a..b5693b7a 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -155,10 +155,12 @@ and region: Enable active tracing to receive the durable backend's X-Ray trace context. OTLP-only backends may lack the referenced native Lambda spans; see [missing parents in an OTLP backend](#missing-parents-in-an-otlp-backend). -For ADOT, add these properties to your existing SAM function. `Policies` is -additive: keep the `AWSLambdaBasicDurableExecutionRolePolicy` the function -already needs, and add `AWSXRayDaemonWriteAccess` so active tracing and X-Ray -export can write segments. +For ADOT, add these properties to your existing SAM function. When SAM creates +its execution role, merge the shown entries into the function's `Policies` list. +If the function specifies `Role`, SAM ignores `Policies`; attach +`AWSXRayDaemonWriteAccess` directly to that execution role and retain its existing +durable execution permissions. The same rule applies to the community-layer +example below. ```yaml Layers: @@ -247,15 +249,14 @@ view. !!! warning "Configure exactly one OTel view" - Registering two OTel factories fails validation, because both belong to the - `durable-otel-view` exclusive group. This applies to naming two views in - `DURABLE_EXECUTION_PLUGINS`, and to naming one there while also registering a - factory in handler code. + Registering two distinct OTel factories fails validation because the OTel + views share an exclusive group in each SDK. This applies to naming two views + in `DURABLE_EXECUTION_PLUGINS`, and to naming one there while also registering + a separate factory in handler code. Two names for the *same* view are not portable. Java builds a separate factory per provider, so `DURABLE_EXECUTION_PLUGINS=otel,otel-execution` - fails at startup with `Conflicting plugin factories otel-execution - (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin)`. Python resolves both + fails at startup with `Conflicting plugin factories otel-execution (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin)`. Python resolves both names to one object, so it loads a single factory and logs that it skipped the duplicate. Set one name in either language. From 977dffb812ed6cd93e41b9eacb11b350788254db Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Wed, 4 Nov 2026 11:50:25 -0700 Subject: [PATCH 14/31] docs: wrap the OTel alias conflict error in a code block Review feedback on #302. `269b991b` inlined the Java startup error as a code span, leaving a 146-column line in a file the repo formats at `wrap = 88`. Move the error into a fenced `text` block, like the other quoted tool output on this page, so the surrounding prose reflows normally. No wording changes. The exclusive group name is deliberately absent: Java uses `durable-otel-view` and Python uses `aws-durable-execution-otel-view`, so naming one would be wrong for the other. --- docs/sdk-reference/observability/opentelemetry.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index b5693b7a..51a94414 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -256,9 +256,14 @@ view. Two names for the *same* view are not portable. Java builds a separate factory per provider, so `DURABLE_EXECUTION_PLUGINS=otel,otel-execution` - fails at startup with `Conflicting plugin factories otel-execution (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin)`. Python resolves both - names to one object, so it loads a single factory and logs that it skipped - the duplicate. Set one name in either language. + fails at startup: + + ```text + Conflicting plugin factories otel-execution (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin) + ``` + + Python resolves both names to one object, so it loads a single factory and + logs that it skipped the duplicate. Set one name in either language. ## Configure an application-owned provider From 3ec8f65b60e7c1af714ea3764540637cd67f1d5f Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Fri, 6 Nov 2026 14:25:03 -0700 Subject: [PATCH 15/31] docs: simplify Java plugin version wording --- docs/sdk-reference/observability/opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 51a94414..49a4778e 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -50,7 +50,7 @@ Upgrade the core and plugin together, including copies supplied by Lambda layers === "Java" - Use the same 3.x version as the core SDK, containing + Use the same version as the core SDK, containing [`OtelPlugin.factory`](https://github.com/aws/aws-durable-execution-sdk-java/pull/873). SDK/plugin 3.0.1 does not include this entry point. From 3f871d3eb3b7ff49df520245499acd27e81db7a4 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 22:40:50 +0000 Subject: [PATCH 16/31] docs(otel): restore span hierarchy diagrams --- .../observability/opentelemetry.md | 53 ++++++++++++++++++- 1 file changed, 51 insertions(+), 2 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 49a4778e..4afaea0c 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -89,9 +89,14 @@ Workflow spans can cover long durations, so backend ingestion limits still apply to both views. Export and flush are best effort; a timeout, stop, or terminated runtime can leave spans missing. +The diagrams below show an execution with a valid remote backend parent and +ambient Lambda spans in the same trace. See [trace identity and sampling](#trace-identity-and-sampling) +for other parent contexts. The tree shows parent-child relationships; arrows +show span links. + ### Execution view -Operation spans are children of `Workflow` and link to the current `Invocation`. +Top-level operation spans are children of `Workflow` and link to the current `Invocation`. The factory selects this view by default. An operation that suspends is exported when it completes, using its durable start and end times. Its span links to the invocation that exports the completion; @@ -115,9 +120,28 @@ it does not accumulate links to every earlier invocation. --8<-- "examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java" ``` +For an execution that suspends at `cooldown` and resumes in a second invocation, +the trace resembles: + +```text +Remote backend server span +|-- Workflow (exported on terminal status) +| |-- Operation: fetch-data (STEP) -> link to Invocation #1 +| | `-- Attempt: fetch-data attempt 1 -> link to Invocation #1 +| |-- Operation: cooldown (WAIT) -> link to Invocation #2 +| `-- Operation: process (STEP) -> link to Invocation #2 +| `-- Attempt: process attempt 1 -> link to Invocation #2 +|-- Ambient Lambda span #1 +| `-- Invocation #1 +`-- Ambient Lambda span #2 + `-- Invocation #2 +``` + +The links to `Invocation` do not make invocation spans children of `Workflow`. + ### Invocation view -Operation spans are children of the `Invocation` that emitted them and link to +Top-level operation spans are children of the `Invocation` that emitted them and link to `Workflow`, so each invocation's spans export as it ends. Select invocation mode explicitly. Completed work uses checkpointed results on replay. A resumed operation or replayed context can produce another span segment without rerunning a completed @@ -141,6 +165,31 @@ step's body. --8<-- "examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java" ``` +The same execution, using invocation view, produces this hierarchy: + +```text +Remote backend server span +|-- Workflow (exported on terminal status) +|-- Ambient Lambda span #1 +| `-- Invocation #1 +| |-- Operation: fetch-data (STEP) -> link to Workflow +| | `-- Attempt: fetch-data attempt 1 +| | `-> link to Workflow +| `-- Operation: cooldown (WAIT) -> link to Workflow +`-- Ambient Lambda span #2 + `-- Invocation #2 + |-- Operation: cooldown (WAIT) -> links to initial cooldown + | span and Workflow + `-- Operation: process (STEP) -> link to Workflow + `-- Attempt: process attempt 1 -> link to Workflow +``` + +An operation that crosses an invocation boundary can produce more than one +span. A continuation segment uses a new span ID and links to the +initial logical operation span, which uses the deterministic operation span ID. +It also retains the `Workflow` link. CloudWatch does not yet visualize span +links. + ## Configure a telemetry layer A language auto-instrumentation layer registers the global provider used by the From 4ae0aa08ac27754d87473baddd765ee9414ef80c Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 22:53:29 +0000 Subject: [PATCH 17/31] docs(otel): consolidate community layer setup --- .../observability/opentelemetry.md | 96 +++++++++++-------- 1 file changed, 54 insertions(+), 42 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 4afaea0c..6f57096f 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -201,15 +201,16 @@ and region: | AWS Distro for OpenTelemetry (ADOT) | `/opt/otel-instrument` | [ADOT Lambda layers](https://aws-otel.github.io/docs/getting-started/lambda) | | OpenTelemetry community language layer | `/opt/otel-handler` | [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) | -Enable active tracing to receive the durable backend's X-Ray trace context. -OTLP-only backends may lack the referenced native Lambda spans; see -[missing parents in an OTLP backend](#missing-parents-in-an-otlp-backend). -For ADOT, add these properties to your existing SAM function. When SAM creates -its execution role, merge the shown entries into the function's `Policies` list. -If the function specifies `Role`, SAM ignores `Policies`; attach -`AWSXRayDaemonWriteAccess` directly to that execution role and retain its existing -durable execution permissions. The same rule applies to the community-layer -example below. +Add the following entries under `Properties` in your existing +`AWS::Serverless::Function`. Merge them with the function's existing layers, +environment variables, and policies. SAM applies `Policies` only when it creates +the execution role. If you specify `Role`, attach the required permissions +directly to that role. + +### ADOT layer + +For ADOT with X-Ray export, enable `Active` tracing and grant +`AWSXRayDaemonWriteAccess` alongside the durable execution permissions. ```yaml Layers: @@ -223,7 +224,18 @@ Policies: - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess ``` -For a community language layer exporting directly to an OTLP backend, use: +### Community language layer + +For a community language layer exporting directly to an OTLP backend, use +[`PassThrough` tracing](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html#services-xray-default). +Lambda passes the upstream X-Ray trace header to the +function without emitting native Lambda X-Ray segments. The layer and durable +plugin still provide application instrumentation. + +Propagate the caller's trace context and sampling decision. Without an upstream +header, Lambda generates a header with `Sampled=0`; the documented plugins honor +that decision even with a local `always_on` sampler. See +[supported sampling modes](#supported-sampling-modes). ```yaml Layers: @@ -236,13 +248,37 @@ Environment: OTEL_EXPORTER_OTLP_ENDPOINT: https:// OTEL_METRICS_EXPORTER: none OTEL_LOGS_EXPORTER: none -Tracing: Active +Tracing: PassThrough Policies: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess ``` Configure `OTEL_EXPORTER_OTLP_HEADERS` if your backend requires authentication. +OTLP export does not require X-Ray write permissions. SAM still adds an X-Ray +policy when it creates a role for `Tracing: PassThrough`. + +With `Active` tracing, a durable span can reference a native Lambda parent sent +only to X-Ray. The plugin preserves that parent without re-exporting it, so a +shared trace ID does not guarantee a complete parent tree in an OTLP backend. +If you need native Lambda platform spans, use an integration that receives both +OTLP spans and their X-Ray ancestors. + +`PassThrough` does not supply missing caller spans. For a complete upstream +hierarchy, export the real caller span to the same backend and propagate its +trace ID, span ID, and sampling decision through `X-Amzn-Trace-Id` when using the +default plugin extractors. Preserve this propagation across downstream durable +invokes. The complete caller-to-OTLP setup remains unvalidated end to end; +[Issue #297](https://github.com/aws/aws-durable-execution-docs/issues/297) tracks +its prerequisites, tested-version scope, and remaining validation. + +If parents are missing, query all spans for the trace ID. Filtering by +`durable.execution.arn` can hide caller, vendor, or application ancestors. +Compare unresolved parent IDs with raw OTLP exports and, when using `Active`, +native X-Ray segments. A parent found only in X-Ray identifies the export +boundary. If the parent appears in the OTLP export, check backend ingestion, +query filters, and UI rendering. For Dash0, +`DASH0_XRAY_TRACES_ENABLED=true` does not import native X-Ray segments into OTLP. + A collector extension is optional for local processing or additional exporters; follow the [collector setup guide](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector) for its layer, receiver, and `OPENTELEMETRY_COLLECTOR_CONFIG_URI` configuration. @@ -251,6 +287,10 @@ A collector alone does not initialize a language provider. For that setup, leave shown below. For collector export to CloudWatch, enable Transaction Search in your account. +### Provider initialization + +These requirements apply to both layer choices. + === "TypeScript" The default factory resolves the global provider at invocation start. @@ -547,8 +587,8 @@ versions above; follow the issue for runtime support and migration guidance. Sampling does not guarantee export or a complete parent tree: a durable span may be sampled while its application parent is dropped, or its native X-Ray parent may be -absent from OTLP. See [missing parents](#missing-parents-in-an-otlp-backend) to distinguish -sampling, raw export, and backend/UI results. +absent from OTLP. See the [community layer setup](#community-language-layer) to +distinguish sampling, raw export, and backend/UI results. ## Span attributes and failures @@ -639,34 +679,6 @@ execution wrapper, and [sampling](#supported-sampling-modes). For Java agents, a `Workflow` requires a terminal invocation to export it. An unresolved parent can belong to the remote backend or an undelivered fallback root. -### Missing parents in an OTLP backend - -With Lambda active tracing, a remote parent can be a native Lambda segment exported -only to X-Ray. The durable plugin preserves that external parent without re-exporting -it, so a shared trace ID does not guarantee a complete parent tree in an OTLP backend. - -To locate a missing parent: - -1. Query all spans for the trace ID. Filtering by `durable.execution.arn` can hide - vendor or application ancestors already stored in the backend. -1. Compare unresolved parent IDs with raw OTLP exports and native X-Ray segments. - A parent found only in X-Ray identifies the export boundary. -1. If the parent is in the OTLP export, check backend ingestion, query filters, - and UI rendering. - -`DASH0_XRAY_TRACES_ENABLED=true` does not import native X-Ray segments into OTLP. -[Issue #297](https://github.com/aws/aws-durable-execution-docs/issues/297) records -this limitation with JavaScript source `721217ae` and Dash0 Node layer v26 across -both views and multiple resumes, with X-Ray mode off and on. Those captures do not -qualify Dash0 backend/UI behavior or Python and Java combinations. - -For native Lambda platform spans, use an integration that receives both OTLP spans -and their X-Ray ancestors. The alternative tracked in #297 uses a real caller span -exported to the same backend, propagates its context and sampling decision, and -selects `PassThrough` while retaining application instrumentation. It gives up native -Lambda X-Ray platform tracing and remains unvalidated end to end. Do not treat it as -a supported workaround. - ## See also - [Plugins](plugins.md) From ee599e73d4a467ce6f6c757968d07dfbde4ec7d3 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 22:57:01 +0000 Subject: [PATCH 18/31] docs(otel): describe Python 2.1.0 behavior --- .../observability/opentelemetry.md | 21 ++++++++----------- 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 6f57096f..b19721dc 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -617,18 +617,15 @@ a backend can report failure without diagnostic details. === "Python" - In OTel 2.0.0, a failed operation without an error object can remain `UNSET`. - [PR #865](https://github.com/aws/aws-durable-execution-sdk-python/pull/865) - fixes that status and aligns terminal exception timestamps with span ends. - [PR #901](https://github.com/aws/aws-durable-execution-sdk-python/pull/901) - preserves only checkpointed exception fields and omits events lacking both a - type and a message. Both fixes are merged on `main` but absent from OTel 2.0.0. - - [PR #869](https://github.com/aws/aws-durable-execution-sdk-python/pull/869) - remains open. It exports visible completion updates when cached-parent replay - skips a completed child, with deduplication within an invocation. It does not - guarantee exactly-once export. Until these fixes ship, use builds containing - the changes. Pruned operation metadata cannot be reconstructed. + `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without an + error object. Exception events preserve the checkpointed type, message, and + stack trace; missing fields are omitted. No exception event is emitted when + both type and message are absent. Terminal operation and attempt exception + events use the span end timestamp. + + Completion spans can be missing when replay returns a cached parent result + without visiting its children, even if their operation metadata is still + visible. Pruned operation metadata cannot be reconstructed. === "Java" From 3a88b7f5b2165a72a8d270cba9b1b40183dc3bc5 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 23:16:10 +0000 Subject: [PATCH 19/31] docs(otel): clarify sampling version scope --- docs/sdk-reference/observability/opentelemetry.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index b19721dc..024d2783 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -557,8 +557,9 @@ instrumentation retains its own policy. ### Supported sampling modes -These rules apply to TypeScript core 3.0.0 / OTel 2.0.0, Python core 3.0.1 / OTel 2.0.0, -and Java SDK/plugin 3.0.1, in both views: +These rules apply to TypeScript core 3.0.0 / OTel 2.1.0, Python core 3.0.2 / OTel 2.1.0, +and Java SDK/plugin 3.1.0, in both views. These versions include the unified +factory APIs used by this guide. - **Explicit upstream decision:** a valid `Sampled=0` or `Sampled=1` overrides the local sampler for durable spans. Thus `Sampled=1` still exports durable spans @@ -580,10 +581,11 @@ and invalid values. They do not override an explicitly configured application provider. Keep the plugin's sampler integration installed; other application and ADOT spans retain their own policy. +Sampling behavior may change in future SDK/plugin releases. [Issue #295](https://github.com/aws/aws-durable-execution-docs/issues/295) proposes letting the configured sampler decide whether to inherit upstream sampling, using -standard direct and `parentbased_*` semantics. That change is not available in the -versions above; follow the issue for runtime support and migration guidance. +standard direct and `parentbased_*` semantics. That proposal is not implemented in +the versions listed above. Check the release notes and this section when upgrading. Sampling does not guarantee export or a complete parent tree: a durable span may be sampled while its application parent is dropped, or its native X-Ray parent may be From e1ec159c8d853ab22fb3e0c1d81f77916dc0cf0d Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 16:21:54 -0700 Subject: [PATCH 20/31] docs(otel): clarify span attribute availability Explain when optional span attributes are emitted and scope TypeScript completion export guidance to newly observed external updates. --- docs/sdk-reference/observability/opentelemetry.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 024d2783..edb49404 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -594,6 +594,9 @@ distinguish sampling, raw export, and backend/UI results. ## Span attributes and failures +Optional attributes such as `durable.operation.name` and +`durable.operation.subtype` are emitted only when supplied. + | Span | Main attributes | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Workflow | `durable.execution.arn`, `durable.execution.status` | @@ -611,9 +614,10 @@ a backend can report failure without diagnostic details. === "TypeScript" `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without - diagnostics. The plugins export visible completion updates even when replay skips the - operation's body. They need the operation metadata to do so. When completed - child contexts would otherwise prune that metadata, configure + diagnostics. The plugins export newly observed external completion updates + for waits, callbacks, and invokes when their metadata is visible, even if replay + skips the operation's body. When completed child contexts would otherwise + prune that metadata, configure `pluginsConfig.childOperationsDepth` as described in [operation detail](telemetry/exporter.md#operation-detail). From 84e178645c9ce64b93fb884925bf843b76d82971 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 23:25:05 +0000 Subject: [PATCH 21/31] docs(otel): document aliases by language --- .../observability/opentelemetry.md | 29 +++++++++++++------ 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index edb49404..84596ea5 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -333,9 +333,6 @@ or a registered provider name. | Python: package entry-point name | `otel` | `otel-invocation` | | Java: `ServiceLoader` provider name | `otel` | `otel-invocation` | -Python and Java also accept the older `otel-execution` name for the execution -view. - !!! warning "Configure exactly one OTel view" Registering two distinct OTel factories fails validation because the OTel @@ -343,16 +340,30 @@ view. in `DURABLE_EXECUTION_PLUGINS`, and to naming one there while also registering a separate factory in handler code. - Two names for the *same* view are not portable. Java builds a separate - factory per provider, so `DURABLE_EXECUTION_PLUGINS=otel,otel-execution` - fails at startup: +=== "TypeScript" + + `@aws/durable-execution-sdk-js-otel/otel-execution` selects the same execution + view as the bare `@aws/durable-execution-sdk-js-otel` package name. Configure + only one of these entry points in `DURABLE_EXECUTION_PLUGINS`. + +=== "Python" + + `otel-execution` is an alias for the default `otel` execution view. Both names + resolve to the same factory object. If both appear in + `DURABLE_EXECUTION_PLUGINS`, the SDK loads one factory and logs a warning that + it skipped the duplicate. Configure one name. + +=== "Java" + + `otel-execution` also selects the execution view. Each provider name creates + a separate factory, so `DURABLE_EXECUTION_PLUGINS=otel-execution,otel` fails + at startup: ```text - Conflicting plugin factories otel-execution (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin) + Conflicting plugin factories otel-execution (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin) in exclusive group 'durable-otel-view'. Configure only one plugin from this group. ``` - Python resolves both names to one object, so it loads a single factory and - logs that it skipped the duplicate. Set one name in either language. + Configure one provider name. ## Configure an application-owned provider From bfddbf8c3659bd0b35897d90f79264badad024ed Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 23:31:37 +0000 Subject: [PATCH 22/31] docs(otel): shorten community layer guidance --- .../observability/opentelemetry.md | 24 +++---------------- 1 file changed, 3 insertions(+), 21 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 84596ea5..78d03e64 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -267,25 +267,7 @@ OTLP spans and their X-Ray ancestors. hierarchy, export the real caller span to the same backend and propagate its trace ID, span ID, and sampling decision through `X-Amzn-Trace-Id` when using the default plugin extractors. Preserve this propagation across downstream durable -invokes. The complete caller-to-OTLP setup remains unvalidated end to end; -[Issue #297](https://github.com/aws/aws-durable-execution-docs/issues/297) tracks -its prerequisites, tested-version scope, and remaining validation. - -If parents are missing, query all spans for the trace ID. Filtering by -`durable.execution.arn` can hide caller, vendor, or application ancestors. -Compare unresolved parent IDs with raw OTLP exports and, when using `Active`, -native X-Ray segments. A parent found only in X-Ray identifies the export -boundary. If the parent appears in the OTLP export, check backend ingestion, -query filters, and UI rendering. For Dash0, -`DASH0_XRAY_TRACES_ENABLED=true` does not import native X-Ray segments into OTLP. - -A collector extension is optional for local processing or additional exporters; -follow the [collector setup guide](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector) -for its layer, receiver, and `OPENTELEMETRY_COLLECTOR_CONFIG_URI` configuration. -A collector alone does not initialize a language provider. For that setup, leave -`AWS_LAMBDA_EXEC_WRAPPER` unset and configure the provider in application code as -shown below. For collector export to CloudWatch, enable Transaction Search in -your account. +invokes. ### Provider initialization @@ -600,8 +582,8 @@ the versions listed above. Check the release notes and this section when upgradi Sampling does not guarantee export or a complete parent tree: a durable span may be sampled while its application parent is dropped, or its native X-Ray parent may be -absent from OTLP. See the [community layer setup](#community-language-layer) to -distinguish sampling, raw export, and backend/UI results. +absent from OTLP. See the [community layer setup](#community-language-layer) for +trace propagation and native X-Ray parents. ## Span attributes and failures From 5e52eae16893f21036ea626d6d21569433fd055a Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 23:37:07 +0000 Subject: [PATCH 23/31] docs(otel): remove unsupported W3C extractors --- docs/sdk-reference/observability/opentelemetry.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 78d03e64..c5b9f2b1 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -451,7 +451,6 @@ listening on loopback port 4318. | `enrichLogger` | `true` | Add trace fields to durable logs | | `flushTimeoutMillis` | `1000` | Total time budget for awaiting provider flushes at invocation end | - `w3cClientContextExtractor` reads `traceparent` from Lambda client context. The default X-Ray extractor reads invocation-local `xRayTraceId` when present, otherwise `_X_AMZN_TRACE_ID`. A flush timeout stops waiting; it does not cancel exports or synchronous @@ -471,7 +470,6 @@ listening on loopback port 4318. | `workflow_span_name` | `Workflow` | Workflow span name | | `enrich_logger` | `True` | Add trace fields through root logger handlers | - `w3c_client_context_extractor` is a placeholder that returns `None`. Use the default X-Ray extractor or implement a custom extractor. === "Java" From 7f66007b3cc3fc7707209f801ba52b8d5c2f9318 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 23:44:10 +0000 Subject: [PATCH 24/31] docs(otel): move Java troubleshooting into tab --- docs/sdk-reference/observability/opentelemetry.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index c5b9f2b1..2cbedf66 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -661,6 +661,9 @@ flag. Application-created spans on that trace can become the active context too. to inspect the hierarchy; the ungrouped X-Ray Segments Timeline cannot nest OTLP spans under the Lambda platform segment. + If spans are missing when using a Java agent, check + `OTEL_JAVAAGENT_EXTENSIONS`. + Test with an execution that waits and resumes. Confirm that its spans share a trace ID, invocations appear as they finish, completed steps do not run again, and logs identify the active span. `Workflow` appears after terminal completion. @@ -668,8 +671,7 @@ Span links correlate the two views but do not change parent-child relationships; CloudWatch does not currently visualize these links. If spans are missing, check provider warnings, exporter configuration, the layer's -execution wrapper, and [sampling](#supported-sampling-modes). For Java agents, also check -`OTEL_JAVAAGENT_EXTENSIONS`. +execution wrapper, and [sampling](#supported-sampling-modes). `Workflow` requires a terminal invocation to export it. An unresolved parent can belong to the remote backend or an undelivered fallback root. From 6ece9da324444a8aa75a37b845152499a2ded6e8 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Mon, 9 Nov 2026 23:55:06 +0000 Subject: [PATCH 25/31] docs(otel): add concise collector guidance --- docs/sdk-reference/observability/opentelemetry.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 2cbedf66..83c90c87 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -269,6 +269,10 @@ trace ID, span ID, and sampling decision through `X-Amzn-Trace-Id` when using th default plugin extractors. Preserve this propagation across downstream durable invokes. +You can optionally export telemetry through a +[Collector extension](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector#configuration), +which does not initialize the OpenTelemetry provider required by the plugin. + ### Provider initialization These requirements apply to both layer choices. From 26d7a52fc7c456a9b0e67090b54c1f4e05bfef01 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 10 Nov 2026 04:33:25 +0000 Subject: [PATCH 26/31] docs(otel): restore copyable setup examples --- .../observability/opentelemetry.md | 60 ++++++++++++------- 1 file changed, 37 insertions(+), 23 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 83c90c87..518764cf 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -11,33 +11,27 @@ provide this integration. ## Install the plugin -These examples use the unified factory APIs. The version ranges below identify -compatible SDK families; they do not guarantee that a release contains the new -entry point. Use a build containing the change linked in your language's tab. +These examples use the unified factory APIs available in the versions below. Upgrade the core and plugin together, including copies supplied by Lambda layers. === "TypeScript" - Requires Node.js 22 or later and a plugin build containing - [`createOtelPluginFactory`](https://github.com/aws/aws-durable-execution-sdk-js/pull/1171). - OTel 2.0.0 does not include this entry point. + Requires Node.js 22 or later and OTel plugin 2.1.0 or later. ```bash - npm install @aws/durable-execution-sdk-js@^3 \ - @aws/durable-execution-sdk-js-otel@^2 \ + npm install @aws/durable-execution-sdk-js@^3.0.0 \ + @aws/durable-execution-sdk-js-otel@^2.1.0 \ @opentelemetry/api @opentelemetry/core @opentelemetry/sdk-trace-node ``` === "Python" - Use a plugin build containing - [`create_otel_plugin_factory`](https://github.com/aws/aws-durable-execution-sdk-python/pull/905). - OTel 2.0.0 does not include this entry point. + Requires OTel plugin 2.1.0 or later. When a telemetry layer supplies the OpenTelemetry libraries: ```bash - pip install "aws-durable-execution-sdk-python-otel>=2,<3" + pip install "aws-durable-execution-sdk-python-otel>=2.1.0,<3" ``` The package installs a compatible core SDK but leaves OpenTelemetry libraries @@ -45,20 +39,18 @@ Upgrade the core and plugin together, including copies supplied by Lambda layers `standalone` extra instead: ```bash - pip install "aws-durable-execution-sdk-python-otel[standalone]>=2,<3" + pip install "aws-durable-execution-sdk-python-otel[standalone]>=2.1.0,<3" ``` === "Java" - Use the same version as the core SDK, containing - [`OtelPlugin.factory`](https://github.com/aws/aws-durable-execution-sdk-java/pull/873). - SDK/plugin 3.0.1 does not include this entry point. + Use matching core SDK and plugin versions, at least 3.1.0. ```xml software.amazon.lambda.durable aws-durable-execution-sdk-java-plugin-otel - ${durable.sdk.version} + 3.1.0 ``` @@ -360,8 +352,14 @@ listening on loopback port 4318. === "TypeScript" - Install `@opentelemetry/exporter-trace-otlp-http`. The provider factory receives - the durable ID-generator factory; pass its result to the provider constructor. + Install `@opentelemetry/exporter-trace-otlp-http`: + + ```bash + npm install @opentelemetry/exporter-trace-otlp-http + ``` + + The provider factory receives the durable ID-generator factory; pass its + result to the provider constructor. ```typescript import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; @@ -408,7 +406,13 @@ listening on loopback port 4318. ) ``` - Pass `mode="invocation"` after the configuration argument to select the invocation view. + For invocation view, replace the factory call with: + + ```python + plugin_factory = create_otel_plugin_factory( + OtelPluginConfig(tracer_provider=provider), mode="invocation" + ) + ``` `tracer_provider` is what the plugin uses, so `set_tracer_provider` is only needed to point *other* instrumentation at the same provider. It takes effect @@ -417,9 +421,19 @@ listening on loopback port 4318. === "Java" - Add `io.opentelemetry:opentelemetry-exporter-otlp` to your dependencies. The - factory wraps the provider builder's ID generator and sampler, then builds - the provider once. + Add `io.opentelemetry:opentelemetry-exporter-otlp` to your dependencies, + using OpenTelemetry 1.67.0 to match plugin 3.1.0: + + ```xml + + io.opentelemetry + opentelemetry-exporter-otlp + 1.67.0 + + ``` + + The factory wraps the provider builder's ID generator and sampler, then + builds the provider once. ```java import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter; From 5bd995b29a6cd99ef7f2a6827dade3520d9deb90 Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 10 Nov 2026 13:13:35 -0700 Subject: [PATCH 27/31] docs: retain SAM structure in telemetry examples --- .../observability/opentelemetry.md | 57 +++++++++++-------- 1 file changed, 32 insertions(+), 25 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index f3456f7e..720953ee 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -193,9 +193,10 @@ and region: | AWS Distro for OpenTelemetry (ADOT) | `/opt/otel-instrument` | [ADOT Lambda layers](https://aws-otel.github.io/docs/getting-started/lambda) | | OpenTelemetry community language layer | `/opt/otel-handler` | [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) | -Add the following entries under `Properties` in your existing -`AWS::Serverless::Function`. Merge them with the function's existing layers, -environment variables, and policies. SAM applies `Policies` only when it creates +The following SAM snippets show the telemetry settings for an +`AWS::Serverless::Function` resource. Merge the entries under `Properties` with +your function's existing layers, environment variables, and policies. +SAM applies `Policies` only when it creates the execution role. If you specify `Role`, attach the required permissions directly to that role. @@ -205,15 +206,18 @@ For ADOT with X-Ray export, enable `Active` tracing and grant `AWSXRayDaemonWriteAccess` alongside the durable execution permissions. ```yaml -Layers: - - -Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument -Tracing: Active -Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess +MyFunction: + Type: AWS::Serverless::Function + Properties: + Layers: + - + Environment: + Variables: + AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument + Tracing: Active + Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy + - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess ``` ### Community language layer @@ -230,19 +234,22 @@ that decision even with a local `always_on` sampler. See [supported sampling modes](#supported-sampling-modes). ```yaml -Layers: - - -Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler - OTEL_TRACES_EXPORTER: otlp - OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf - OTEL_EXPORTER_OTLP_ENDPOINT: https:// - OTEL_METRICS_EXPORTER: none - OTEL_LOGS_EXPORTER: none -Tracing: PassThrough -Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy +MyFunction: + Type: AWS::Serverless::Function + Properties: + Layers: + - + Environment: + Variables: + AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler + OTEL_TRACES_EXPORTER: otlp + OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf + OTEL_EXPORTER_OTLP_ENDPOINT: https:// + OTEL_METRICS_EXPORTER: none + OTEL_LOGS_EXPORTER: none + Tracing: PassThrough + Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy ``` Configure `OTEL_EXPORTER_OTLP_HEADERS` if your backend requires authentication. From 9c49c029df9d6ca978c811dbcd85b099429d9b1b Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 10 Nov 2026 13:44:06 -0700 Subject: [PATCH 28/31] docs: clarify when custom providers are needed --- docs/sdk-reference/observability/opentelemetry.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 720953ee..f804b67a 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -352,6 +352,11 @@ or a registered provider name. ## Configure an application-owned provider +This setup is for advanced use cases that require direct control over the +OpenTelemetry provider. You do not need it when using a +[language auto-instrumentation layer](#configure-a-telemetry-layer): the layer +initializes the global provider used by the plugin's default factories. + Configure the exporter and processors once, then register the resulting factory as in the handler examples above. The plugin flushes at invocation end but does not take over provider shutdown. These examples send OTLP HTTP to a collector From 7ea0c7d290d48f7972e913eb5b2d69ab296d94eb Mon Sep 17 00:00:00 2001 From: Zhongke Chen <65260095+zhongkechen@users.noreply.github.com> Date: Tue, 10 Nov 2026 20:56:30 +0000 Subject: [PATCH 29/31] docs: clarify CloudWatch OTLP prerequisite --- docs/sdk-reference/observability/opentelemetry.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index f804b67a..859c616d 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -252,9 +252,16 @@ MyFunction: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy ``` +**Exporting to CloudWatch:** +[Enable Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html) +in the destination account and Region, and confirm that **Ingest OpenTelemetry spans** +shows **Enabled**. Otherwise, CloudWatch's OTLP endpoint rejects the spans. This setting +is not required when exporting directly to another observability provider. + Configure `OTEL_EXPORTER_OTLP_HEADERS` if your backend requires authentication. -OTLP export does not require X-Ray write permissions. SAM still adds an X-Ray -policy when it creates a role for `Tracing: PassThrough`. +Direct OTLP export to another observability provider does not require X-Ray +write permissions. SAM still adds an X-Ray policy when it creates a role for +`Tracing: PassThrough`. With `Active` tracing, a durable span can reference a native Lambda parent sent only to X-Ray. The plugin preserves that parent without re-exporting it, so a From e79890aa8a9333703569624b32b0e4ed9c563f53 Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Tue, 10 Nov 2026 20:56:26 +0000 Subject: [PATCH 30/31] docs: show trace fields in log examples Add TypeScript and Python examples that display the trace fields enabled by default. Use the Powertools adapter for per-call enrichment and safe formatter defaults for Python logs outside an invocation. Align the existing Powertools examples and logging guide with the adapter so they preserve durable metadata and plugin fields. --- docs/sdk-reference/observability/logging.md | 16 +++-- .../observability/opentelemetry.md | 61 ++++++++++++++++--- .../opentelemetry/log-correlation.py | 23 +++++++ .../observability/custom-logger.ts | 8 ++- .../opentelemetry/log-correlation.ts | 18 ++++++ .../observability/powertools-logger.ts | 14 +++-- 6 files changed, 121 insertions(+), 19 deletions(-) create mode 100644 examples/python/sdk-reference/observability/opentelemetry/log-correlation.py create mode 100644 examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts diff --git a/docs/sdk-reference/observability/logging.md b/docs/sdk-reference/observability/logging.md index 893b6dba..383c9bc6 100644 --- a/docs/sdk-reference/observability/logging.md +++ b/docs/sdk-reference/observability/logging.md @@ -547,7 +547,11 @@ You can change where the SDK's logger sends its entries. === "TypeScript" - Any object that implements `DurableLogger` works. Pass it to `configureLogger`. + Pass an object that implements `DurableLogger` to `configureLogger`. To + include execution metadata and plugin fields, the logger must also implement + `configureDurableLoggingContext`. Store the supplied context and read + `getDurableLogData()` and `getLogEnrichment?.()` on each log call. + `createPowertoolsLogger` handles this for a Powertools logger: ```typescript --8<-- "examples/typescript/sdk-reference/observability/custom-logger.ts" @@ -811,14 +815,18 @@ You can change where the SDK's logger sends its entries. ## Powertools for AWS Lambda -[Powertools for AWS Lambda](https://docs.aws.amazon.com/powertools/) provides a -structured logger that works as a drop-in replacement for the SDK's default logger. +[Powertools for AWS Lambda](https://docs.aws.amazon.com/powertools/) provides structured +logging that you can use with the SDK. === "TypeScript" The [Powertools for AWS Lambda (TypeScript)](https://docs.aws.amazon.com/powertools/typescript/latest/features/logger/) - `Logger` satisfies the `DurableLogger` interface. Pass it via `configureLogger`. + `Logger` works with `createPowertoolsLogger`. Pass the wrapped logger via + `configureLogger` so each entry includes execution metadata and fields added + by plugins, such as OpenTelemetry trace IDs. See + [log correlation](opentelemetry.md#correlate-logs-and-verify-traces) for an example + with the OpenTelemetry plugin. ```typescript --8<-- "examples/typescript/sdk-reference/observability/powertools-logger.ts" diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 859c616d..bf3fcbf0 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -679,17 +679,62 @@ flag. Application-created spans on that trace can become the active context too. === "TypeScript" - `enrichLogger` adds fields through the durable logger's `enrichLogContext` - hook. In execution view, handler code between operations uses `Workflow`; - invocation view uses `Invocation`. Step bodies use their attempt span. - Custom logger adapters must forward the enriched context. + `enrichLogger` defaults to `true`. The default logger includes trace fields + when you call `context.logger.info()` or another log method. + + For a custom Powertools logger, use `createPowertoolsLogger` to include the + same fields. Install Powertools alongside the SDK and plugin: + + ```bash + npm install @aws-lambda-powertools/logger + ``` + + With the OpenTelemetry provider configured as shown above: + + ```typescript + --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts" + ``` + + The resulting JSON record includes these fields; trace and span IDs vary: + + ```json + { + "message": "Processing order", + "traceId": "1234567890abcdef1234567890abcdef", + "spanId": "1234567890abcdef", + "otelTraceSampled": true + } + ``` + + The adapter reads the current trace fields on each log call. In execution + view, handler code between operations uses `Workflow`; invocation view uses + `Invocation`. Step bodies use their attempt span. If you write your own + adapter, call `getLogEnrichment()` from its `DurableLoggingContext` on each + log call and include the returned fields in the output. See + [custom loggers](logging.md#custom-logger). === "Python" - `enrich_logger` installs a filter on root logger handlers. Handler code uses - `Invocation`, step bodies use their attempt, and child-context bodies use - their context span. Fields are optional: initialization logs and threads - without invocation context are left uncorrelated. + `enrich_logger` defaults to `True`. Its filter adds trace fields to log + records processed by the root logger's handlers. For standard-library text + logging, configure the formatter to print those fields: + + ```python + --8<-- "examples/python/sdk-reference/observability/opentelemetry/log-correlation.py" + ``` + + Configure the OpenTelemetry provider as shown above. The example keeps + existing handlers and sets their formatter before the first invocation. + Initialization logs and threads without invocation context have no trace + fields, so the formatter supplies `-` for missing values: + + ```text + Function initialized traceId=- spanId=- otelTraceSampled=- + Processing request traceId=1234567890abcdef1234567890abcdef spanId=1234567890abcdef otelTraceSampled=True + ``` + + Trace and span IDs vary. Handler code uses `Invocation`, step bodies use + their attempt, and child-context bodies use their context span. === "Java" diff --git a/examples/python/sdk-reference/observability/opentelemetry/log-correlation.py b/examples/python/sdk-reference/observability/opentelemetry/log-correlation.py new file mode 100644 index 00000000..8ecb6b9e --- /dev/null +++ b/examples/python/sdk-reference/observability/opentelemetry/log-correlation.py @@ -0,0 +1,23 @@ +import logging + +from aws_durable_execution_sdk_python import DurableContext, durable_execution +from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory + + +root_logger = logging.getLogger() +root_logger.setLevel(logging.INFO) +formatter = logging.Formatter( + "%(message)s traceId=%(traceId)s spanId=%(spanId)s " + "otelTraceSampled=%(otelTraceSampled)s", + defaults={"traceId": "-", "spanId": "-", "otelTraceSampled": "-"}, +) +for log_handler in root_logger.handlers: + log_handler.setFormatter(formatter) + +root_logger.info("Function initialized") + + +@durable_execution(plugins=[create_otel_plugin_factory()]) +def handler(event: dict, context: DurableContext) -> str: + context.logger.info("Processing request") + return "done" diff --git a/examples/typescript/sdk-reference/observability/custom-logger.ts b/examples/typescript/sdk-reference/observability/custom-logger.ts index e6d76ea8..4cb86180 100644 --- a/examples/typescript/sdk-reference/observability/custom-logger.ts +++ b/examples/typescript/sdk-reference/observability/custom-logger.ts @@ -1,9 +1,13 @@ -import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; +import { + createPowertoolsLogger, + DurableContext, + withDurableExecution, +} from "@aws/durable-execution-sdk-js"; import { Logger } from "@aws-lambda-powertools/logger"; const powertoolsLogger = new Logger({ serviceName: "my-service" }); export const handler = withDurableExecution(async (event, context: DurableContext) => { - context.configureLogger({ customLogger: powertoolsLogger }); + context.configureLogger({ customLogger: createPowertoolsLogger(powertoolsLogger) }); context.logger.info("Using Powertools logger"); }); diff --git a/examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts b/examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts new file mode 100644 index 00000000..75fd0f0f --- /dev/null +++ b/examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts @@ -0,0 +1,18 @@ +import { Logger } from "@aws-lambda-powertools/logger"; +import { + createPowertoolsLogger, + DurableContext, + withDurableExecution, +} from "@aws/durable-execution-sdk-js"; +import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; + +const logger = createPowertoolsLogger(new Logger({ serviceName: "orders" })); + +export const handler = withDurableExecution( + async (_event: unknown, context: DurableContext) => { + context.configureLogger({ customLogger: logger }); + context.logger.info("Processing order"); + return "processed"; + }, + { plugins: [createOtelPluginFactory()] }, +); diff --git a/examples/typescript/sdk-reference/observability/powertools-logger.ts b/examples/typescript/sdk-reference/observability/powertools-logger.ts index 88a94ee2..3e248f53 100644 --- a/examples/typescript/sdk-reference/observability/powertools-logger.ts +++ b/examples/typescript/sdk-reference/observability/powertools-logger.ts @@ -1,9 +1,13 @@ -import { Logger } from '@aws-lambda-powertools/logger'; -import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; +import { Logger } from "@aws-lambda-powertools/logger"; +import { + createPowertoolsLogger, + DurableContext, + withDurableExecution, +} from "@aws/durable-execution-sdk-js"; -const powertoolsLogger = new Logger({ serviceName: 'my-service' }); +const powertoolsLogger = new Logger({ serviceName: "my-service" }); export const handler = withDurableExecution(async (event, context: DurableContext) => { - context.configureLogger({ customLogger: powertoolsLogger }); - context.logger.info('Running handler'); + context.configureLogger({ customLogger: createPowertoolsLogger(powertoolsLogger) }); + context.logger.info("Running handler"); }); From e10915266bcf62394dffe63bed0be64c5731293a Mon Sep 17 00:00:00 2001 From: Frank Chen Date: Tue, 10 Nov 2026 21:12:53 +0000 Subject: [PATCH 31/31] docs: add Java log correlation example Show the Logback dependency, resource configuration, and log output for initialization, handler code, and a step. Document the execution-view MDC behavior verified with SDK/plugin 3.1.0, including missing fields outside step scopes and placeholder values before an invocation. --- .../observability/opentelemetry.md | 50 ++++++++++++++++--- .../opentelemetry/log-correlation-logback.xml | 10 ++++ .../opentelemetry/log-correlation.java | 26 ++++++++++ 3 files changed, 80 insertions(+), 6 deletions(-) create mode 100644 examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml create mode 100644 examples/java/sdk-reference/observability/opentelemetry/log-correlation.java diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index bf3fcbf0..918ce545 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -738,13 +738,51 @@ flag. Application-created spans on that trace can become the active context too. === "Java" - `enableMdc` adds the fields to SLF4J MDC. Configure your logging framework - to include MDC fields in its output. In CloudWatch, use **Group by nodes** - to inspect the hierarchy; the ungrouped X-Ray Segments Timeline cannot nest - OTLP spans under the Lambda platform segment. + `enableMdc` defaults to `true`. The plugin puts the trace fields in SLF4J's + mapped diagnostic context (MDC). Your logging layout must include those + fields to print them. - If spans are missing when using a Java agent, check - `OTEL_JAVAAGENT_EXTENSIONS`. + For text logging with Logback, add its SLF4J provider to your `pom.xml`: + + ```xml + + ch.qos.logback + logback-classic + 1.6.6 + runtime + + ``` + + Save this configuration as `src/main/resources/logback.xml`. If you already + have a Logback configuration, add the three `%X` fields to its existing + encoder pattern: + + ```xml title="src/main/resources/logback.xml" + --8<-- "examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml" + ``` + + With the OpenTelemetry provider configured as shown above, log through + the durable context: + + ```java + --8<-- "examples/java/sdk-reference/observability/opentelemetry/log-correlation.java" + ``` + + The `%X` expressions read the current MDC fields. Their defaults print `-` + when a field is absent. With the default execution view in SDK/plugin + 3.1.0, the plugin sets only `traceId` between operations. The step log + includes all three fields and identifies its attempt span: + + ```text + Function initialized traceId=- spanId=- otelTraceSampled=- + Processing request traceId=1234567890abcdef1234567890abcdef spanId=- otelTraceSampled=- + Processing step traceId=1234567890abcdef1234567890abcdef spanId=1234567890abcdef otelTraceSampled=true + ``` + + Trace and span IDs vary. In CloudWatch, use **Group by nodes** to inspect + the hierarchy; the ungrouped X-Ray Segments Timeline cannot nest OTLP spans + under the Lambda platform segment. If spans are missing when using a Java + agent, check `OTEL_JAVAAGENT_EXTENSIONS`. Test with an execution that waits and resumes. Confirm that its spans share a trace ID, invocations appear as they finish, completed steps do not run again, diff --git a/examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml b/examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml new file mode 100644 index 00000000..165e6265 --- /dev/null +++ b/examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml @@ -0,0 +1,10 @@ + + + + %msg traceId=%X{traceId:--} spanId=%X{spanId:--} otelTraceSampled=%X{otelTraceSampled:--}%n + + + + + + diff --git a/examples/java/sdk-reference/observability/opentelemetry/log-correlation.java b/examples/java/sdk-reference/observability/opentelemetry/log-correlation.java new file mode 100644 index 00000000..f2a8cab7 --- /dev/null +++ b/examples/java/sdk-reference/observability/opentelemetry/log-correlation.java @@ -0,0 +1,26 @@ +import java.util.Map; +import org.slf4j.LoggerFactory; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.DurableContext; +import software.amazon.lambda.durable.DurableHandler; +import software.amazon.lambda.durable.otel.OtelPlugin; + +public class LogCorrelationExample extends DurableHandler, String> { + static { + LoggerFactory.getLogger(LogCorrelationExample.class).info("Function initialized"); + } + + @Override + protected DurableConfig createConfiguration() { + return DurableConfig.builder().withPlugins(OtelPlugin.factory()).build(); + } + + @Override + public String handleRequest(Map event, DurableContext context) { + context.getLogger().info("Processing request"); + return context.step("process", String.class, stepContext -> { + stepContext.getLogger().info("Processing step"); + return "done"; + }); + } +}