From 8927f7a4219e5b1a84003b0c3a69442ae5c8d6ed Mon Sep 17 00:00:00 2001
From: Frank Chen <65260095+zhongkechen@users.noreply.github.com>
Date: Wed, 28 Oct 2026 14:51:07 -0700
Subject: [PATCH 01/31] docs: simplify and update OpenTelemetry guide
---
.../observability/opentelemetry.md | 1028 +++++------------
.../opentelemetry/execution-plugin.java | 4 +-
.../opentelemetry/invocation-plugin.java | 4 +-
.../opentelemetry/execution-plugin.py | 4 +-
.../opentelemetry/invocation-plugin.py | 4 +-
.../opentelemetry/execution-plugin.ts | 4 +-
.../opentelemetry/invocation-plugin.ts | 4 +-
7 files changed, 319 insertions(+), 733 deletions(-)
diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md
index 21998b5f..5e6841fd 100644
--- a/docs/sdk-reference/observability/opentelemetry.md
+++ b/docs/sdk-reference/observability/opentelemetry.md
@@ -1,101 +1,54 @@
# OpenTelemetry
-The OpenTelemetry plugin instruments a durable execution and emits distributed
-traces to any OpenTelemetry backend, such as Amazon CloudWatch. It builds on the
-[plugin interface](plugins.md): you register it in your handler configuration,
-and it opens spans at the workflow, invocation, operation, and attempt
-boundaries.
-
-A durable execution can run across many Lambda invocations. The durable backend
-propagates a stable server span through `_X_AMZN_TRACE_ID`. When the header
-contains valid `Root` and `Parent` fields, every plugin span for one execution
-uses that canonical trace ID. The execution-scoped `Workflow` span is a direct
-child of the remote backend span and keeps a deterministic span ID across
-reinvocations.
-
-Each Lambda invocation also produces an `Invocation` span. It uses the active
-ambient Lambda span as its parent only when that span belongs to the canonical
-trace. Otherwise, it is a direct child of the remote backend span. This places
-the workflow, invocations, operations, and attempts in one trace.
-
-The two plugins provide different operation views within that trace:
-
-- **Workflow view**: `ExecutionOtelPlugin` parents operations to `Workflow`.
- Each operation or attempt span links to the current `Invocation` span when
- that span is created.
-- **Invocation view**: `InvocationOtelPlugin` parents each operation segment to
- the current `Invocation`. Operations and attempts link to `Workflow`.
- Continuation and replay segments also link to the initial logical operation
- span.
-
-Links provide correlation and do not replace parent-child relationships. The
-deterministic ID overrides are scoped to spans created by the plugin, so
-unrelated OpenTelemetry instrumentation continues to use the provider's normal
-ID generator.
-
-Provider and export-pipeline configuration are external to the plugins. By
-default, each plugin uses the global provider. For an application-owned
-pipeline, supply the provider, provider factory, or provider builder and
-configure its processors, exporter, sampling, resources, propagators, and
-library instrumentation.
-
-Some names on this page keep the X-Ray label, such as the X-Ray trace header and
-the `awsxray` collector exporter, but the resulting traces are available in
-CloudWatch.
+The OpenTelemetry plugins trace durable operations across Lambda invocations.
+Choose a workflow or invocation view, register its factory, and configure an
+OpenTelemetry provider through a telemetry layer or application code. The
+provider owns exporters, resources, library instrumentation, and sampling policy.
!!! note "Experimental feature"
- The OpenTelemetry plugin is experimental and may change in a future
- release. It is not recommended for production workloads yet. Feedback is
- welcome through our
- [GitHub Discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206).
-
-!!! note "C# support"
-
- The OpenTelemetry plugin for C# is a work in progress. Until it ships, this
- page covers TypeScript, Python, and Java.
-
-!!! note "Go support"
-
- The OpenTelemetry plugin for Go is a work in progress. Until it ships, this
- page covers TypeScript, Python, and Java.
+ The plugins are experimental and are not yet recommended for production workloads.
+ They are available for TypeScript,
+ Python, and Java; Go and C# do not yet provide this integration. Feedback is welcome
+ in the [GitHub Discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206).
## Install the plugin
+These examples use the current factory API: core SDK 3.x with OTel plugin 2.x
+for TypeScript and Python, and SDK/plugin 3.x for Java. Upgrade the core and
+plugin together, including copies supplied by Lambda layers. Older plugin
+instances cannot be registered with the 3.x factory API.
+
=== "TypeScript"
+ Requires Node.js 22 or later:
+
```bash
- npm install @aws/durable-execution-sdk-js-otel \
- @opentelemetry/api \
- @opentelemetry/core \
- @opentelemetry/sdk-trace-node
+ npm install @aws/durable-execution-sdk-js@^3 \
+ @aws/durable-execution-sdk-js-otel@^2 \
+ @opentelemetry/api @opentelemetry/core @opentelemetry/sdk-trace-node
```
- The TypeScript plugin requires Node.js 22 or later. The OpenTelemetry
- packages are peer dependencies used by the configured provider or telemetry
- layer.
-
=== "Python"
- When a Lambda telemetry layer supplies the OpenTelemetry libraries:
+ When a telemetry layer supplies the OpenTelemetry libraries:
```bash
- pip install aws-durable-execution-sdk-python-otel
+ pip install "aws-durable-execution-sdk-python-otel>=2,<3"
```
- The base package intentionally does not install OpenTelemetry libraries,
- which prevents function dependencies from shadowing the version-aligned
- libraries in the layer.
-
- When the application configures its own provider, install the `standalone`
- extra:
+ The package installs a compatible core SDK but leaves OpenTelemetry libraries
+ to the layer. When the application supplies its own provider, install the
+ `standalone` extra instead:
```bash
- pip install "aws-durable-execution-sdk-python-otel[standalone]"
+ pip install "aws-durable-execution-sdk-python-otel[standalone]>=2,<3"
```
=== "Java"
+ Use the same 3.x version as the core SDK:
+
```xml
software.amazon.lambda.durable
@@ -104,78 +57,31 @@ CloudWatch.
```
- Add an exporter dependency when the application supplies its own tracer
- provider. The ADOT Java agent supplies the export pipeline for the no-arg
- plugin constructors.
+## Choose and register a view
-## The two plugins
+Register one OTel view factory. The SDK creates a separate plugin instance for
+each invocation, while the provider can be shared across invocations.
-Both plugins emit the same span types in one canonical trace. They differ in the
-parent of operation spans, the links on those spans, and when the spans become
-visible. Register exactly one of them.
+| Aspect | Execution view | Invocation view |
+| ------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------- |
+| Top-level operation parent | `Workflow` | Current `Invocation` |
+| Operation and attempt links | Current `Invocation` | `Workflow` |
+| Operation spanning a suspension | One logical span, exported on completion | Separate segments across invocations; later segments link to the initial operation span |
+| Best suited to | Inspecting the workflow hierarchy | Inspecting progress during long executions |
-### ExecutionOtelPlugin
+Nested operations remain under their child-context operation span; attempts
+remain under their operation. Both views export `Invocation` spans as invocations
+end and `Workflow` only when the SDK observes execution success or failure.
+Workflow spans can cover long durations, so backend ingestion limits still apply
+to both views. Export and flush are best effort; a timeout, stop, or terminated
+runtime can leave spans missing.
-`ExecutionOtelPlugin` provides a workflow-centered view. Operation spans are
-children of the `Workflow` span and link to the current `Invocation` span when
-they are created. The `Workflow` span is exported only when the execution
-reaches a terminal status. An operation that suspends is also left unexported
-until it completes.
+### Invocation view
-Choose this plugin for shorter executions or when the durable workflow hierarchy
-is more important than seeing each invocation as it completes. Long-open
-workflow spans can exceed the ingestion limits of some observability platforms.
-
-=== "TypeScript"
-
- ```typescript
- --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts"
- ```
-
-=== "Python"
-
- ```python
- --8<-- "examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py"
- ```
-
-=== "Java"
-
- ```java
- --8<-- "examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java"
- ```
-
-For an execution that suspends at a wait, the trace resembles:
-
-```text
-Remote backend server span
-|-- Workflow (exported on terminal status)
-| |-- Operation: fetch-data (STEP) -> link to Invocation #1
-| | `-- Attempt: fetch-data attempt 1 -> link to Invocation #1
-| |-- Operation: cooldown (WAIT) -> link to Invocation #2
-| `-- Operation: process (STEP) -> link to Invocation #2
-| `-- Attempt: process attempt 1 -> link to Invocation #2
-|-- Ambient Lambda span #1
-| `-- Invocation #1
-`-- Ambient Lambda span #2
- `-- Invocation #2
-```
-
-Each operation or attempt span links only to the current invocation when that
-span is created. If an operation starts in one invocation and completes in
-another, its exported operation span links to the invocation that completes it;
-the plugin does not retain the earlier invocation span ID. These links do not
-make the invocation spans children of `Workflow`.
-
-### InvocationOtelPlugin
-
-`InvocationOtelPlugin` provides an invocation-centered view. Each `Invocation`
-span parents the operation and attempt spans emitted during that Lambda
-invocation. Those operation and attempt spans link to the deterministic
-`Workflow` span for execution-scoped correlation.
-
-The plugin exports an invocation and its child spans when that invocation ends.
-Choose it for long-running executions or when you want traces to appear as the
-execution progresses.
+Use this view to see each invocation's work as the execution progresses.
+Completed work uses checkpointed results on replay. A resumed operation or
+replayed context can produce another span segment without rerunning a completed
+step's body.
=== "TypeScript"
@@ -195,452 +101,142 @@ execution progresses.
--8<-- "examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java"
```
-For an execution that suspends at a wait, the trace resembles:
-
-```text
-Remote backend server span
-|-- Workflow
-|-- Ambient Lambda span #1
-| `-- Invocation #1
-| |-- Operation: fetch-data (STEP) -> link to Workflow
-| | `-- Attempt: fetch-data attempt 1
-| | `-> link to Workflow
-| `-- Operation: cooldown (WAIT) -> link to Workflow
-`-- Ambient Lambda span #2
- `-- Invocation #2
- |-- Operation: cooldown (WAIT) -> links to initial cooldown
- | span and Workflow
- `-- Operation: process (STEP) -> link to Workflow
- `-- Attempt: process attempt 1 -> link to Workflow
-```
+### Execution view
-An operation that crosses an invocation boundary can produce more than one
-span. A continuation or replay segment uses a new span ID and links to the
-initial logical operation span, which uses the deterministic operation span ID.
-It also retains the `Workflow` link. CloudWatch does not yet visualize span
-links.
-
-### Choosing a plugin
-
-For both plugins, your observability platform's quotas and limits apply.
-
-| Consideration | ExecutionOtelPlugin | InvocationOtelPlugin |
-| ---------------------- | --------------------------------------------------------- | ----------------------------------------------------- |
-| Operation parent | `Workflow` | Current `Invocation` |
-| Correlation link | Operation or attempt to current `Invocation` | Operation or attempt to `Workflow` |
-| Continuation link | Reconstructed span links only to current `Invocation` | Later segment links to initial logical operation span |
-| Workflow export | On terminal execution status | On terminal execution status |
-| Operation visibility | When the operation completes | At each invocation boundary |
-| In-progress visibility | Limited until operations and the execution complete | Each invocation appears as it completes |
-| Better for | Short executions and a workflow-centered hierarchy | Long executions and an invocation-centered hierarchy |
-| Platform compatibility | Long-open spans can exceed platform ingestion time limits | Spans stay within the Lambda invocation time limit |
-
-## Trace parent and sampling
-
-The plugins resolve the execution parent and the sampling decision
-independently. A missing or unusable `Sampled` field does not replace a valid
-remote parent with a synthetic root.
-
-| `_X_AMZN_TRACE_ID` state | Canonical trace ID | Common execution ancestor | Sampling |
-| --------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------- | ----------------------------------------------------------------------- |
-| Valid `Root`, `Parent`, `Sampled=1` | Reuse `Root` | Remote `Parent` | Preserve sampled |
-| Valid `Root`, `Parent`, `Sampled=0` | Reuse `Root` | Remote `Parent` | Preserve not sampled |
-| Valid `Root`, `Parent`, no valid `Sampled` | Reuse `Root` | Remote `Parent` | Leave the sampled trace flag unset; configured sampler behavior applies |
-| Valid `Root`, missing or invalid `Parent`, `Sampled=1` or `Sampled=0` | Reuse `Root` | Synthetic execution root | Preserve the explicit decision |
-| Valid `Root`, missing or invalid `Parent`, no valid `Sampled` | Reuse `Root` | Synthetic execution root | Configured root sampler decides |
-| Missing or invalid `Root` | Derive from the execution ARN and stable execution start time | Synthetic execution root | Configured root sampler decides |
-
-Only `Sampled=0` and `Sampled=1` carry authoritative upstream decisions. An
-absent or unusable value does not mean that the upstream system explicitly
-chose not to sample. Its OpenTelemetry span context still represents the value
-with an unset sampled bit.
-
-A `ParentBased` sampler treats a remote parent with an unset sampled bit as not
-sampled. A directly configured non-parent-based `TraceIdRatioBased` sampler can
-decide from the canonical trace ID instead. The reused or derived trace ID
-remains stable across reinvocations, so trace-ID-ratio decisions also remain
-stable. Other sampler decisions can change between reinvocations unless the
-application persists them.
-
-When `Root` and `Parent` are valid, the normal hierarchy is:
-
-```text
-Remote backend server span (`Root` / `Parent`)
-|-- Workflow
-|-- Ambient Lambda span #1
-| `-- Invocation #1
-|-- Ambient Lambda span #2
-| `-- Invocation #2
-`-- Invocation #N (when no valid same-trace ambient span exists)
-```
-
-`Workflow` inherits the canonical trace ID and keeps a deterministic,
-execution-scoped span ID. An ambient span can parent `Invocation` only when it
-uses the same trace ID. A valid remote parent remains the real ancestor when
-`Sampled` is missing. The plugins do not create a synthetic-root link to it.
-
-When the plugins cannot construct a valid remote parent, they use:
-
-```text
-Synthetic execution root
-|-- Workflow
-|-- Invocation #1
-|-- Invocation #2
-`-- Invocation #N
-```
-
-The synthetic execution root has a deterministic span ID that remains stable
-across reinvocations and differs from Workflow and operation span IDs. A valid
-`Root` still supplies the canonical trace ID. If `Root` is also invalid, the
-plugins derive the trace ID from the execution ARN and stable execution start
-time. Fallback mode ignores unrelated ambient context. TypeScript, Python, and
-Java use this same topology with global and application-owned providers. The
-selected sampling decision applies consistently to every descendant.
-
-## Span structure and attributes
-
-Both plugins emit four span types. An invocation span covers one Lambda
-invocation. Operation spans cover durable operations such as steps, waits, child
-contexts, callbacks, and chained invokes. Attempt spans cover user function
-attempts, so retries appear as sibling spans.
-
-| Span | Attributes |
-| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| Workflow | `durable.execution.arn`, `durable.execution.status` |
-| Invocation | `durable.execution.arn`, `durable.invocation.status`, `durable.invocation.first` |
-| Operation | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status` |
-| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.attempt.number`, `durable.attempt.outcome` |
-
-`durable.operation.type` includes `STEP`, `WAIT`, `CONTEXT`, `CHAINED_INVOKE`,
-and `CALLBACK`. A `CONTEXT` operation gets an operation span but not a separate
-attempt span.
-
-## Deploy with the ADOT layer
-
-The AWS Distro for OpenTelemetry (ADOT) Lambda layer supplies a global provider,
-auto-instrumentation, and a collector extension. Add the language-specific
-layer, set `AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument`, enable active tracing
-so the runtime supplies `_X_AMZN_TRACE_ID`, and grant the function role the
-`AWSXRayDaemonWriteAccess` managed policy. Auto-instrumentation can then create
-an ambient Lambda span in the same trace as the durable backend parent.
-
-The no-arg plugin constructors use the global provider. You do not need a
-provider option.
+Use this view to keep the durable operation hierarchy beneath `Workflow`.
+An operation that suspends is exported when it completes, using its durable
+start and end times. Its span links to the invocation that exports the completion;
+it does not accumulate links to every earlier invocation.
=== "TypeScript"
- Find the current ADOT JavaScript layer ARN for your region and architecture
- in the
- [ADOT Lambda layer documentation](https://aws-otel.github.io/docs/getting-started/lambda#adot-lambda-layer-arns).
-
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: nodejs24.x
- Handler: index.handler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
-
```typescript
- new ExecutionOtelPlugin();
- // or
- new InvocationOtelPlugin();
+ --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts"
```
=== "Python"
- Find the current ADOT Python layer ARN for your region and architecture in
- the
- [ADOT Lambda layer documentation](https://aws-otel.github.io/docs/getting-started/lambda#adot-lambda-layer-arns).
-
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: python3.14
- Handler: index.handler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
-
```python
- ExecutionOtelPlugin()
- # or
- InvocationOtelPlugin()
+ --8<-- "examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py"
```
=== "Java"
- The Java agent must also load the plugin JAR as an agent extension so its
- auto-configuration SPI can scope deterministic ID generation to durable
- spans. Package the JAR in a layer under `java/lib`, then point
- `OTEL_JAVAAGENT_EXTENSIONS` to the deployed path.
-
- Find the current ADOT Java layer ARN in the
- [ADOT Java instrumentation releases](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest).
-
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: java25
- Handler: com.example.ExampleHandler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- -
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
- OTEL_JAVAAGENT_EXTENSIONS: /opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
-
```java
- new ExecutionOtelPlugin();
- // or
- new InvocationOtelPlugin();
+ --8<-- "examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java"
```
- To load the plugin from the layer without registering it in handler code,
- also set `DURABLE_EXECUTION_PLUGINS` to `otel-execution` or
- `otel-invocation`.
+## Configure a telemetry layer
-When the plugins use the global provider, they resolve it at invocation start.
-If a usable SDK provider or the integration required for scoped deterministic
-IDs is unavailable, all three SDKs log a warning and disable telemetry for that
-invocation. They retry provider resolution on the next invocation instead of
-emitting spans without deterministic durable IDs.
+A language auto-instrumentation layer registers the global provider used by the
+default factories. Select a layer compatible with your runtime, architecture,
+and region:
-## Deploy with the community auto-instrumentation layer
+| Layer | `AWS_LAMBDA_EXEC_WRAPPER` | Setup |
+| -------------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------- |
+| AWS Distro for OpenTelemetry (ADOT) | `/opt/otel-instrument` | [ADOT Lambda layers](https://aws-otel.github.io/docs/getting-started/lambda) |
+| OpenTelemetry community language layer | `/opt/otel-handler` | [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) |
-The OpenTelemetry community language layer initializes auto-instrumentation,
-registers a global OpenTelemetry provider, and exports spans using OTLP. Set
-`AWS_LAMBDA_EXEC_WRAPPER` to `/opt/otel-handler` and configure the OTLP endpoint
-for your observability backend. Enable active tracing so the plugin can reuse
-the durable backend parent and the layer can create a same-trace ambient Lambda
-span.
+Enable active tracing to receive the durable backend's X-Ray trace context.
+For ADOT, add these properties to your existing SAM function and grant its role
+`AWSXRayDaemonWriteAccess` to export traces to CloudWatch through X-Ray:
-The no-arg plugin constructors use the global provider. You do not need to
-configure or pass a provider to the plugin.
+```yaml
+Layers:
+ -
+Environment:
+ Variables:
+ AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
+Tracing: Active
+```
+
+For a community language layer exporting directly to an OTLP backend, use:
-The examples below use OTLP HTTP and disable metrics and log export. If the
-backend requires authentication, also configure `OTEL_EXPORTER_OTLP_HEADERS`.
+```yaml
+Layers:
+ -
+Environment:
+ Variables:
+ AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
+ OTEL_TRACES_EXPORTER: otlp
+ OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
+ OTEL_EXPORTER_OTLP_ENDPOINT: https://
+ OTEL_METRICS_EXPORTER: none
+ OTEL_LOGS_EXPORTER: none
+Tracing: Active
+```
-Follow the
-[OpenTelemetry Lambda auto-instrumentation documentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/)
-to obtain the language layer.
+Configure `OTEL_EXPORTER_OTLP_HEADERS` if your backend requires authentication.
+A collector extension is optional for local processing or additional exporters;
+follow the [collector setup guide](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector)
+for its layer, receiver, and `OPENTELEMETRY_COLLECTOR_CONFIG_URI` configuration.
+A collector alone does not initialize a language provider. For that setup, leave
+`AWS_LAMBDA_EXEC_WRAPPER` unset and configure the provider in application code as
+shown below. For collector export to CloudWatch, enable Transaction Search in
+your account.
=== "TypeScript"
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: nodejs24.x
- Handler: index.handler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
- OTEL_TRACES_EXPORTER: otlp
- OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
- OTEL_EXPORTER_OTLP_ENDPOINT: https://
- OTEL_METRICS_EXPORTER: none
- OTEL_LOGS_EXPORTER: none
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
-
- ```typescript
- new ExecutionOtelPlugin();
- // or
- new InvocationOtelPlugin();
- ```
+ The default factory resolves the global provider at invocation start.
=== "Python"
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: python3.14
- Handler: index.handler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
- OTEL_TRACES_EXPORTER: otlp
- OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
- OTEL_EXPORTER_OTLP_ENDPOINT: https://
- OTEL_METRICS_EXPORTER: none
- OTEL_LOGS_EXPORTER: none
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
-
- ```python
- ExecutionOtelPlugin()
- # or
- InvocationOtelPlugin()
- ```
+ Use the base plugin package with a telemetry layer so function dependencies
+ do not shadow the layer's OpenTelemetry libraries.
=== "Java"
- Use the community Java agent layer for auto-instrumentation. The agent must
- also load the plugin JAR as an agent extension so its auto-configuration SPI
- can scope deterministic ID generation to durable spans. Package the JAR in
- a layer under `java/lib`, then point `OTEL_JAVAAGENT_EXTENSIONS` to the
- deployed path.
-
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: java25
- Handler: com.example.ExampleHandler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- -
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
- OTEL_TRACES_EXPORTER: otlp
- OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
- OTEL_EXPORTER_OTLP_ENDPOINT: https://
- OTEL_METRICS_EXPORTER: none
- OTEL_LOGS_EXPORTER: none
- OTEL_JAVAAGENT_EXTENSIONS: /opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
+ The Java agent must also load the plugin JAR as an extension to install
+ durable ID generation and sampling. Package the JAR in a layer under
+ `java/lib`, add that layer to the function, and set:
- ```java
- new ExecutionOtelPlugin();
- // or
- new InvocationOtelPlugin();
+ ```text
+ OTEL_JAVAAGENT_EXTENSIONS=/opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar
```
- To load the plugin from the layer without registering it in handler code,
- also set `DURABLE_EXECUTION_PLUGINS` to `otel-execution` or
- `otel-invocation`.
-
-### Add a collector extension (optional)
-
-Add the community collector extension layer when you need local processing,
-routing, or an exporter that is not available in the language layer. Keep
-`AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler` so the language layer continues to
-initialize the provider and the ambient Lambda span.
+ Obtain the ADOT Java layer from the
+ [Java instrumentation releases](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest).
-Append the collector layer, point the OTLP exporter at its local receiver, and
-set the collector configuration path:
+If the global SDK provider or required durable ID integration is unavailable,
+the plugin logs a warning and disables its telemetry for that invocation.
+The next invocation retries initialization.
-```yaml
-Layers:
- -
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
-Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
- OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost:4318
- OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml
-```
+### Load the factory without handler registration
-For example, include the following `collector.yaml` in the function bundle to
-export traces to X-Ray:
+`DURABLE_EXECUTION_PLUGINS` selects a default-configured factory from an installed
+package or layer. It still requires a global provider. Choose one view through
+either code or environment configuration.
-```yaml
-receivers:
- otlp:
- protocols:
- http:
- endpoint: "localhost:4318"
-exporters:
- awsxray:
- region: "${AWS_REGION}"
-service:
- pipelines:
- traces:
- receivers: [otlp]
- exporters: [awsxray]
-```
+| Language | Invocation view | Execution view |
+| ---------- | ---------------------------------------------------- | --------------------------------------------------- |
+| TypeScript | `@aws/durable-execution-sdk-js-otel/otel-invocation` | `@aws/durable-execution-sdk-js-otel/otel-execution` |
+| Python | `otel-invocation` | `otel-execution` |
+| Java | `otel-invocation` | `otel-execution` |
-Change the collector exporter to route spans to another platform. See the
-[collector extension documentation](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector)
-for available components and layer deployment instructions.
+## Configure an application-owned provider
-### Collector-only setup (advanced)
-
-Use the collector extension without a language auto-instrumentation layer when
-you need to configure the OpenTelemetry provider in application code. Add only
-the collector layer, leave `AWS_LAMBDA_EXEC_WRAPPER` unset, keep active tracing
-enabled, and pass the application-owned provider to the plugin. Without an
-ambient Lambda span, `Invocation` uses the remote backend span directly.
-
-```yaml
-Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
-Environment:
- Variables:
- OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml
-Tracing: Active
-```
+Configure the exporter and processors once, then register the resulting factory
+as in the handler examples above. The plugin flushes at invocation end but does
+not take over provider shutdown. These examples send OTLP HTTP to a collector
+listening on loopback port 4318.
=== "TypeScript"
- `tracerProviderFactory` receives a function that creates the plugin's scoped
- deterministic ID generator. Use it when constructing the provider. The
- application owns provider registration and shutdown.
-
- ```bash
- npm install @opentelemetry/exporter-trace-otlp-http
- ```
+ Install `@opentelemetry/exporter-trace-otlp-http`. The provider factory receives
+ the durable ID-generator factory; pass its result to the provider constructor.
```typescript
- import { ExecutionOtelPlugin } from "@aws/durable-execution-sdk-js-otel";
+ import { createInvocationOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
- import {
- BatchSpanProcessor,
- NodeTracerProvider,
- } from "@opentelemetry/sdk-trace-node";
+ import { BatchSpanProcessor, NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
- const plugin = new ExecutionOtelPlugin({
+ const pluginFactory = createInvocationOtelPluginFactory({
tracerProviderFactory: (createIdGenerator) => {
const provider = new NodeTracerProvider({
idGenerator: createIdGenerator(),
- spanProcessors: [
- new BatchSpanProcessor(
- new OTLPTraceExporter({
- url: "http://localhost:4318/v1/traces",
- }),
- ),
- ],
+ spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter({
+ url: "http://localhost:4318/v1/traces",
+ }))],
});
provider.register();
return provider;
@@ -648,270 +244,260 @@ Tracing: Active
});
```
- Substitute `InvocationOtelPlugin` to use the invocation-centered view. To
- preserve a custom application ID generator for unrelated spans, pass it to
- `createIdGenerator(applicationIdGenerator)`.
+ Use `createExecutionOtelPluginFactory` for the execution view.
=== "Python"
- Install the standalone dependencies and an OTLP HTTP exporter:
-
- ```bash
- pip install "aws-durable-execution-sdk-python-otel[standalone]" \
- opentelemetry-exporter-otlp-proto-http
- ```
-
- Pass the application-owned provider through the shared configuration
- object:
+ Install the `standalone` extra and `opentelemetry-exporter-otlp-proto-http`.
```python
+ from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
-
from aws_durable_execution_sdk_python_otel import (
- ExecutionOtelPlugin,
+ InvocationOtelPluginFactory,
OtelPluginConfig,
)
provider = TracerProvider()
- provider.add_span_processor(
- BatchSpanProcessor(
- OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
- )
- )
-
- plugin = ExecutionOtelPlugin(
+ provider.add_span_processor(BatchSpanProcessor(
+ OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
+ ))
+ trace.set_tracer_provider(provider)
+ plugin_factory = InvocationOtelPluginFactory(
OtelPluginConfig(tracer_provider=provider)
)
```
- Substitute `InvocationOtelPlugin` to use the invocation-centered view.
+ Use `ExecutionOtelPluginFactory` for the execution view.
=== "Java"
- Add the OpenTelemetry OTLP exporter dependency, then pass a provider builder
- to either plugin. The plugin wraps the builder's ID generator and builds the
- application-owned provider.
-
- ```xml
-
- io.opentelemetry
- opentelemetry-exporter-otlp
- ${opentelemetry.version}
-
- ```
+ Add `io.opentelemetry:opentelemetry-exporter-otlp` to your dependencies. The
+ factory wraps the provider builder's ID generator and sampler, then builds
+ the provider once.
```java
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
- import software.amazon.lambda.durable.otel.ExecutionOtelPlugin;
+ import software.amazon.lambda.durable.otel.InvocationOtelPlugin;
var exporter = OtlpHttpSpanExporter.builder()
.setEndpoint("http://localhost:4318/v1/traces")
.build();
-
- var providerBuilder = SdkTracerProvider.builder()
- .addSpanProcessor(BatchSpanProcessor.builder(exporter).build());
-
- var plugin = new ExecutionOtelPlugin(providerBuilder);
+ var pluginFactory = InvocationOtelPlugin.factory(
+ SdkTracerProvider.builder()
+ .addSpanProcessor(BatchSpanProcessor.builder(exporter).build()));
```
- Substitute `InvocationOtelPlugin` to use the invocation-centered view.
+ Use `ExecutionOtelPlugin.factory` for the execution view.
-## Configuration
+### Plugin options
=== "TypeScript"
- Both plugins accept an `OtelPluginConfig` object:
+ Pass `OtelPluginConfig` to either `createInvocationOtelPluginFactory(config)`
+ or `createExecutionOtelPluginFactory(config)`.
- - **tracerProviderFactory** Creates an application-owned provider. Omit it
- to use the global provider.
- - **contextExtractor** Extracts upstream trace context. Defaults to
- `xRayContextExtractor`. Use `w3cClientContextExtractor` for W3C
- `traceparent` data in Lambda client context.
- - **instrumentationName** Instrumentation scope name. Defaults to
- `aws-durable-execution-sdk-js`.
- - **workflowSpanName** Name of the execution-scoped `Workflow` span.
- Defaults to `Workflow`.
- - **enrichLogger** Adds `traceId`, `spanId`, and `otelTraceSampled` to each
- durable log record. Defaults to `true`.
+ | Option | Default | Purpose |
+ | ----------------------- | ------------------------------ | ----------------------------------------------------------------- |
+ | `tracerProviderFactory` | Global provider | Build an application-owned provider |
+ | `contextExtractor` | `xRayContextExtractor` | Extract this execution's upstream context |
+ | `instrumentationName` | `aws-durable-execution-sdk-js` | Instrumentation scope |
+ | `workflowSpanName` | `Workflow` | Workflow span name |
+ | `enrichLogger` | `true` | Add trace fields to durable logs |
+ | `flushTimeoutMillis` | `1000` | Total time budget for awaiting provider flushes at invocation end |
- The application or telemetry layer owns exporters, propagators, resources,
- sampling, and HTTP or AWS SDK instrumentation.
+ `w3cClientContextExtractor` reads `traceparent` from Lambda client context.
+ The default X-Ray extractor reads invocation-local `xRayTraceId` when
+ present, otherwise `_X_AMZN_TRACE_ID`.
+ A flush timeout stops waiting; it does not cancel exports or synchronous
+ provider work.
=== "Python"
- Both plugins accept the same `OtelPluginConfig` dataclass:
+ Pass `OtelPluginConfig` to `InvocationOtelPluginFactory(config)` or
+ `ExecutionOtelPluginFactory(config)`.
- - **tracer_provider** Application-owned provider. Omit it to use the global
- provider.
- - **context_extractor** Defaults to `xray_context_extractor`. Use
- `w3c_client_context_extractor` for W3C `traceparent` data in Lambda
- client context.
- - **instrument_name** Instrumentation scope name. Defaults to
- `aws-durable-execution-sdk-python`.
- - **workflow_span_name** Name of the execution-scoped `Workflow` span.
- Defaults to `Workflow`.
- - **enrich_logger** Installs a root-logger filter that stamps trace context
- onto log records. Defaults to `True`.
+ | Option | Default | Purpose |
+ | -------------------- | ---------------------------------- | ------------------------------------------------ |
+ | `tracer_provider` | Global provider | Use an application-owned SDK provider |
+ | `context_extractor` | `xray_context_extractor` | Extract upstream context from `_X_AMZN_TRACE_ID` |
+ | `instrument_name` | `aws-durable-execution-sdk-python` | Instrumentation scope |
+ | `workflow_span_name` | `Workflow` | Workflow span name |
+ | `enrich_logger` | `True` | Add trace fields through root logger handlers |
- ```python
- plugin = InvocationOtelPlugin(
- OtelPluginConfig(
- instrument_name="my-service",
- enrich_logger=False,
- )
- )
- ```
+ `w3c_client_context_extractor` is a placeholder that returns `None`.
+ Use the default X-Ray extractor or implement a custom extractor.
=== "Java"
- Each plugin supports the following constructor forms:
+ Both plugins expose `factory()`, `factory(config)`, `factory(providerBuilder)`,
+ and `factory(providerBuilder, config)`. Build `OtelPluginConfig` with:
- ```java
- new InvocationOtelPlugin();
- new InvocationOtelPlugin(otelPluginConfig);
- new InvocationOtelPlugin(tracerProviderBuilder);
- new InvocationOtelPlugin(tracerProviderBuilder, otelPluginConfig);
- ```
+ | Builder method | Default | Purpose |
+ | -------------------------- | -------------------------------- | ------------------------------ |
+ | `contextExtractor(...)` | `new XRayContextExtractor()` | Extract upstream trace context |
+ | `instrumentationName(...)` | `aws-durable-execution-sdk-java` | Instrumentation scope |
+ | `workflowSpanName(...)` | `Workflow` | Workflow span name |
+ | `enableMdc(...)` | `true` | Add trace fields to SLF4J MDC |
- `ExecutionOtelPlugin` provides the same forms. The no-arg and config-only
- constructors use the global provider. The builder constructors create an
- application-owned provider.
+A custom extractor must return the durable execution's own context consistently
+across resumes. An unrelated or per-invocation trace ID splits the execution trace.
- Build `OtelPluginConfig` with:
+## Trace identity and sampling
- - **contextExtractor(...)** Defaults to `new XRayContextExtractor()`.
- - **enableMdc(...)** Adds `traceId`, `spanId`, and `otelTraceSampled` to the
- SLF4J MDC. Defaults to `true`.
- - **workflowSpanName(...)** Defaults to `Workflow`.
- - **instrumentationName(...)** Defaults to
- `aws-durable-execution-sdk-java`.
+Both views keep one canonical trace ID per execution. A valid propagated X-Ray
+`Root` supplies that ID, and a valid `Parent` becomes the remote execution
+ancestor. Missing `Sampled` does not invalidate a complete remote parent.
+`Workflow` has a stable execution-scoped span ID and parents to that ancestor.
+`Invocation` uses an active ambient span, such as the Lambda span, only when it
+belongs to the same trace; otherwise it parents to the execution ancestor.
- ```java
- var config = OtelPluginConfig.builder()
- .instrumentationName("my-service")
- .enableMdc(false)
- .build();
+```text
+Execution ancestor
+|-- Workflow
+|-- Ambient Lambda span #1
+| `-- Invocation #1
+`-- Ambient Lambda span #2
+ `-- Invocation #2
+```
- var plugin = new InvocationOtelPlugin(config);
- ```
+When no complete remote parent is available, the SDK uses a synthetic
+`DurableExecutionRoot`. It retains a valid propagated trace ID, or derives one
+from the execution ARN and stable execution start time. With a supported provider
+and that start time, sampled invocations export a zero-duration root with stable
+identity to preserve ancestry before the execution completes. Resumes can export
+another copy. `Workflow` carries duration and outcome; the fallback root carries
+neither. Remote backend spans are owned by the backend and are never exported
+by the plugin.
-Configure sampling through the provider. With ADOT, use the standard
-`OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` environment variables.
+Configure sampling through the provider, or through `OTEL_TRACES_SAMPLER` and
+`OTEL_TRACES_SAMPLER_ARG` when using auto-configuration. Keep the plugin's sampler
+integration installed so durable spans share the resolved decision while other
+instrumentation retains its own policy.
-## Load plugins dynamically
+=== "TypeScript"
-The SDK can discover a plugin from a Lambda layer or installed package without
-handler registration code. Dynamic loading uses default configuration and
-therefore requires a global OpenTelemetry provider.
+ An explicit `Sampled=1` or `Sampled=0` on a valid extracted trace takes
+ precedence. Otherwise, the configured sampler evaluates its root policy
+ using the canonical trace ID. Missing `Sampled` therefore does not make a
+ `ParentBased` sampler automatically drop the trace.
-=== "TypeScript"
+ A tracer that hides its sampler cannot enforce a shared durable sampling
+ decision and retains terminal-only fallback-root export.
- ```text
- DURABLE_EXECUTION_PLUGINS=@aws/durable-execution-sdk-js-otel/otel-invocation
- ```
+=== "Python"
+
+ Sampling follows the explicit backend decision, then the recording/sampled
+ state of a same-trace ambient span, then the configured sampler's root policy.
+ The plugin resolves the result once per invocation for its durable spans.
- Use `@aws/durable-execution-sdk-js-otel/otel-execution` for the
- workflow-centered plugin.
+=== "Java"
+
+ Sampling follows the explicit backend decision, then a same-trace ambient
+ span, then the configured sampler. Retain `DurableSampler`, installed by the
+ provider-builder and agent-extension paths, as the final sampler. Replacing
+ it with a plain sampler removes the shared-decision guarantee.
+
+Sampling decisions are not checkpointed across invocations. An explicit upstream
+decision or deterministic trace-ID-based policy keeps them consistent across
+resumes; a stateful custom sampler can choose differently on a later invocation.
+
+## Span attributes and failures
+
+| Span | Main attributes |
+| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Workflow | `durable.execution.arn`, `durable.execution.status` |
+| Invocation | `durable.execution.arn`, `durable.invocation.status`, `durable.invocation.first` |
+| Operation | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status` |
+| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.attempt.number`, `durable.attempt.outcome` |
+
+Operation types include `STEP`, `WAIT`, `CONTEXT`, `CHAINED_INVOKE`, and `CALLBACK`.
+A context has an operation span without a separate attempt span. An invocation
+that will retry reports `durable.invocation.status=RETRYING` with OTel `UNSET`.
+A caught operation failure can have an `ERROR` span while the enclosing invocation
+and workflow succeed. Inspect operation status as well as exception events;
+a backend can report failure without diagnostic details.
+
+=== "TypeScript"
+
+ `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without
+ diagnostics. The plugins export visible completion updates even when replay skips the
+ operation's body. They need the operation metadata to do so. When completed
+ child contexts would otherwise prune that metadata, configure
+ `pluginsConfig.childOperationsDepth` as described in
+ [operation detail](telemetry/exporter.md#operation-detail).
=== "Python"
- ```text
- DURABLE_EXECUTION_PLUGINS=otel-invocation
- ```
+ Two open fixes refine this behavior:
- Use `otel-execution` for the workflow-centered plugin.
+ - [PR #865](https://github.com/aws/aws-durable-execution-sdk-python/pull/865)
+ marks `FAILED`, `TIMED_OUT`, and `STOPPED` operations as `ERROR` even when
+ diagnostics are absent, and places terminal exception events at the span's
+ normalized end time. Without this fix, these outcomes can remain `UNSET`
+ when no error object is supplied.
+ - [PR #869](https://github.com/aws/aws-durable-execution-sdk-python/pull/869)
+ exports visible terminal updates from invocation input and checkpoint
+ notifications when cached-parent replay skips the completed child. It
+ deduplicates successful observations within an invocation, without promising
+ exactly-once delivery across retries or exporter failures.
-=== "Java"
+ These changes require builds containing the PRs until they are merged and released.
+ The plugins cannot reconstruct operation metadata that the service has pruned.
- ```text
- DURABLE_EXECUTION_PLUGINS=otel-invocation
- ```
+=== "Java"
- Use `otel-execution` for the workflow-centered plugin. Put the plugin JAR
- under `java/lib` in the layer so it is on the Lambda class path.
+ `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without
+ diagnostics. Missing details do not produce an exception event. Execution-view
+ operation exceptions use the known backend end timestamp when available;
+ invocation-view exceptions retain normal OpenTelemetry event timing.
-## Correlate logs with traces
+## Correlate logs and verify traces
-The plugin stamps active trace and span IDs onto log records. Logs from every
-reinvocation of one execution use the same canonical `traceId`. The `spanId`
-identifies the active invocation, operation, or attempt, and
-`otelTraceSampled` reflects the effective sampled flag. See
-[Logging](logging.md) for the SDK logger.
+Log correlation is enabled by default. `traceId` identifies the execution trace,
+`spanId` identifies the active span, and `otelTraceSampled` carries its sampled
+flag. Application-created spans on that trace can become the active context too.
=== "TypeScript"
- With `enrichLogger` enabled, the plugin adds `traceId`, `spanId`, and
- `otelTraceSampled` to each durable log record through the
- `enrichLogContext()` hook described in
- [Logging from a plugin](plugins.md#logging-from-a-plugin).
+ `enrichLogger` adds fields through the durable logger's `enrichLogContext`
+ hook. In execution view, handler code between operations uses `Workflow`;
+ invocation view uses `Invocation`. Step bodies use their attempt span.
+ Custom logger adapters must forward the enriched context.
=== "Python"
- With `enrich_logger=True`, the plugin installs a filter on root logger
- handlers. It adds `traceId`, `spanId`, and `otelTraceSampled` when a valid
- span is active. Treat these fields as optional in your formatter.
+ `enrich_logger` installs a filter on root logger handlers. Handler code uses
+ `Invocation`, step bodies use their attempt, and child-context bodies use
+ their context span. Fields are optional: initialization logs and threads
+ without invocation context are left uncorrelated.
=== "Java"
- With `enableMdc(true)`, the plugin puts `traceId`, `spanId`, and
- `otelTraceSampled` into the SLF4J MDC. Configure your logging framework to
- include MDC fields in its output.
-
-## Verify
-
-Invoke a durable function that includes a wait or resume so it runs across more
-than one Lambda invocation. In CloudWatch Traces, expect one canonical trace ID
-for every span carrying the execution ARN:
-
-- The durable backend server span is the common ancestor when CloudWatch
- returns it. `Workflow` is its direct child.
-- Each `Invocation` is a child of a same-trace ambient Lambda span, or a direct
- child of the backend server span when no valid ambient span exists.
-- `ExecutionOtelPlugin` operations are children of `Workflow` and link only to
- the current invocation when each span is created.
-- `InvocationOtelPlugin` operation segments are children of `Invocation` and
- link to `Workflow`. A resumed wait or replayed operation also links to its
- initial logical operation span.
-- Log records retain the canonical `traceId`. Their `spanId` values match the
- active invocation, operation, or attempt.
-
-Some backend queries omit the remote backend span. In that case, `Workflow` and
-directly parented `Invocation` spans still contain its non-null parent span ID.
-Do not interpret the unresolved parent as a parentless Workflow span.
-
-For Java, use the CloudWatch **Group by nodes** view to inspect the hierarchy.
-The ungrouped X-Ray Segments Timeline cannot attach OTLP-exported spans beneath
-the Lambda platform segment.
-
-When you use the community collector layer, enable CloudWatch Transaction
-Search in your account for traces to appear.
-
-If no plugin spans appear:
-
-- Confirm a provider with a span processor and exporter is registered.
-- Check for a warning that the global SDK provider or its deterministic ID
- integration was unavailable at invocation start.
-- With ADOT, confirm `AWS_LAMBDA_EXEC_WRAPPER` and active tracing are enabled.
-- With the community collector, confirm the layer and
- `OPENTELEMETRY_COLLECTOR_CONFIG_URI` are configured.
-- For Java ADOT, confirm the plugin JAR is listed in
- `OTEL_JAVAAGENT_EXTENSIONS`.
-- Check `_X_AMZN_TRACE_ID` and provider sampling configuration. An explicit
- `Sampled=0` suppresses export, and `ParentBased` treats a missing sampled bit
- as not sampled.
-
-If spans carrying one execution ARN use different trace IDs, confirm active
-tracing is enabled and that every invocation uses the same current plugin and
-context extractor. Missing or invalid remote parent data should still produce
-one canonical fallback trace, not one trace per invocation.
+ `enableMdc` adds the fields to SLF4J MDC. Configure your logging framework
+ to include MDC fields in its output. In CloudWatch, use **Group by nodes**
+ to inspect the hierarchy; the ungrouped X-Ray Segments Timeline cannot nest
+ OTLP spans under the Lambda platform segment.
+
+Test with an execution that waits and resumes. Confirm that its spans share a
+trace ID, invocations appear as they finish, completed steps do not run again,
+and logs identify the active span. `Workflow` appears after terminal completion.
+Span links correlate the two views but do not change parent-child relationships;
+CloudWatch does not currently visualize these links.
+
+If plugin spans are missing, check provider initialization warnings, exporter
+configuration, the layer's execution wrapper, and sampling. For the Java agent,
+also check `OTEL_JAVAAGENT_EXTENSIONS`. A missing `Workflow` can mean the execution
+has not finished or no terminal invocation exported it. A parent ID without a
+visible span can belong to the remote backend or an undelivered fallback root.
## See also
- [Plugins](plugins.md)
-- [Durable Execution Telemetry](telemetry/index.md)
- [Logging](logging.md)
+- [Durable Execution Telemetry](telemetry/index.md)
- [Steps](../operations/step.md)
diff --git a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java
index c1541cbf..162c1651 100644
--- a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java
+++ b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java
@@ -8,11 +8,11 @@
public class ExecutionOtelExample extends DurableHandler