diff --git a/docs/sdk-reference/languages/java/index.md b/docs/sdk-reference/languages/java/index.md
index b71f1b99..ce9a4762 100644
--- a/docs/sdk-reference/languages/java/index.md
+++ b/docs/sdk-reference/languages/java/index.md
@@ -217,9 +217,13 @@ Java can also load plugin providers from the application class path. Set
`DURABLE_EXECUTION_PLUGINS` to an ordered, comma-separated list of provider names:
```text
-DURABLE_EXECUTION_PLUGINS=otel-invocation,com.example.audit
+DURABLE_EXECUTION_PLUGINS=otel,com.example.audit
```
+The `otel` provider selects the default
+[OpenTelemetry execution view](../../observability/opentelemetry.md#execution-view)
+and requires SDK/plugin 3.1.0 or later.
+
The SDK discovers `DurableExecutionPluginProvider` implementations with `ServiceLoader`
and selects the providers named in the environment variable. Each provider implements
`createPlugin(InvocationInfo)` to return a fresh plugin instance. Selected dynamically
diff --git a/docs/sdk-reference/observability/logging.md b/docs/sdk-reference/observability/logging.md
index 893b6dba..383c9bc6 100644
--- a/docs/sdk-reference/observability/logging.md
+++ b/docs/sdk-reference/observability/logging.md
@@ -547,7 +547,11 @@ You can change where the SDK's logger sends its entries.
=== "TypeScript"
- Any object that implements `DurableLogger` works. Pass it to `configureLogger`.
+ Pass an object that implements `DurableLogger` to `configureLogger`. To
+ include execution metadata and plugin fields, the logger must also implement
+ `configureDurableLoggingContext`. Store the supplied context and read
+ `getDurableLogData()` and `getLogEnrichment?.()` on each log call.
+ `createPowertoolsLogger` handles this for a Powertools logger:
```typescript
--8<-- "examples/typescript/sdk-reference/observability/custom-logger.ts"
@@ -811,14 +815,18 @@ You can change where the SDK's logger sends its entries.
## Powertools for AWS Lambda
-[Powertools for AWS Lambda](https://docs.aws.amazon.com/powertools/) provides a
-structured logger that works as a drop-in replacement for the SDK's default logger.
+[Powertools for AWS Lambda](https://docs.aws.amazon.com/powertools/) provides structured
+logging that you can use with the SDK.
=== "TypeScript"
The
[Powertools for AWS Lambda (TypeScript)](https://docs.aws.amazon.com/powertools/typescript/latest/features/logger/)
- `Logger` satisfies the `DurableLogger` interface. Pass it via `configureLogger`.
+ `Logger` works with `createPowertoolsLogger`. Pass the wrapped logger via
+ `configureLogger` so each entry includes execution metadata and fields added
+ by plugins, such as OpenTelemetry trace IDs. See
+ [log correlation](opentelemetry.md#correlate-logs-and-verify-traces) for an example
+ with the OpenTelemetry plugin.
```typescript
--8<-- "examples/typescript/sdk-reference/observability/powertools-logger.ts"
diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md
index 603d6565..918ce545 100644
--- a/docs/sdk-reference/observability/opentelemetry.md
+++ b/docs/sdk-reference/observability/opentelemetry.md
@@ -1,135 +1,98 @@
# OpenTelemetry
-The OpenTelemetry plugin instruments a durable execution and emits distributed
-traces to any OpenTelemetry backend, such as Amazon CloudWatch. It builds on the
-[plugin interface](plugins.md): you register it in your handler configuration,
-and it opens spans at the workflow, invocation, operation, and attempt
-boundaries.
+The OpenTelemetry plugins trace durable operations across Lambda invocations.
+Create a factory for the execution view (the default) or select the invocation
+view explicitly. Configure an OpenTelemetry provider through a telemetry layer
+or application code. The
+provider owns exporters, resources, library instrumentation, and sampling policy.
-A durable execution can run across many Lambda invocations. The durable backend
-propagates a stable server span through `_X_AMZN_TRACE_ID`. When the header
-contains valid `Root` and `Parent` fields, every plugin span for one execution
-uses that canonical trace ID. The execution-scoped `Workflow` span is a direct
-child of the remote backend span and keeps a deterministic span ID across
-reinvocations.
-
-Each Lambda invocation also produces an `Invocation` span. It uses the active
-ambient Lambda span as its parent only when that span belongs to the canonical
-trace. Otherwise, it is a direct child of the remote backend span. This places
-the workflow, invocations, operations, and attempts in one trace.
-
-The two plugins provide different operation views within that trace:
-
-- **Workflow view**: `ExecutionOtelPlugin` parents operations to `Workflow`.
- Each operation or attempt span links to the current `Invocation` span when
- that span is created.
-- **Invocation view**: `InvocationOtelPlugin` parents each operation segment to
- the current `Invocation`. Operations and attempts link to `Workflow`.
- Continuation and replay segments also link to the initial logical operation
- span.
-
-Links provide correlation and do not replace parent-child relationships. The
-deterministic ID overrides are scoped to spans created by the plugin, so
-unrelated OpenTelemetry instrumentation continues to use the provider's normal
-ID generator.
-
-Provider and export-pipeline configuration are external to the plugins. By
-default, each plugin uses the global provider. For an application-owned
-pipeline, supply the provider, provider factory, or provider builder and
-configure its processors, exporter, sampling, resources, propagators, and
-library instrumentation.
-
-Some names on this page keep the X-Ray label, such as the X-Ray trace header and
-the `awsxray` collector exporter, but the resulting traces are available in
-CloudWatch.
-
-!!! note "Experimental feature"
-
- The OpenTelemetry plugin is experimental and may change in a future
- release. It is not recommended for production workloads yet. Feedback is
- welcome through our
- [GitHub Discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206).
-
-!!! note "C# support"
-
- The OpenTelemetry plugin for C# is a work in progress. Until it ships, this
- page covers TypeScript, Python, and Java.
-
-!!! note "Go support"
-
- The OpenTelemetry plugin for Go is a work in progress. Until it ships, this
- page covers TypeScript, Python, and Java.
-
-!!! note "Rust support"
-
- The OpenTelemetry plugin for Rust is a work in progress. Until it ships, this
- page covers TypeScript, Python, and Java.
+The plugins are available for TypeScript, Python, and Java. Go, Rust, and C# do
+not yet provide this integration.
## Install the plugin
+These examples use the unified factory APIs available in the versions below.
+Upgrade the core and plugin together, including copies supplied by Lambda layers.
+
=== "TypeScript"
+ Requires Node.js 22 or later and OTel plugin 2.1.0 or later.
+
```bash
- npm install @aws/durable-execution-sdk-js-otel \
- @opentelemetry/api \
- @opentelemetry/core \
- @opentelemetry/sdk-trace-node
+ npm install @aws/durable-execution-sdk-js@^3.0.0 \
+ @aws/durable-execution-sdk-js-otel@^2.1.0 \
+ @opentelemetry/api @opentelemetry/core @opentelemetry/sdk-trace-node
```
- The TypeScript plugin requires Node.js 22 or later. The OpenTelemetry
- packages are peer dependencies used by the configured provider or telemetry
- layer.
-
=== "Python"
- When a Lambda telemetry layer supplies the OpenTelemetry libraries:
+ Requires OTel plugin 2.1.0 or later.
+
+ When a telemetry layer supplies the OpenTelemetry libraries:
```bash
- pip install aws-durable-execution-sdk-python-otel
+ pip install "aws-durable-execution-sdk-python-otel>=2.1.0,<3"
```
- The base package intentionally does not install OpenTelemetry libraries,
- which prevents function dependencies from shadowing the version-aligned
- libraries in the layer.
-
- When the application configures its own provider, install the `standalone`
- extra:
+ The package installs a compatible core SDK but leaves OpenTelemetry libraries
+ to the layer. When the application supplies its own provider, install the
+ `standalone` extra instead:
```bash
- pip install "aws-durable-execution-sdk-python-otel[standalone]"
+ pip install "aws-durable-execution-sdk-python-otel[standalone]>=2.1.0,<3"
```
=== "Java"
+ Use matching core SDK and plugin versions, at least 3.1.0.
+
```xml
software.amazon.lambda.durableaws-durable-execution-sdk-java-plugin-otel
- ${durable.sdk.version}
+ 3.1.0
```
- Add an exporter dependency when the application supplies its own tracer
- provider. The ADOT Java agent supplies the export pipeline for the no-arg
- plugin constructors.
-
-## The two plugins
+## Choose and register a view
-Both plugins emit the same span types in one canonical trace. They differ in the
-parent of operation spans, the links on those spans, and when the spans become
-visible. Register exactly one of them.
+Register one factory using the unified entry point for your language. Omitting
+the mode selects the execution view. The SDK creates a separate plugin instance
+for each invocation, while the provider can be shared across invocations.
-### ExecutionOtelPlugin
-
-`ExecutionOtelPlugin` provides a workflow-centered view. Operation spans are
-children of the `Workflow` span and link to the current `Invocation` span when
-they are created. The `Workflow` span is exported only when the execution
-reaches a terminal status. An operation that suspends is also left unexported
-until it completes.
+### Choosing a plugin
-Choose this plugin for shorter executions or when the durable workflow hierarchy
-is more important than seeing each invocation as it completes. Long-open
-workflow spans can exceed the ingestion limits of some observability platforms.
+For both views, your observability platform's quotas and limits apply.
+
+| Consideration | Execution view | Invocation view |
+| -------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
+| Top-level operation parent | `Workflow` | Current `Invocation` |
+| Correlation link | Operation or attempt to current `Invocation` | Operation or attempt to `Workflow` |
+| Continuation link | Completed operation links only to the invocation that exports it | Later segment links to the initial logical operation span |
+| Operation visibility | When the operation completes | At each invocation boundary |
+| In-progress visibility | Limited until operations and the execution complete | Each invocation appears as it completes |
+| Better for | Shorter executions and a workflow-centered hierarchy | Long executions and progress through individual invocations |
+| Platform compatibility | Long-open spans can exceed ingestion time limits | Operation segments stay within an invocation; long Workflow spans still face ingestion limits |
+
+Nested operations remain under their child-context operation span; attempts
+remain under their operation. Both views export `Invocation` spans as invocations
+end and `Workflow` only when the SDK observes execution success or failure.
+Workflow spans can cover long durations, so backend ingestion limits still apply
+to both views. Export and flush are best effort; a timeout, stop, or terminated
+runtime can leave spans missing.
+
+The diagrams below show an execution with a valid remote backend parent and
+ambient Lambda spans in the same trace. See [trace identity and sampling](#trace-identity-and-sampling)
+for other parent contexts. The tree shows parent-child relationships; arrows
+show span links.
+
+### Execution view
+
+Top-level operation spans are children of `Workflow` and link to the current `Invocation`.
+The factory selects this view by default. An operation that suspends is exported
+when it completes, using its durable
+start and end times. Its span links to the invocation that exports the completion;
+it does not accumulate links to every earlier invocation.
=== "TypeScript"
@@ -149,15 +112,16 @@ workflow spans can exceed the ingestion limits of some observability platforms.
--8<-- "examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java"
```
-For an execution that suspends at a wait, the trace resembles:
+For an execution that suspends at `cooldown` and resumes in a second invocation,
+the trace resembles:
```text
Remote backend server span
|-- Workflow (exported on terminal status)
-| |-- Operation: fetch-data (STEP) -> link to Invocation #1
+| |-- Operation: fetch-data (STEP) -> link to Invocation #1
| | `-- Attempt: fetch-data attempt 1 -> link to Invocation #1
-| |-- Operation: cooldown (WAIT) -> link to Invocation #2
-| `-- Operation: process (STEP) -> link to Invocation #2
+| |-- Operation: cooldown (WAIT) -> link to Invocation #2
+| `-- Operation: process (STEP) -> link to Invocation #2
| `-- Attempt: process attempt 1 -> link to Invocation #2
|-- Ambient Lambda span #1
| `-- Invocation #1
@@ -165,22 +129,15 @@ Remote backend server span
`-- Invocation #2
```
-Each operation or attempt span links only to the current invocation when that
-span is created. If an operation starts in one invocation and completes in
-another, its exported operation span links to the invocation that completes it;
-the plugin does not retain the earlier invocation span ID. These links do not
-make the invocation spans children of `Workflow`.
-
-### InvocationOtelPlugin
+The links to `Invocation` do not make invocation spans children of `Workflow`.
-`InvocationOtelPlugin` provides an invocation-centered view. Each `Invocation`
-span parents the operation and attempt spans emitted during that Lambda
-invocation. Those operation and attempt spans link to the deterministic
-`Workflow` span for execution-scoped correlation.
+### Invocation view
-The plugin exports an invocation and its child spans when that invocation ends.
-Choose it for long-running executions or when you want traces to appear as the
-execution progresses.
+Top-level operation spans are children of the `Invocation` that emitted them and link to
+`Workflow`, so each invocation's spans export as it ends. Select invocation mode
+explicitly. Completed work uses checkpointed results on replay. A resumed operation or
+replayed context can produce another span segment without rerunning a completed
+step's body.
=== "TypeScript"
@@ -200,11 +157,11 @@ execution progresses.
--8<-- "examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java"
```
-For an execution that suspends at a wait, the trace resembles:
+The same execution, using invocation view, produces this hierarchy:
```text
Remote backend server span
-|-- Workflow
+|-- Workflow (exported on terminal status)
|-- Ambient Lambda span #1
| `-- Invocation #1
| |-- Operation: fetch-data (STEP) -> link to Workflow
@@ -220,432 +177,221 @@ Remote backend server span
```
An operation that crosses an invocation boundary can produce more than one
-span. A continuation or replay segment uses a new span ID and links to the
+span. A continuation segment uses a new span ID and links to the
initial logical operation span, which uses the deterministic operation span ID.
It also retains the `Workflow` link. CloudWatch does not yet visualize span
links.
-### Choosing a plugin
+## Configure a telemetry layer
-For both plugins, your observability platform's quotas and limits apply.
-
-| Consideration | ExecutionOtelPlugin | InvocationOtelPlugin |
-| ---------------------- | --------------------------------------------------------- | ----------------------------------------------------- |
-| Operation parent | `Workflow` | Current `Invocation` |
-| Correlation link | Operation or attempt to current `Invocation` | Operation or attempt to `Workflow` |
-| Continuation link | Reconstructed span links only to current `Invocation` | Later segment links to initial logical operation span |
-| Workflow export | On terminal execution status | On terminal execution status |
-| Operation visibility | When the operation completes | At each invocation boundary |
-| In-progress visibility | Limited until operations and the execution complete | Each invocation appears as it completes |
-| Better for | Short executions and a workflow-centered hierarchy | Long executions and an invocation-centered hierarchy |
-| Platform compatibility | Long-open spans can exceed platform ingestion time limits | Spans stay within the Lambda invocation time limit |
-
-## Trace parent and sampling
-
-The plugins resolve the execution parent and the sampling decision
-independently. A missing or unusable `Sampled` field does not replace a valid
-remote parent with a synthetic root.
-
-| `_X_AMZN_TRACE_ID` state | Canonical trace ID | Common execution ancestor | Sampling |
-| --------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------- | ----------------------------------------------------------------------- |
-| Valid `Root`, `Parent`, `Sampled=1` | Reuse `Root` | Remote `Parent` | Preserve sampled |
-| Valid `Root`, `Parent`, `Sampled=0` | Reuse `Root` | Remote `Parent` | Preserve not sampled |
-| Valid `Root`, `Parent`, no valid `Sampled` | Reuse `Root` | Remote `Parent` | Leave the sampled trace flag unset; configured sampler behavior applies |
-| Valid `Root`, missing or invalid `Parent`, `Sampled=1` or `Sampled=0` | Reuse `Root` | Synthetic execution root | Preserve the explicit decision |
-| Valid `Root`, missing or invalid `Parent`, no valid `Sampled` | Reuse `Root` | Synthetic execution root | Configured root sampler decides |
-| Missing or invalid `Root` | Derive from the execution ARN and stable execution start time | Synthetic execution root | Configured root sampler decides |
-
-Only `Sampled=0` and `Sampled=1` carry authoritative upstream decisions. An
-absent or unusable value does not mean that the upstream system explicitly
-chose not to sample. Its OpenTelemetry span context still represents the value
-with an unset sampled bit.
-
-A `ParentBased` sampler treats a remote parent with an unset sampled bit as not
-sampled. A directly configured non-parent-based `TraceIdRatioBased` sampler can
-decide from the canonical trace ID instead. The reused or derived trace ID
-remains stable across reinvocations, so trace-ID-ratio decisions also remain
-stable. Other sampler decisions can change between reinvocations unless the
-application persists them.
-
-When `Root` and `Parent` are valid, the normal hierarchy is:
+A language auto-instrumentation layer registers the global provider used by the
+default factories. Select a layer compatible with your runtime, architecture,
+and region:
-```text
-Remote backend server span (`Root` / `Parent`)
-|-- Workflow
-|-- Ambient Lambda span #1
-| `-- Invocation #1
-|-- Ambient Lambda span #2
-| `-- Invocation #2
-`-- Invocation #N (when no valid same-trace ambient span exists)
-```
+| Layer | `AWS_LAMBDA_EXEC_WRAPPER` | Setup |
+| -------------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------- |
+| AWS Distro for OpenTelemetry (ADOT) | `/opt/otel-instrument` | [ADOT Lambda layers](https://aws-otel.github.io/docs/getting-started/lambda) |
+| OpenTelemetry community language layer | `/opt/otel-handler` | [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) |
-`Workflow` inherits the canonical trace ID and keeps a deterministic,
-execution-scoped span ID. An ambient span can parent `Invocation` only when it
-uses the same trace ID. A valid remote parent remains the real ancestor when
-`Sampled` is missing. The plugins do not create a synthetic-root link to it.
+The following SAM snippets show the telemetry settings for an
+`AWS::Serverless::Function` resource. Merge the entries under `Properties` with
+your function's existing layers, environment variables, and policies.
+SAM applies `Policies` only when it creates
+the execution role. If you specify `Role`, attach the required permissions
+directly to that role.
-When the plugins cannot construct a valid remote parent, they use:
+### ADOT layer
-```text
-Synthetic execution root
-|-- Workflow
-|-- Invocation #1
-|-- Invocation #2
-`-- Invocation #N
+For ADOT with X-Ray export, enable `Active` tracing and grant
+`AWSXRayDaemonWriteAccess` alongside the durable execution permissions.
+
+```yaml
+MyFunction:
+ Type: AWS::Serverless::Function
+ Properties:
+ Layers:
+ -
+ Environment:
+ Variables:
+ AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
+ Tracing: Active
+ Policies:
+ - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
+ - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
```
-The synthetic execution root has a deterministic span ID that remains stable
-across reinvocations and differs from Workflow and operation span IDs. A valid
-`Root` still supplies the canonical trace ID. If `Root` is also invalid, the
-plugins derive the trace ID from the execution ARN and stable execution start
-time. Fallback mode ignores unrelated ambient context. TypeScript, Python, and
-Java use this same topology with global and application-owned providers. The
-selected sampling decision applies consistently to every descendant.
+### Community language layer
-## Span structure and attributes
+For a community language layer exporting directly to an OTLP backend, use
+[`PassThrough` tracing](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html#services-xray-default).
+Lambda passes the upstream X-Ray trace header to the
+function without emitting native Lambda X-Ray segments. The layer and durable
+plugin still provide application instrumentation.
-Both plugins emit four span types. An invocation span covers one Lambda
-invocation. Operation spans cover durable operations such as steps, waits, child
-contexts, callbacks, and chained invokes. Attempt spans cover user function
-attempts, so retries appear as sibling spans.
+Propagate the caller's trace context and sampling decision. Without an upstream
+header, Lambda generates a header with `Sampled=0`; the documented plugins honor
+that decision even with a local `always_on` sampler. See
+[supported sampling modes](#supported-sampling-modes).
-| Span | Attributes |
-| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| Workflow | `durable.execution.arn`, `durable.execution.status` |
-| Invocation | `durable.execution.arn`, `durable.invocation.status`, `durable.invocation.first` |
-| Operation | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status` |
-| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.attempt.number`, `durable.attempt.outcome` |
+```yaml
+MyFunction:
+ Type: AWS::Serverless::Function
+ Properties:
+ Layers:
+ -
+ Environment:
+ Variables:
+ AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
+ OTEL_TRACES_EXPORTER: otlp
+ OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
+ OTEL_EXPORTER_OTLP_ENDPOINT: https://
+ OTEL_METRICS_EXPORTER: none
+ OTEL_LOGS_EXPORTER: none
+ Tracing: PassThrough
+ Policies:
+ - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
+```
-`durable.operation.type` includes `STEP`, `WAIT`, `CONTEXT`, `CHAINED_INVOKE`,
-and `CALLBACK`. A `CONTEXT` operation gets an operation span but not a separate
-attempt span.
+**Exporting to CloudWatch:**
+[Enable Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html)
+in the destination account and Region, and confirm that **Ingest OpenTelemetry spans**
+shows **Enabled**. Otherwise, CloudWatch's OTLP endpoint rejects the spans. This setting
+is not required when exporting directly to another observability provider.
-## Deploy with the ADOT layer
+Configure `OTEL_EXPORTER_OTLP_HEADERS` if your backend requires authentication.
+Direct OTLP export to another observability provider does not require X-Ray
+write permissions. SAM still adds an X-Ray policy when it creates a role for
+`Tracing: PassThrough`.
-The AWS Distro for OpenTelemetry (ADOT) Lambda layer supplies a global provider,
-auto-instrumentation, and a collector extension. Add the language-specific
-layer, set `AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument`, enable active tracing
-so the runtime supplies `_X_AMZN_TRACE_ID`, and grant the function role the
-`AWSXRayDaemonWriteAccess` managed policy. Auto-instrumentation can then create
-an ambient Lambda span in the same trace as the durable backend parent.
+With `Active` tracing, a durable span can reference a native Lambda parent sent
+only to X-Ray. The plugin preserves that parent without re-exporting it, so a
+shared trace ID does not guarantee a complete parent tree in an OTLP backend.
+If you need native Lambda platform spans, use an integration that receives both
+OTLP spans and their X-Ray ancestors.
-The no-arg plugin constructors use the global provider. You do not need a
-provider option.
+`PassThrough` does not supply missing caller spans. For a complete upstream
+hierarchy, export the real caller span to the same backend and propagate its
+trace ID, span ID, and sampling decision through `X-Amzn-Trace-Id` when using the
+default plugin extractors. Preserve this propagation across downstream durable
+invokes.
-=== "TypeScript"
+You can optionally export telemetry through a
+[Collector extension](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector#configuration),
+which does not initialize the OpenTelemetry provider required by the plugin.
- Find the current ADOT JavaScript layer ARN for your region and architecture
- in the
- [ADOT Lambda layer documentation](https://aws-otel.github.io/docs/getting-started/lambda#adot-lambda-layer-arns).
-
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: nodejs24.x
- Handler: index.handler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
+### Provider initialization
- ```typescript
- new ExecutionOtelPlugin();
- // or
- new InvocationOtelPlugin();
- ```
+These requirements apply to both layer choices.
-=== "Python"
+=== "TypeScript"
- Find the current ADOT Python layer ARN for your region and architecture in
- the
- [ADOT Lambda layer documentation](https://aws-otel.github.io/docs/getting-started/lambda#adot-lambda-layer-arns).
-
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: python3.14
- Handler: index.handler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
+ The default factory resolves the global provider at invocation start.
- ```python
- ExecutionOtelPlugin()
- # or
- InvocationOtelPlugin()
- ```
+=== "Python"
+
+ Use the base plugin package with a telemetry layer so function dependencies
+ do not shadow the layer's OpenTelemetry libraries.
=== "Java"
- The Java agent must also load the plugin JAR as an agent extension so its
- auto-configuration SPI can scope deterministic ID generation to durable
- spans. Package the JAR in a layer under `java/lib`, then point
- `OTEL_JAVAAGENT_EXTENSIONS` to the deployed path.
-
- Find the current ADOT Java layer ARN in the
- [ADOT Java instrumentation releases](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest).
-
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: java25
- Handler: com.example.ExampleHandler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- -
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
- OTEL_JAVAAGENT_EXTENSIONS: /opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
+ The Java agent must also load the plugin JAR as an extension to install
+ durable ID generation and sampling. Package the JAR in a layer under
+ `java/lib`, add that layer to the function, and set:
- ```java
- new ExecutionOtelPlugin();
- // or
- new InvocationOtelPlugin();
+ ```text
+ OTEL_JAVAAGENT_EXTENSIONS=/opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar
```
- To load the plugin from the layer without registering it in handler code,
- also set `DURABLE_EXECUTION_PLUGINS` to `otel-execution` or
- `otel-invocation`.
+ Obtain the ADOT Java layer from the
+ [Java instrumentation releases](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest).
-When the plugins use the global provider, they resolve it at invocation start.
-If a usable SDK provider or the integration required for scoped deterministic
-IDs is unavailable, all three SDKs log a warning and disable telemetry for that
-invocation. They retry provider resolution on the next invocation instead of
-emitting spans without deterministic durable IDs.
+If the global SDK provider or required durable ID integration is unavailable,
+the plugin logs a warning and disables its telemetry for that invocation.
+The next invocation retries initialization.
-## Deploy with the community auto-instrumentation layer
+### Load the factory without handler registration
-The OpenTelemetry community language layer initializes auto-instrumentation,
-registers a global OpenTelemetry provider, and exports spans using OTLP. Set
-`AWS_LAMBDA_EXEC_WRAPPER` to `/opt/otel-handler` and configure the OTLP endpoint
-for your observability backend. Enable active tracing so the plugin can reuse
-the durable backend parent and the layer can create a same-trace ambient Lambda
-span.
+`DURABLE_EXECUTION_PLUGINS` selects a default-configured OTel factory from an installed
+package or layer. These factories use the global OpenTelemetry tracer provider
+initialized by your telemetry layer or application. Choose one view through either
+code or environment configuration.
-The no-arg plugin constructors use the global provider. You do not need to
-configure or pass a provider to the plugin.
+The value follows each SDK's discovery mechanism: an importable module specifier
+or a registered provider name.
-The examples below use OTLP HTTP and disable metrics and log export. If the
-backend requires authentication, also configure `OTEL_EXPORTER_OTLP_HEADERS`.
+| Language and lookup | Execution view (default) | Invocation view |
+| ------------------------------------ | ------------------------------------ | ---------------------------------------------------- |
+| TypeScript: Node.js module specifier | `@aws/durable-execution-sdk-js-otel` | `@aws/durable-execution-sdk-js-otel/otel-invocation` |
+| Python: package entry-point name | `otel` | `otel-invocation` |
+| Java: `ServiceLoader` provider name | `otel` | `otel-invocation` |
-Follow the
-[OpenTelemetry Lambda auto-instrumentation documentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/)
-to obtain the language layer.
+!!! warning "Configure exactly one OTel view"
-=== "TypeScript"
+ Registering two distinct OTel factories fails validation because the OTel
+ views share an exclusive group in each SDK. This applies to naming two views
+ in `DURABLE_EXECUTION_PLUGINS`, and to naming one there while also registering
+ a separate factory in handler code.
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: nodejs24.x
- Handler: index.handler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
- OTEL_TRACES_EXPORTER: otlp
- OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
- OTEL_EXPORTER_OTLP_ENDPOINT: https://
- OTEL_METRICS_EXPORTER: none
- OTEL_LOGS_EXPORTER: none
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
+=== "TypeScript"
- ```typescript
- new ExecutionOtelPlugin();
- // or
- new InvocationOtelPlugin();
- ```
+ `@aws/durable-execution-sdk-js-otel/otel-execution` selects the same execution
+ view as the bare `@aws/durable-execution-sdk-js-otel` package name. Configure
+ only one of these entry points in `DURABLE_EXECUTION_PLUGINS`.
=== "Python"
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: python3.14
- Handler: index.handler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
- OTEL_TRACES_EXPORTER: otlp
- OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
- OTEL_EXPORTER_OTLP_ENDPOINT: https://
- OTEL_METRICS_EXPORTER: none
- OTEL_LOGS_EXPORTER: none
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
-
- ```python
- ExecutionOtelPlugin()
- # or
- InvocationOtelPlugin()
- ```
+ `otel-execution` is an alias for the default `otel` execution view. Both names
+ resolve to the same factory object. If both appear in
+ `DURABLE_EXECUTION_PLUGINS`, the SDK loads one factory and logs a warning that
+ it skipped the duplicate. Configure one name.
=== "Java"
- Use the community Java agent layer for auto-instrumentation. The agent must
- also load the plugin JAR as an agent extension so its auto-configuration SPI
- can scope deterministic ID generation to durable spans. Package the JAR in
- a layer under `java/lib`, then point `OTEL_JAVAAGENT_EXTENSIONS` to the
- deployed path.
-
- ```yaml
- MyFunction:
- Type: AWS::Serverless::Function
- Properties:
- Runtime: java25
- Handler: com.example.ExampleHandler
- Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
- -
- Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
- OTEL_TRACES_EXPORTER: otlp
- OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
- OTEL_EXPORTER_OTLP_ENDPOINT: https://
- OTEL_METRICS_EXPORTER: none
- OTEL_LOGS_EXPORTER: none
- OTEL_JAVAAGENT_EXTENSIONS: /opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar
- Tracing: Active
- Policies:
- - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- ```
+ `otel-execution` also selects the execution view. Each provider name creates
+ a separate factory, so `DURABLE_EXECUTION_PLUGINS=otel-execution,otel` fails
+ at startup:
- ```java
- new ExecutionOtelPlugin();
- // or
- new InvocationOtelPlugin();
+ ```text
+ Conflicting plugin factories otel-execution (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin) in exclusive group 'durable-otel-view'. Configure only one plugin from this group.
```
- To load the plugin from the layer without registering it in handler code,
- also set `DURABLE_EXECUTION_PLUGINS` to `otel-execution` or
- `otel-invocation`.
+ Configure one provider name.
-### Add a collector extension (optional)
+## Configure an application-owned provider
-Add the community collector extension layer when you need local processing,
-routing, or an exporter that is not available in the language layer. Keep
-`AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler` so the language layer continues to
-initialize the provider and the ambient Lambda span.
+This setup is for advanced use cases that require direct control over the
+OpenTelemetry provider. You do not need it when using a
+[language auto-instrumentation layer](#configure-a-telemetry-layer): the layer
+initializes the global provider used by the plugin's default factories.
-Append the collector layer, point the OTLP exporter at its local receiver, and
-set the collector configuration path:
-
-```yaml
-Layers:
- -
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
-Environment:
- Variables:
- AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler
- OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost:4318
- OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml
-```
-
-For example, include the following `collector.yaml` in the function bundle to
-export traces to X-Ray:
-
-```yaml
-receivers:
- otlp:
- protocols:
- http:
- endpoint: "localhost:4318"
-exporters:
- awsxray:
- region: "${AWS_REGION}"
-service:
- pipelines:
- traces:
- receivers: [otlp]
- exporters: [awsxray]
-```
-
-Change the collector exporter to route spans to another platform. See the
-[collector extension documentation](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector)
-for available components and layer deployment instructions.
-
-### Collector-only setup (advanced)
-
-Use the collector extension without a language auto-instrumentation layer when
-you need to configure the OpenTelemetry provider in application code. Add only
-the collector layer, leave `AWS_LAMBDA_EXEC_WRAPPER` unset, keep active tracing
-enabled, and pass the application-owned provider to the plugin. Without an
-ambient Lambda span, `Invocation` uses the remote backend span directly.
-
-```yaml
-Layers:
- - !Sub arn:aws:lambda:${AWS::Region}::layer::
-Environment:
- Variables:
- OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml
-Tracing: Active
-```
+Configure the exporter and processors once, then register the resulting factory
+as in the handler examples above. The plugin flushes at invocation end but does
+not take over provider shutdown. These examples send OTLP HTTP to a collector
+listening on loopback port 4318.
=== "TypeScript"
- `tracerProviderFactory` receives a function that creates the plugin's scoped
- deterministic ID generator. Use it when constructing the provider. The
- application owns provider registration and shutdown.
+ Install `@opentelemetry/exporter-trace-otlp-http`:
```bash
npm install @opentelemetry/exporter-trace-otlp-http
```
+ The provider factory receives the durable ID-generator factory; pass its
+ result to the provider constructor.
+
```typescript
- import { ExecutionOtelPlugin } from "@aws/durable-execution-sdk-js-otel";
+ import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
- import {
- BatchSpanProcessor,
- NodeTracerProvider,
- } from "@opentelemetry/sdk-trace-node";
+ import { BatchSpanProcessor, NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
- const plugin = new ExecutionOtelPlugin({
+ const pluginFactory = createOtelPluginFactory({
tracerProviderFactory: (createIdGenerator) => {
const provider = new NodeTracerProvider({
idGenerator: createIdGenerator(),
- spanProcessors: [
- new BatchSpanProcessor(
- new OTLPTraceExporter({
- url: "http://localhost:4318/v1/traces",
- }),
- ),
- ],
+ spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter({
+ url: "http://localhost:4318/v1/traces",
+ }))],
});
provider.register();
return provider;
@@ -653,270 +399,406 @@ Tracing: Active
});
```
- Substitute `InvocationOtelPlugin` to use the invocation-centered view. To
- preserve a custom application ID generator for unrelated spans, pass it to
- `createIdGenerator(applicationIdGenerator)`.
+ Set `mode: "invocation"` in the same options object to select the invocation view.
=== "Python"
- Install the standalone dependencies and an OTLP HTTP exporter:
-
- ```bash
- pip install "aws-durable-execution-sdk-python-otel[standalone]" \
- opentelemetry-exporter-otlp-proto-http
- ```
-
- Pass the application-owned provider through the shared configuration
- object:
+ Install the `standalone` extra and `opentelemetry-exporter-otlp-proto-http`.
```python
+ from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
-
from aws_durable_execution_sdk_python_otel import (
- ExecutionOtelPlugin,
+ create_otel_plugin_factory,
OtelPluginConfig,
)
provider = TracerProvider()
- provider.add_span_processor(
- BatchSpanProcessor(
- OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
- )
+ provider.add_span_processor(BatchSpanProcessor(
+ OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
+ ))
+ trace.set_tracer_provider(provider)
+ plugin_factory = create_otel_plugin_factory(
+ OtelPluginConfig(tracer_provider=provider)
)
+ ```
- plugin = ExecutionOtelPlugin(
- OtelPluginConfig(tracer_provider=provider)
+ For invocation view, replace the factory call with:
+
+ ```python
+ plugin_factory = create_otel_plugin_factory(
+ OtelPluginConfig(tracer_provider=provider), mode="invocation"
)
```
- Substitute `InvocationOtelPlugin` to use the invocation-centered view.
+ `tracer_provider` is what the plugin uses, so `set_tracer_provider` is only
+ needed to point *other* instrumentation at the same provider. It takes effect
+ once per process and is ignored with a warning if a telemetry layer already
+ registered a global provider; omit it in that case.
=== "Java"
- Add the OpenTelemetry OTLP exporter dependency, then pass a provider builder
- to either plugin. The plugin wraps the builder's ID generator and builds the
- application-owned provider.
+ Add `io.opentelemetry:opentelemetry-exporter-otlp` to your dependencies,
+ using OpenTelemetry 1.67.0 to match plugin 3.1.0:
```xml
io.opentelemetryopentelemetry-exporter-otlp
- ${opentelemetry.version}
+ 1.67.0
```
+ The factory wraps the provider builder's ID generator and sampler, then
+ builds the provider once.
+
```java
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
- import software.amazon.lambda.durable.otel.ExecutionOtelPlugin;
+ import software.amazon.lambda.durable.otel.OtelPlugin;
var exporter = OtlpHttpSpanExporter.builder()
.setEndpoint("http://localhost:4318/v1/traces")
.build();
-
- var providerBuilder = SdkTracerProvider.builder()
- .addSpanProcessor(BatchSpanProcessor.builder(exporter).build());
-
- var plugin = new ExecutionOtelPlugin(providerBuilder);
+ var pluginFactory = OtelPlugin.factory(
+ OtelPlugin.Mode.EXECUTION,
+ SdkTracerProvider.builder()
+ .addSpanProcessor(BatchSpanProcessor.builder(exporter).build()));
```
- Substitute `InvocationOtelPlugin` to use the invocation-centered view.
+ Use `OtelPlugin.Mode.INVOCATION` as the first argument to select the invocation view.
-## Configuration
+### Plugin options
=== "TypeScript"
- Both plugins accept an `OtelPluginConfig` object:
+ Pass `OtelPluginFactoryOptions` to `createOtelPluginFactory(options)`. These
+ options extend `OtelPluginConfig` with the view selection.
- - **tracerProviderFactory** Creates an application-owned provider. Omit it
- to use the global provider.
- - **contextExtractor** Extracts upstream trace context. Defaults to
- `xRayContextExtractor`. Use `w3cClientContextExtractor` for W3C
- `traceparent` data in Lambda client context.
- - **instrumentationName** Instrumentation scope name. Defaults to
- `aws-durable-execution-sdk-js`.
- - **workflowSpanName** Name of the execution-scoped `Workflow` span.
- Defaults to `Workflow`.
- - **enrichLogger** Adds `traceId`, `spanId`, and `otelTraceSampled` to each
- durable log record. Defaults to `true`.
+ | Option | Default | Purpose |
+ | ----------------------- | ------------------------------ | ----------------------------------------------------------------- |
+ | `mode` | `"execution"` | Select `"execution"` or `"invocation"` |
+ | `tracerProviderFactory` | Global provider | Build an application-owned provider |
+ | `contextExtractor` | `xRayContextExtractor` | Extract this execution's upstream context |
+ | `instrumentationName` | `aws-durable-execution-sdk-js` | Instrumentation scope |
+ | `workflowSpanName` | `Workflow` | Workflow span name |
+ | `enrichLogger` | `true` | Add trace fields to durable logs |
+ | `flushTimeoutMillis` | `1000` | Total time budget for awaiting provider flushes at invocation end |
- The application or telemetry layer owns exporters, propagators, resources,
- sampling, and HTTP or AWS SDK instrumentation.
+ The default X-Ray extractor reads invocation-local `xRayTraceId` when
+ present, otherwise `_X_AMZN_TRACE_ID`.
+ A flush timeout stops waiting; it does not cancel exports or synchronous
+ provider work.
=== "Python"
- Both plugins accept the same `OtelPluginConfig` dataclass:
+ Pass `OtelPluginConfig` to `create_otel_plugin_factory(config)`. Select the
+ invocation view with `create_otel_plugin_factory(config, mode="invocation")`;
+ `mode` is a keyword argument, separate from `OtelPluginConfig`.
- - **tracer_provider** Application-owned provider. Omit it to use the global
- provider.
- - **context_extractor** Defaults to `xray_context_extractor`. Use
- `w3c_client_context_extractor` for W3C `traceparent` data in Lambda
- client context.
- - **instrument_name** Instrumentation scope name. Defaults to
- `aws-durable-execution-sdk-python`.
- - **workflow_span_name** Name of the execution-scoped `Workflow` span.
- Defaults to `Workflow`.
- - **enrich_logger** Installs a root-logger filter that stamps trace context
- onto log records. Defaults to `True`.
+ | Option | Default | Purpose |
+ | -------------------- | ---------------------------------- | ------------------------------------------------ |
+ | `tracer_provider` | Global provider | Use an application-owned SDK provider |
+ | `context_extractor` | `xray_context_extractor` | Extract upstream context from `_X_AMZN_TRACE_ID` |
+ | `instrument_name` | `aws-durable-execution-sdk-python` | Instrumentation scope |
+ | `workflow_span_name` | `Workflow` | Workflow span name |
+ | `enrich_logger` | `True` | Add trace fields through root logger handlers |
- ```python
- plugin = InvocationOtelPlugin(
- OtelPluginConfig(
- instrument_name="my-service",
- enrich_logger=False,
- )
- )
- ```
+ Use the default X-Ray extractor or implement a custom extractor.
=== "Java"
- Each plugin supports the following constructor forms:
+ `OtelPlugin` exposes `factory()`, `factory(mode)`, `factory(mode, config)`,
+ `factory(mode, providerBuilder)`, and `factory(mode, providerBuilder, config)`.
+ Use `OtelPlugin.Mode.EXECUTION` or `OtelPlugin.Mode.INVOCATION` for `mode`.
+ Only `factory()` omits the mode and defaults to execution. Build
+ `OtelPluginConfig` with:
- ```java
- new InvocationOtelPlugin();
- new InvocationOtelPlugin(otelPluginConfig);
- new InvocationOtelPlugin(tracerProviderBuilder);
- new InvocationOtelPlugin(tracerProviderBuilder, otelPluginConfig);
- ```
+ | Builder method | Default | Purpose |
+ | -------------------------- | -------------------------------- | ------------------------------ |
+ | `contextExtractor(...)` | `new XRayContextExtractor()` | Extract upstream trace context |
+ | `instrumentationName(...)` | `aws-durable-execution-sdk-java` | Instrumentation scope |
+ | `workflowSpanName(...)` | `Workflow` | Workflow span name |
+ | `enableMdc(...)` | `true` | Add trace fields to SLF4J MDC |
- `ExecutionOtelPlugin` provides the same forms. The no-arg and config-only
- constructors use the global provider. The builder constructors create an
- application-owned provider.
+A custom extractor must return the durable execution's own context consistently
+across resumes. An unrelated or per-invocation trace ID splits the execution trace.
- Build `OtelPluginConfig` with:
+## Trace identity and sampling
- - **contextExtractor(...)** Defaults to `new XRayContextExtractor()`.
- - **enableMdc(...)** Adds `traceId`, `spanId`, and `otelTraceSampled` to the
- SLF4J MDC. Defaults to `true`.
- - **workflowSpanName(...)** Defaults to `Workflow`.
- - **instrumentationName(...)** Defaults to
- `aws-durable-execution-sdk-java`.
+Both views keep one canonical trace ID per execution. A valid propagated X-Ray
+`Root` supplies that ID, and a valid `Parent` becomes the remote execution
+ancestor. Missing `Sampled` does not invalidate a complete remote parent.
+`Workflow` has a stable execution-scoped span ID and parents to that ancestor.
+`Invocation` uses an active ambient span, such as the Lambda span, only when it
+belongs to the same trace; otherwise it parents to the execution ancestor.
- ```java
- var config = OtelPluginConfig.builder()
- .instrumentationName("my-service")
- .enableMdc(false)
- .build();
+```text
+Execution ancestor
+|-- Workflow
+|-- Ambient Lambda span #1
+| `-- Invocation #1
+`-- Ambient Lambda span #2
+ `-- Invocation #2
+```
- var plugin = new InvocationOtelPlugin(config);
- ```
+When no complete remote parent is available, the SDK uses a synthetic
+`DurableExecutionRoot`. It retains a valid propagated trace ID, or derives one
+from the execution ARN and stable execution start time. With a supported provider
+and that start time, sampled invocations export a zero-duration root with stable
+identity to preserve ancestry before the execution completes. Resumes can export
+another copy. `Workflow` carries duration and outcome; the fallback root carries
+neither. Remote backend spans are owned by the backend and are never exported
+by the plugin.
+
+Configure a [supported sampling mode](#supported-sampling-modes) through the provider,
+or through `OTEL_TRACES_SAMPLER` and
+`OTEL_TRACES_SAMPLER_ARG` when using auto-configuration. Keep the plugin's sampler
+integration installed so durable spans share the resolved decision while other
+instrumentation retains its own policy.
-Configure sampling through the provider. With ADOT, use the standard
-`OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` environment variables.
+=== "TypeScript"
-## Load plugins dynamically
+ An explicit `Sampled=1` or `Sampled=0` on a valid extracted trace takes
+ precedence. Otherwise, the configured sampler evaluates its root policy
+ using the canonical trace ID. Missing `Sampled` therefore does not make a
+ `ParentBased` sampler automatically drop the trace.
-The SDK can discover a plugin from a Lambda layer or installed package without
-handler registration code. Dynamic loading uses default configuration and
-therefore requires a global OpenTelemetry provider.
+ A tracer that hides its sampler cannot enforce a shared durable sampling
+ decision and retains terminal-only fallback-root export.
+
+=== "Python"
+
+ Sampling follows the explicit backend decision, then the recording/sampled
+ state of a same-trace ambient span, then the configured sampler's root policy.
+ The plugin resolves the result once per invocation for its durable spans.
+
+=== "Java"
+
+ Sampling follows the explicit backend decision, then a same-trace ambient
+ span, then the configured sampler. Retain `DurableSampler`, installed by the
+ provider-builder and agent-extension paths, as the final sampler. Replacing
+ it with a plain sampler removes the shared-decision guarantee.
+
+### Supported sampling modes
+
+These rules apply to TypeScript core 3.0.0 / OTel 2.1.0, Python core 3.0.2 / OTel 2.1.0,
+and Java SDK/plugin 3.1.0, in both views. These versions include the unified
+factory APIs used by this guide.
+
+- **Explicit upstream decision:** a valid `Sampled=0` or `Sampled=1` overrides the
+ local sampler for durable spans. Thus `Sampled=1` still exports durable spans
+ with local `always_off`, and `Sampled=0` drops them with local `always_on`.
+ Active tracing alone does not imply `Sampled=1`.
+- **Otherwise:** use a deterministic policy based on the canonical trace ID, such
+ as `always_on`, `always_off`, or `traceidratio` with a fixed ratio. For
+ `traceidratio`, `OTEL_TRACES_SAMPLER_ARG=0.5` selects whole execution traces when
+ the local policy is consulted.
+
+Keep the policy, ratio, and any inherited upstream or ambient decision stable across
+retries, resumes, and processes. Random, time-, load-, or mutable-state-dependent
+policies are unsupported and can produce partial traces. The plugins do not
+checkpoint sampling decisions; caching or application persistence does not make
+another policy supported.
+
+The provider or auto-configuration layer resolves the environment settings, defaults,
+and invalid values. They do not override an explicitly configured application
+provider. Keep the plugin's sampler integration installed; other application and
+ADOT spans retain their own policy.
+
+Sampling behavior may change in future SDK/plugin releases.
+[Issue #295](https://github.com/aws/aws-durable-execution-docs/issues/295) proposes
+letting the configured sampler decide whether to inherit upstream sampling, using
+standard direct and `parentbased_*` semantics. That proposal is not implemented in
+the versions listed above. Check the release notes and this section when upgrading.
+
+Sampling does not guarantee export or a complete parent tree: a durable span may be
+sampled while its application parent is dropped, or its native X-Ray parent may be
+absent from OTLP. See the [community layer setup](#community-language-layer) for
+trace propagation and native X-Ray parents.
+
+## Span attributes and failures
+
+Optional attributes such as `durable.operation.name` and
+`durable.operation.subtype` are emitted only when supplied.
+
+| Span | Main attributes |
+| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Workflow | `durable.execution.arn`, `durable.execution.status` |
+| Invocation | `durable.execution.arn`, `durable.invocation.status`, `durable.invocation.first` |
+| Operation | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status` |
+| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.attempt.number`, `durable.attempt.outcome` |
+
+Operation types include `STEP`, `WAIT`, `CONTEXT`, `CHAINED_INVOKE`, and `CALLBACK`.
+A context has an operation span without a separate attempt span. An invocation
+that will retry reports `durable.invocation.status=RETRYING` with OTel `UNSET`.
+A caught operation failure can have an `ERROR` span while the enclosing invocation
+and workflow succeed. Inspect operation status as well as exception events;
+a backend can report failure without diagnostic details.
=== "TypeScript"
- ```text
- DURABLE_EXECUTION_PLUGINS=@aws/durable-execution-sdk-js-otel/otel-invocation
+ `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without
+ diagnostics. The plugins export newly observed external completion updates
+ for waits, callbacks, and invokes when their metadata is visible, even if replay
+ skips the operation's body. When completed child contexts would otherwise
+ prune that metadata, configure
+ `pluginsConfig.childOperationsDepth` as described in
+ [operation detail](telemetry/exporter.md#operation-detail).
+
+=== "Python"
+
+ `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without an
+ error object. Exception events preserve the checkpointed type, message, and
+ stack trace; missing fields are omitted. No exception event is emitted when
+ both type and message are absent. Terminal operation and attempt exception
+ events use the span end timestamp.
+
+ Completion spans can be missing when replay returns a cached parent result
+ without visiting its children, even if their operation metadata is still
+ visible. Pruned operation metadata cannot be reconstructed.
+
+=== "Java"
+
+ `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without
+ diagnostics. Missing details do not produce an exception event. Execution-view
+ operation exceptions use the known backend end timestamp when available;
+ invocation-view exceptions retain normal OpenTelemetry event timing.
+ Since SDK/plugin 3.0.1, operation exception events report the stored original
+ error type when available, including errors from another language, while
+ retaining the throwable's message and stack diagnostics.
+
+## Correlate logs and verify traces
+
+Log correlation is enabled by default. `traceId` identifies the execution trace,
+`spanId` identifies the active span, and `otelTraceSampled` carries its sampled
+flag. Application-created spans on that trace can become the active context too.
+
+=== "TypeScript"
+
+ `enrichLogger` defaults to `true`. The default logger includes trace fields
+ when you call `context.logger.info()` or another log method.
+
+ For a custom Powertools logger, use `createPowertoolsLogger` to include the
+ same fields. Install Powertools alongside the SDK and plugin:
+
+ ```bash
+ npm install @aws-lambda-powertools/logger
+ ```
+
+ With the OpenTelemetry provider configured as shown above:
+
+ ```typescript
+ --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts"
```
- Use `@aws/durable-execution-sdk-js-otel/otel-execution` for the
- workflow-centered plugin.
+ The resulting JSON record includes these fields; trace and span IDs vary:
+
+ ```json
+ {
+ "message": "Processing order",
+ "traceId": "1234567890abcdef1234567890abcdef",
+ "spanId": "1234567890abcdef",
+ "otelTraceSampled": true
+ }
+ ```
+
+ The adapter reads the current trace fields on each log call. In execution
+ view, handler code between operations uses `Workflow`; invocation view uses
+ `Invocation`. Step bodies use their attempt span. If you write your own
+ adapter, call `getLogEnrichment()` from its `DurableLoggingContext` on each
+ log call and include the returned fields in the output. See
+ [custom loggers](logging.md#custom-logger).
=== "Python"
+ `enrich_logger` defaults to `True`. Its filter adds trace fields to log
+ records processed by the root logger's handlers. For standard-library text
+ logging, configure the formatter to print those fields:
+
+ ```python
+ --8<-- "examples/python/sdk-reference/observability/opentelemetry/log-correlation.py"
+ ```
+
+ Configure the OpenTelemetry provider as shown above. The example keeps
+ existing handlers and sets their formatter before the first invocation.
+ Initialization logs and threads without invocation context have no trace
+ fields, so the formatter supplies `-` for missing values:
+
```text
- DURABLE_EXECUTION_PLUGINS=otel-invocation
+ Function initialized traceId=- spanId=- otelTraceSampled=-
+ Processing request traceId=1234567890abcdef1234567890abcdef spanId=1234567890abcdef otelTraceSampled=True
```
- Use `otel-execution` for the workflow-centered plugin.
+ Trace and span IDs vary. Handler code uses `Invocation`, step bodies use
+ their attempt, and child-context bodies use their context span.
=== "Java"
- ```text
- DURABLE_EXECUTION_PLUGINS=otel-invocation
+ `enableMdc` defaults to `true`. The plugin puts the trace fields in SLF4J's
+ mapped diagnostic context (MDC). Your logging layout must include those
+ fields to print them.
+
+ For text logging with Logback, add its SLF4J provider to your `pom.xml`:
+
+ ```xml
+
+ ch.qos.logback
+ logback-classic
+ 1.6.6
+ runtime
+
```
- Use `otel-execution` for the workflow-centered plugin. Put the plugin JAR
- under `java/lib` in the layer so it is on the Lambda class path.
+ Save this configuration as `src/main/resources/logback.xml`. If you already
+ have a Logback configuration, add the three `%X` fields to its existing
+ encoder pattern:
-## Correlate logs with traces
+ ```xml title="src/main/resources/logback.xml"
+ --8<-- "examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml"
+ ```
-The plugin stamps active trace and span IDs onto log records. Logs from every
-reinvocation of one execution use the same canonical `traceId`. The `spanId`
-identifies the active invocation, operation, or attempt, and
-`otelTraceSampled` reflects the effective sampled flag. See
-[Logging](logging.md) for the SDK logger.
+ With the OpenTelemetry provider configured as shown above, log through
+ the durable context:
-=== "TypeScript"
+ ```java
+ --8<-- "examples/java/sdk-reference/observability/opentelemetry/log-correlation.java"
+ ```
- With `enrichLogger` enabled, the plugin adds `traceId`, `spanId`, and
- `otelTraceSampled` to each durable log record through the
- `enrichLogContext()` hook described in
- [Logging from a plugin](plugins.md#logging-from-a-plugin).
+ The `%X` expressions read the current MDC fields. Their defaults print `-`
+ when a field is absent. With the default execution view in SDK/plugin
+ 3.1.0, the plugin sets only `traceId` between operations. The step log
+ includes all three fields and identifies its attempt span:
-=== "Python"
+ ```text
+ Function initialized traceId=- spanId=- otelTraceSampled=-
+ Processing request traceId=1234567890abcdef1234567890abcdef spanId=- otelTraceSampled=-
+ Processing step traceId=1234567890abcdef1234567890abcdef spanId=1234567890abcdef otelTraceSampled=true
+ ```
- With `enrich_logger=True`, the plugin installs a filter on root logger
- handlers. It adds `traceId`, `spanId`, and `otelTraceSampled` when a valid
- span is active. Treat these fields as optional in your formatter.
+ Trace and span IDs vary. In CloudWatch, use **Group by nodes** to inspect
+ the hierarchy; the ungrouped X-Ray Segments Timeline cannot nest OTLP spans
+ under the Lambda platform segment. If spans are missing when using a Java
+ agent, check `OTEL_JAVAAGENT_EXTENSIONS`.
-=== "Java"
+Test with an execution that waits and resumes. Confirm that its spans share a
+trace ID, invocations appear as they finish, completed steps do not run again,
+and logs identify the active span. `Workflow` appears after terminal completion.
+Span links correlate the two views but do not change parent-child relationships;
+CloudWatch does not currently visualize these links.
- With `enableMdc(true)`, the plugin puts `traceId`, `spanId`, and
- `otelTraceSampled` into the SLF4J MDC. Configure your logging framework to
- include MDC fields in its output.
-
-## Verify
-
-Invoke a durable function that includes a wait or resume so it runs across more
-than one Lambda invocation. In CloudWatch Traces, expect one canonical trace ID
-for every span carrying the execution ARN:
-
-- The durable backend server span is the common ancestor when CloudWatch
- returns it. `Workflow` is its direct child.
-- Each `Invocation` is a child of a same-trace ambient Lambda span, or a direct
- child of the backend server span when no valid ambient span exists.
-- `ExecutionOtelPlugin` operations are children of `Workflow` and link only to
- the current invocation when each span is created.
-- `InvocationOtelPlugin` operation segments are children of `Invocation` and
- link to `Workflow`. A resumed wait or replayed operation also links to its
- initial logical operation span.
-- Log records retain the canonical `traceId`. Their `spanId` values match the
- active invocation, operation, or attempt.
-
-Some backend queries omit the remote backend span. In that case, `Workflow` and
-directly parented `Invocation` spans still contain its non-null parent span ID.
-Do not interpret the unresolved parent as a parentless Workflow span.
-
-For Java, use the CloudWatch **Group by nodes** view to inspect the hierarchy.
-The ungrouped X-Ray Segments Timeline cannot attach OTLP-exported spans beneath
-the Lambda platform segment.
-
-When you use the community collector layer, enable CloudWatch Transaction
-Search in your account for traces to appear.
-
-If no plugin spans appear:
-
-- Confirm a provider with a span processor and exporter is registered.
-- Check for a warning that the global SDK provider or its deterministic ID
- integration was unavailable at invocation start.
-- With ADOT, confirm `AWS_LAMBDA_EXEC_WRAPPER` and active tracing are enabled.
-- With the community collector, confirm the layer and
- `OPENTELEMETRY_COLLECTOR_CONFIG_URI` are configured.
-- For Java ADOT, confirm the plugin JAR is listed in
- `OTEL_JAVAAGENT_EXTENSIONS`.
-- Check `_X_AMZN_TRACE_ID` and provider sampling configuration. An explicit
- `Sampled=0` suppresses export, and `ParentBased` treats a missing sampled bit
- as not sampled.
-
-If spans carrying one execution ARN use different trace IDs, confirm active
-tracing is enabled and that every invocation uses the same current plugin and
-context extractor. Missing or invalid remote parent data should still produce
-one canonical fallback trace, not one trace per invocation.
+If spans are missing, check provider warnings, exporter configuration, the layer's
+execution wrapper, and [sampling](#supported-sampling-modes).
+`Workflow` requires a terminal invocation to export it. An unresolved parent can
+belong to the remote backend or an undelivered fallback root.
## See also
- [Plugins](plugins.md)
-- [Durable Execution Telemetry](telemetry/index.md)
- [Logging](logging.md)
+- [Durable Execution Telemetry](telemetry/index.md)
- [Steps](../operations/step.md)
+- [Feedback and discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206)
diff --git a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java
index c1541cbf..6ee6922e 100644
--- a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java
+++ b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java
@@ -3,16 +3,16 @@
import software.amazon.lambda.durable.DurableConfig;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
-import software.amazon.lambda.durable.otel.ExecutionOtelPlugin;
+import software.amazon.lambda.durable.otel.OtelPlugin;
public class ExecutionOtelExample extends DurableHandler