diff --git a/docs/sdk-reference/languages/java/index.md b/docs/sdk-reference/languages/java/index.md index b71f1b99..ce9a4762 100644 --- a/docs/sdk-reference/languages/java/index.md +++ b/docs/sdk-reference/languages/java/index.md @@ -217,9 +217,13 @@ Java can also load plugin providers from the application class path. Set `DURABLE_EXECUTION_PLUGINS` to an ordered, comma-separated list of provider names: ```text -DURABLE_EXECUTION_PLUGINS=otel-invocation,com.example.audit +DURABLE_EXECUTION_PLUGINS=otel,com.example.audit ``` +The `otel` provider selects the default +[OpenTelemetry execution view](../../observability/opentelemetry.md#execution-view) +and requires SDK/plugin 3.1.0 or later. + The SDK discovers `DurableExecutionPluginProvider` implementations with `ServiceLoader` and selects the providers named in the environment variable. Each provider implements `createPlugin(InvocationInfo)` to return a fresh plugin instance. Selected dynamically diff --git a/docs/sdk-reference/observability/logging.md b/docs/sdk-reference/observability/logging.md index 893b6dba..383c9bc6 100644 --- a/docs/sdk-reference/observability/logging.md +++ b/docs/sdk-reference/observability/logging.md @@ -547,7 +547,11 @@ You can change where the SDK's logger sends its entries. === "TypeScript" - Any object that implements `DurableLogger` works. Pass it to `configureLogger`. + Pass an object that implements `DurableLogger` to `configureLogger`. To + include execution metadata and plugin fields, the logger must also implement + `configureDurableLoggingContext`. Store the supplied context and read + `getDurableLogData()` and `getLogEnrichment?.()` on each log call. + `createPowertoolsLogger` handles this for a Powertools logger: ```typescript --8<-- "examples/typescript/sdk-reference/observability/custom-logger.ts" @@ -811,14 +815,18 @@ You can change where the SDK's logger sends its entries. ## Powertools for AWS Lambda -[Powertools for AWS Lambda](https://docs.aws.amazon.com/powertools/) provides a -structured logger that works as a drop-in replacement for the SDK's default logger. +[Powertools for AWS Lambda](https://docs.aws.amazon.com/powertools/) provides structured +logging that you can use with the SDK. === "TypeScript" The [Powertools for AWS Lambda (TypeScript)](https://docs.aws.amazon.com/powertools/typescript/latest/features/logger/) - `Logger` satisfies the `DurableLogger` interface. Pass it via `configureLogger`. + `Logger` works with `createPowertoolsLogger`. Pass the wrapped logger via + `configureLogger` so each entry includes execution metadata and fields added + by plugins, such as OpenTelemetry trace IDs. See + [log correlation](opentelemetry.md#correlate-logs-and-verify-traces) for an example + with the OpenTelemetry plugin. ```typescript --8<-- "examples/typescript/sdk-reference/observability/powertools-logger.ts" diff --git a/docs/sdk-reference/observability/opentelemetry.md b/docs/sdk-reference/observability/opentelemetry.md index 603d6565..918ce545 100644 --- a/docs/sdk-reference/observability/opentelemetry.md +++ b/docs/sdk-reference/observability/opentelemetry.md @@ -1,135 +1,98 @@ # OpenTelemetry -The OpenTelemetry plugin instruments a durable execution and emits distributed -traces to any OpenTelemetry backend, such as Amazon CloudWatch. It builds on the -[plugin interface](plugins.md): you register it in your handler configuration, -and it opens spans at the workflow, invocation, operation, and attempt -boundaries. +The OpenTelemetry plugins trace durable operations across Lambda invocations. +Create a factory for the execution view (the default) or select the invocation +view explicitly. Configure an OpenTelemetry provider through a telemetry layer +or application code. The +provider owns exporters, resources, library instrumentation, and sampling policy. -A durable execution can run across many Lambda invocations. The durable backend -propagates a stable server span through `_X_AMZN_TRACE_ID`. When the header -contains valid `Root` and `Parent` fields, every plugin span for one execution -uses that canonical trace ID. The execution-scoped `Workflow` span is a direct -child of the remote backend span and keeps a deterministic span ID across -reinvocations. - -Each Lambda invocation also produces an `Invocation` span. It uses the active -ambient Lambda span as its parent only when that span belongs to the canonical -trace. Otherwise, it is a direct child of the remote backend span. This places -the workflow, invocations, operations, and attempts in one trace. - -The two plugins provide different operation views within that trace: - -- **Workflow view**: `ExecutionOtelPlugin` parents operations to `Workflow`. - Each operation or attempt span links to the current `Invocation` span when - that span is created. -- **Invocation view**: `InvocationOtelPlugin` parents each operation segment to - the current `Invocation`. Operations and attempts link to `Workflow`. - Continuation and replay segments also link to the initial logical operation - span. - -Links provide correlation and do not replace parent-child relationships. The -deterministic ID overrides are scoped to spans created by the plugin, so -unrelated OpenTelemetry instrumentation continues to use the provider's normal -ID generator. - -Provider and export-pipeline configuration are external to the plugins. By -default, each plugin uses the global provider. For an application-owned -pipeline, supply the provider, provider factory, or provider builder and -configure its processors, exporter, sampling, resources, propagators, and -library instrumentation. - -Some names on this page keep the X-Ray label, such as the X-Ray trace header and -the `awsxray` collector exporter, but the resulting traces are available in -CloudWatch. - -!!! note "Experimental feature" - - The OpenTelemetry plugin is experimental and may change in a future - release. It is not recommended for production workloads yet. Feedback is - welcome through our - [GitHub Discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206). - -!!! note "C# support" - - The OpenTelemetry plugin for C# is a work in progress. Until it ships, this - page covers TypeScript, Python, and Java. - -!!! note "Go support" - - The OpenTelemetry plugin for Go is a work in progress. Until it ships, this - page covers TypeScript, Python, and Java. - -!!! note "Rust support" - - The OpenTelemetry plugin for Rust is a work in progress. Until it ships, this - page covers TypeScript, Python, and Java. +The plugins are available for TypeScript, Python, and Java. Go, Rust, and C# do +not yet provide this integration. ## Install the plugin +These examples use the unified factory APIs available in the versions below. +Upgrade the core and plugin together, including copies supplied by Lambda layers. + === "TypeScript" + Requires Node.js 22 or later and OTel plugin 2.1.0 or later. + ```bash - npm install @aws/durable-execution-sdk-js-otel \ - @opentelemetry/api \ - @opentelemetry/core \ - @opentelemetry/sdk-trace-node + npm install @aws/durable-execution-sdk-js@^3.0.0 \ + @aws/durable-execution-sdk-js-otel@^2.1.0 \ + @opentelemetry/api @opentelemetry/core @opentelemetry/sdk-trace-node ``` - The TypeScript plugin requires Node.js 22 or later. The OpenTelemetry - packages are peer dependencies used by the configured provider or telemetry - layer. - === "Python" - When a Lambda telemetry layer supplies the OpenTelemetry libraries: + Requires OTel plugin 2.1.0 or later. + + When a telemetry layer supplies the OpenTelemetry libraries: ```bash - pip install aws-durable-execution-sdk-python-otel + pip install "aws-durable-execution-sdk-python-otel>=2.1.0,<3" ``` - The base package intentionally does not install OpenTelemetry libraries, - which prevents function dependencies from shadowing the version-aligned - libraries in the layer. - - When the application configures its own provider, install the `standalone` - extra: + The package installs a compatible core SDK but leaves OpenTelemetry libraries + to the layer. When the application supplies its own provider, install the + `standalone` extra instead: ```bash - pip install "aws-durable-execution-sdk-python-otel[standalone]" + pip install "aws-durable-execution-sdk-python-otel[standalone]>=2.1.0,<3" ``` === "Java" + Use matching core SDK and plugin versions, at least 3.1.0. + ```xml software.amazon.lambda.durable aws-durable-execution-sdk-java-plugin-otel - ${durable.sdk.version} + 3.1.0 ``` - Add an exporter dependency when the application supplies its own tracer - provider. The ADOT Java agent supplies the export pipeline for the no-arg - plugin constructors. - -## The two plugins +## Choose and register a view -Both plugins emit the same span types in one canonical trace. They differ in the -parent of operation spans, the links on those spans, and when the spans become -visible. Register exactly one of them. +Register one factory using the unified entry point for your language. Omitting +the mode selects the execution view. The SDK creates a separate plugin instance +for each invocation, while the provider can be shared across invocations. -### ExecutionOtelPlugin - -`ExecutionOtelPlugin` provides a workflow-centered view. Operation spans are -children of the `Workflow` span and link to the current `Invocation` span when -they are created. The `Workflow` span is exported only when the execution -reaches a terminal status. An operation that suspends is also left unexported -until it completes. +### Choosing a plugin -Choose this plugin for shorter executions or when the durable workflow hierarchy -is more important than seeing each invocation as it completes. Long-open -workflow spans can exceed the ingestion limits of some observability platforms. +For both views, your observability platform's quotas and limits apply. + +| Consideration | Execution view | Invocation view | +| -------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| Top-level operation parent | `Workflow` | Current `Invocation` | +| Correlation link | Operation or attempt to current `Invocation` | Operation or attempt to `Workflow` | +| Continuation link | Completed operation links only to the invocation that exports it | Later segment links to the initial logical operation span | +| Operation visibility | When the operation completes | At each invocation boundary | +| In-progress visibility | Limited until operations and the execution complete | Each invocation appears as it completes | +| Better for | Shorter executions and a workflow-centered hierarchy | Long executions and progress through individual invocations | +| Platform compatibility | Long-open spans can exceed ingestion time limits | Operation segments stay within an invocation; long Workflow spans still face ingestion limits | + +Nested operations remain under their child-context operation span; attempts +remain under their operation. Both views export `Invocation` spans as invocations +end and `Workflow` only when the SDK observes execution success or failure. +Workflow spans can cover long durations, so backend ingestion limits still apply +to both views. Export and flush are best effort; a timeout, stop, or terminated +runtime can leave spans missing. + +The diagrams below show an execution with a valid remote backend parent and +ambient Lambda spans in the same trace. See [trace identity and sampling](#trace-identity-and-sampling) +for other parent contexts. The tree shows parent-child relationships; arrows +show span links. + +### Execution view + +Top-level operation spans are children of `Workflow` and link to the current `Invocation`. +The factory selects this view by default. An operation that suspends is exported +when it completes, using its durable +start and end times. Its span links to the invocation that exports the completion; +it does not accumulate links to every earlier invocation. === "TypeScript" @@ -149,15 +112,16 @@ workflow spans can exceed the ingestion limits of some observability platforms. --8<-- "examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java" ``` -For an execution that suspends at a wait, the trace resembles: +For an execution that suspends at `cooldown` and resumes in a second invocation, +the trace resembles: ```text Remote backend server span |-- Workflow (exported on terminal status) -| |-- Operation: fetch-data (STEP) -> link to Invocation #1 +| |-- Operation: fetch-data (STEP) -> link to Invocation #1 | | `-- Attempt: fetch-data attempt 1 -> link to Invocation #1 -| |-- Operation: cooldown (WAIT) -> link to Invocation #2 -| `-- Operation: process (STEP) -> link to Invocation #2 +| |-- Operation: cooldown (WAIT) -> link to Invocation #2 +| `-- Operation: process (STEP) -> link to Invocation #2 | `-- Attempt: process attempt 1 -> link to Invocation #2 |-- Ambient Lambda span #1 | `-- Invocation #1 @@ -165,22 +129,15 @@ Remote backend server span `-- Invocation #2 ``` -Each operation or attempt span links only to the current invocation when that -span is created. If an operation starts in one invocation and completes in -another, its exported operation span links to the invocation that completes it; -the plugin does not retain the earlier invocation span ID. These links do not -make the invocation spans children of `Workflow`. - -### InvocationOtelPlugin +The links to `Invocation` do not make invocation spans children of `Workflow`. -`InvocationOtelPlugin` provides an invocation-centered view. Each `Invocation` -span parents the operation and attempt spans emitted during that Lambda -invocation. Those operation and attempt spans link to the deterministic -`Workflow` span for execution-scoped correlation. +### Invocation view -The plugin exports an invocation and its child spans when that invocation ends. -Choose it for long-running executions or when you want traces to appear as the -execution progresses. +Top-level operation spans are children of the `Invocation` that emitted them and link to +`Workflow`, so each invocation's spans export as it ends. Select invocation mode +explicitly. Completed work uses checkpointed results on replay. A resumed operation or +replayed context can produce another span segment without rerunning a completed +step's body. === "TypeScript" @@ -200,11 +157,11 @@ execution progresses. --8<-- "examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java" ``` -For an execution that suspends at a wait, the trace resembles: +The same execution, using invocation view, produces this hierarchy: ```text Remote backend server span -|-- Workflow +|-- Workflow (exported on terminal status) |-- Ambient Lambda span #1 | `-- Invocation #1 | |-- Operation: fetch-data (STEP) -> link to Workflow @@ -220,432 +177,221 @@ Remote backend server span ``` An operation that crosses an invocation boundary can produce more than one -span. A continuation or replay segment uses a new span ID and links to the +span. A continuation segment uses a new span ID and links to the initial logical operation span, which uses the deterministic operation span ID. It also retains the `Workflow` link. CloudWatch does not yet visualize span links. -### Choosing a plugin +## Configure a telemetry layer -For both plugins, your observability platform's quotas and limits apply. - -| Consideration | ExecutionOtelPlugin | InvocationOtelPlugin | -| ---------------------- | --------------------------------------------------------- | ----------------------------------------------------- | -| Operation parent | `Workflow` | Current `Invocation` | -| Correlation link | Operation or attempt to current `Invocation` | Operation or attempt to `Workflow` | -| Continuation link | Reconstructed span links only to current `Invocation` | Later segment links to initial logical operation span | -| Workflow export | On terminal execution status | On terminal execution status | -| Operation visibility | When the operation completes | At each invocation boundary | -| In-progress visibility | Limited until operations and the execution complete | Each invocation appears as it completes | -| Better for | Short executions and a workflow-centered hierarchy | Long executions and an invocation-centered hierarchy | -| Platform compatibility | Long-open spans can exceed platform ingestion time limits | Spans stay within the Lambda invocation time limit | - -## Trace parent and sampling - -The plugins resolve the execution parent and the sampling decision -independently. A missing or unusable `Sampled` field does not replace a valid -remote parent with a synthetic root. - -| `_X_AMZN_TRACE_ID` state | Canonical trace ID | Common execution ancestor | Sampling | -| --------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------- | ----------------------------------------------------------------------- | -| Valid `Root`, `Parent`, `Sampled=1` | Reuse `Root` | Remote `Parent` | Preserve sampled | -| Valid `Root`, `Parent`, `Sampled=0` | Reuse `Root` | Remote `Parent` | Preserve not sampled | -| Valid `Root`, `Parent`, no valid `Sampled` | Reuse `Root` | Remote `Parent` | Leave the sampled trace flag unset; configured sampler behavior applies | -| Valid `Root`, missing or invalid `Parent`, `Sampled=1` or `Sampled=0` | Reuse `Root` | Synthetic execution root | Preserve the explicit decision | -| Valid `Root`, missing or invalid `Parent`, no valid `Sampled` | Reuse `Root` | Synthetic execution root | Configured root sampler decides | -| Missing or invalid `Root` | Derive from the execution ARN and stable execution start time | Synthetic execution root | Configured root sampler decides | - -Only `Sampled=0` and `Sampled=1` carry authoritative upstream decisions. An -absent or unusable value does not mean that the upstream system explicitly -chose not to sample. Its OpenTelemetry span context still represents the value -with an unset sampled bit. - -A `ParentBased` sampler treats a remote parent with an unset sampled bit as not -sampled. A directly configured non-parent-based `TraceIdRatioBased` sampler can -decide from the canonical trace ID instead. The reused or derived trace ID -remains stable across reinvocations, so trace-ID-ratio decisions also remain -stable. Other sampler decisions can change between reinvocations unless the -application persists them. - -When `Root` and `Parent` are valid, the normal hierarchy is: +A language auto-instrumentation layer registers the global provider used by the +default factories. Select a layer compatible with your runtime, architecture, +and region: -```text -Remote backend server span (`Root` / `Parent`) -|-- Workflow -|-- Ambient Lambda span #1 -| `-- Invocation #1 -|-- Ambient Lambda span #2 -| `-- Invocation #2 -`-- Invocation #N (when no valid same-trace ambient span exists) -``` +| Layer | `AWS_LAMBDA_EXEC_WRAPPER` | Setup | +| -------------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------- | +| AWS Distro for OpenTelemetry (ADOT) | `/opt/otel-instrument` | [ADOT Lambda layers](https://aws-otel.github.io/docs/getting-started/lambda) | +| OpenTelemetry community language layer | `/opt/otel-handler` | [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) | -`Workflow` inherits the canonical trace ID and keeps a deterministic, -execution-scoped span ID. An ambient span can parent `Invocation` only when it -uses the same trace ID. A valid remote parent remains the real ancestor when -`Sampled` is missing. The plugins do not create a synthetic-root link to it. +The following SAM snippets show the telemetry settings for an +`AWS::Serverless::Function` resource. Merge the entries under `Properties` with +your function's existing layers, environment variables, and policies. +SAM applies `Policies` only when it creates +the execution role. If you specify `Role`, attach the required permissions +directly to that role. -When the plugins cannot construct a valid remote parent, they use: +### ADOT layer -```text -Synthetic execution root -|-- Workflow -|-- Invocation #1 -|-- Invocation #2 -`-- Invocation #N +For ADOT with X-Ray export, enable `Active` tracing and grant +`AWSXRayDaemonWriteAccess` alongside the durable execution permissions. + +```yaml +MyFunction: + Type: AWS::Serverless::Function + Properties: + Layers: + - + Environment: + Variables: + AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument + Tracing: Active + Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy + - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess ``` -The synthetic execution root has a deterministic span ID that remains stable -across reinvocations and differs from Workflow and operation span IDs. A valid -`Root` still supplies the canonical trace ID. If `Root` is also invalid, the -plugins derive the trace ID from the execution ARN and stable execution start -time. Fallback mode ignores unrelated ambient context. TypeScript, Python, and -Java use this same topology with global and application-owned providers. The -selected sampling decision applies consistently to every descendant. +### Community language layer -## Span structure and attributes +For a community language layer exporting directly to an OTLP backend, use +[`PassThrough` tracing](https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html#services-xray-default). +Lambda passes the upstream X-Ray trace header to the +function without emitting native Lambda X-Ray segments. The layer and durable +plugin still provide application instrumentation. -Both plugins emit four span types. An invocation span covers one Lambda -invocation. Operation spans cover durable operations such as steps, waits, child -contexts, callbacks, and chained invokes. Attempt spans cover user function -attempts, so retries appear as sibling spans. +Propagate the caller's trace context and sampling decision. Without an upstream +header, Lambda generates a header with `Sampled=0`; the documented plugins honor +that decision even with a local `always_on` sampler. See +[supported sampling modes](#supported-sampling-modes). -| Span | Attributes | -| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Workflow | `durable.execution.arn`, `durable.execution.status` | -| Invocation | `durable.execution.arn`, `durable.invocation.status`, `durable.invocation.first` | -| Operation | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status` | -| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.attempt.number`, `durable.attempt.outcome` | +```yaml +MyFunction: + Type: AWS::Serverless::Function + Properties: + Layers: + - + Environment: + Variables: + AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler + OTEL_TRACES_EXPORTER: otlp + OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf + OTEL_EXPORTER_OTLP_ENDPOINT: https:// + OTEL_METRICS_EXPORTER: none + OTEL_LOGS_EXPORTER: none + Tracing: PassThrough + Policies: + - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy +``` -`durable.operation.type` includes `STEP`, `WAIT`, `CONTEXT`, `CHAINED_INVOKE`, -and `CALLBACK`. A `CONTEXT` operation gets an operation span but not a separate -attempt span. +**Exporting to CloudWatch:** +[Enable Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html) +in the destination account and Region, and confirm that **Ingest OpenTelemetry spans** +shows **Enabled**. Otherwise, CloudWatch's OTLP endpoint rejects the spans. This setting +is not required when exporting directly to another observability provider. -## Deploy with the ADOT layer +Configure `OTEL_EXPORTER_OTLP_HEADERS` if your backend requires authentication. +Direct OTLP export to another observability provider does not require X-Ray +write permissions. SAM still adds an X-Ray policy when it creates a role for +`Tracing: PassThrough`. -The AWS Distro for OpenTelemetry (ADOT) Lambda layer supplies a global provider, -auto-instrumentation, and a collector extension. Add the language-specific -layer, set `AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument`, enable active tracing -so the runtime supplies `_X_AMZN_TRACE_ID`, and grant the function role the -`AWSXRayDaemonWriteAccess` managed policy. Auto-instrumentation can then create -an ambient Lambda span in the same trace as the durable backend parent. +With `Active` tracing, a durable span can reference a native Lambda parent sent +only to X-Ray. The plugin preserves that parent without re-exporting it, so a +shared trace ID does not guarantee a complete parent tree in an OTLP backend. +If you need native Lambda platform spans, use an integration that receives both +OTLP spans and their X-Ray ancestors. -The no-arg plugin constructors use the global provider. You do not need a -provider option. +`PassThrough` does not supply missing caller spans. For a complete upstream +hierarchy, export the real caller span to the same backend and propagate its +trace ID, span ID, and sampling decision through `X-Amzn-Trace-Id` when using the +default plugin extractors. Preserve this propagation across downstream durable +invokes. -=== "TypeScript" +You can optionally export telemetry through a +[Collector extension](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector#configuration), +which does not initialize the OpenTelemetry provider required by the plugin. - Find the current ADOT JavaScript layer ARN for your region and architecture - in the - [ADOT Lambda layer documentation](https://aws-otel.github.io/docs/getting-started/lambda#adot-lambda-layer-arns). - - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: nodejs24.x - Handler: index.handler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` +### Provider initialization - ```typescript - new ExecutionOtelPlugin(); - // or - new InvocationOtelPlugin(); - ``` +These requirements apply to both layer choices. -=== "Python" +=== "TypeScript" - Find the current ADOT Python layer ARN for your region and architecture in - the - [ADOT Lambda layer documentation](https://aws-otel.github.io/docs/getting-started/lambda#adot-lambda-layer-arns). - - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: python3.14 - Handler: index.handler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` + The default factory resolves the global provider at invocation start. - ```python - ExecutionOtelPlugin() - # or - InvocationOtelPlugin() - ``` +=== "Python" + + Use the base plugin package with a telemetry layer so function dependencies + do not shadow the layer's OpenTelemetry libraries. === "Java" - The Java agent must also load the plugin JAR as an agent extension so its - auto-configuration SPI can scope deterministic ID generation to durable - spans. Package the JAR in a layer under `java/lib`, then point - `OTEL_JAVAAGENT_EXTENSIONS` to the deployed path. - - Find the current ADOT Java layer ARN in the - [ADOT Java instrumentation releases](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest). - - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: java25 - Handler: com.example.ExampleHandler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - - - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument - OTEL_JAVAAGENT_EXTENSIONS: /opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` + The Java agent must also load the plugin JAR as an extension to install + durable ID generation and sampling. Package the JAR in a layer under + `java/lib`, add that layer to the function, and set: - ```java - new ExecutionOtelPlugin(); - // or - new InvocationOtelPlugin(); + ```text + OTEL_JAVAAGENT_EXTENSIONS=/opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar ``` - To load the plugin from the layer without registering it in handler code, - also set `DURABLE_EXECUTION_PLUGINS` to `otel-execution` or - `otel-invocation`. + Obtain the ADOT Java layer from the + [Java instrumentation releases](https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest). -When the plugins use the global provider, they resolve it at invocation start. -If a usable SDK provider or the integration required for scoped deterministic -IDs is unavailable, all three SDKs log a warning and disable telemetry for that -invocation. They retry provider resolution on the next invocation instead of -emitting spans without deterministic durable IDs. +If the global SDK provider or required durable ID integration is unavailable, +the plugin logs a warning and disables its telemetry for that invocation. +The next invocation retries initialization. -## Deploy with the community auto-instrumentation layer +### Load the factory without handler registration -The OpenTelemetry community language layer initializes auto-instrumentation, -registers a global OpenTelemetry provider, and exports spans using OTLP. Set -`AWS_LAMBDA_EXEC_WRAPPER` to `/opt/otel-handler` and configure the OTLP endpoint -for your observability backend. Enable active tracing so the plugin can reuse -the durable backend parent and the layer can create a same-trace ambient Lambda -span. +`DURABLE_EXECUTION_PLUGINS` selects a default-configured OTel factory from an installed +package or layer. These factories use the global OpenTelemetry tracer provider +initialized by your telemetry layer or application. Choose one view through either +code or environment configuration. -The no-arg plugin constructors use the global provider. You do not need to -configure or pass a provider to the plugin. +The value follows each SDK's discovery mechanism: an importable module specifier +or a registered provider name. -The examples below use OTLP HTTP and disable metrics and log export. If the -backend requires authentication, also configure `OTEL_EXPORTER_OTLP_HEADERS`. +| Language and lookup | Execution view (default) | Invocation view | +| ------------------------------------ | ------------------------------------ | ---------------------------------------------------- | +| TypeScript: Node.js module specifier | `@aws/durable-execution-sdk-js-otel` | `@aws/durable-execution-sdk-js-otel/otel-invocation` | +| Python: package entry-point name | `otel` | `otel-invocation` | +| Java: `ServiceLoader` provider name | `otel` | `otel-invocation` | -Follow the -[OpenTelemetry Lambda auto-instrumentation documentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/) -to obtain the language layer. +!!! warning "Configure exactly one OTel view" -=== "TypeScript" + Registering two distinct OTel factories fails validation because the OTel + views share an exclusive group in each SDK. This applies to naming two views + in `DURABLE_EXECUTION_PLUGINS`, and to naming one there while also registering + a separate factory in handler code. - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: nodejs24.x - Handler: index.handler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler - OTEL_TRACES_EXPORTER: otlp - OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf - OTEL_EXPORTER_OTLP_ENDPOINT: https:// - OTEL_METRICS_EXPORTER: none - OTEL_LOGS_EXPORTER: none - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` +=== "TypeScript" - ```typescript - new ExecutionOtelPlugin(); - // or - new InvocationOtelPlugin(); - ``` + `@aws/durable-execution-sdk-js-otel/otel-execution` selects the same execution + view as the bare `@aws/durable-execution-sdk-js-otel` package name. Configure + only one of these entry points in `DURABLE_EXECUTION_PLUGINS`. === "Python" - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: python3.14 - Handler: index.handler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler - OTEL_TRACES_EXPORTER: otlp - OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf - OTEL_EXPORTER_OTLP_ENDPOINT: https:// - OTEL_METRICS_EXPORTER: none - OTEL_LOGS_EXPORTER: none - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` - - ```python - ExecutionOtelPlugin() - # or - InvocationOtelPlugin() - ``` + `otel-execution` is an alias for the default `otel` execution view. Both names + resolve to the same factory object. If both appear in + `DURABLE_EXECUTION_PLUGINS`, the SDK loads one factory and logs a warning that + it skipped the duplicate. Configure one name. === "Java" - Use the community Java agent layer for auto-instrumentation. The agent must - also load the plugin JAR as an agent extension so its auto-configuration SPI - can scope deterministic ID generation to durable spans. Package the JAR in - a layer under `java/lib`, then point `OTEL_JAVAAGENT_EXTENSIONS` to the - deployed path. - - ```yaml - MyFunction: - Type: AWS::Serverless::Function - Properties: - Runtime: java25 - Handler: com.example.ExampleHandler - Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: - - - Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler - OTEL_TRACES_EXPORTER: otlp - OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf - OTEL_EXPORTER_OTLP_ENDPOINT: https:// - OTEL_METRICS_EXPORTER: none - OTEL_LOGS_EXPORTER: none - OTEL_JAVAAGENT_EXTENSIONS: /opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-.jar - Tracing: Active - Policies: - - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy - - arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess - ``` + `otel-execution` also selects the execution view. Each provider name creates + a separate factory, so `DURABLE_EXECUTION_PLUGINS=otel-execution,otel` fails + at startup: - ```java - new ExecutionOtelPlugin(); - // or - new InvocationOtelPlugin(); + ```text + Conflicting plugin factories otel-execution (ExecutionOtelPlugin) and otel (ExecutionOtelPlugin) in exclusive group 'durable-otel-view'. Configure only one plugin from this group. ``` - To load the plugin from the layer without registering it in handler code, - also set `DURABLE_EXECUTION_PLUGINS` to `otel-execution` or - `otel-invocation`. + Configure one provider name. -### Add a collector extension (optional) +## Configure an application-owned provider -Add the community collector extension layer when you need local processing, -routing, or an exporter that is not available in the language layer. Keep -`AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler` so the language layer continues to -initialize the provider and the ambient Lambda span. +This setup is for advanced use cases that require direct control over the +OpenTelemetry provider. You do not need it when using a +[language auto-instrumentation layer](#configure-a-telemetry-layer): the layer +initializes the global provider used by the plugin's default factories. -Append the collector layer, point the OTLP exporter at its local receiver, and -set the collector configuration path: - -```yaml -Layers: - - - - !Sub arn:aws:lambda:${AWS::Region}::layer:: -Environment: - Variables: - AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler - OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost:4318 - OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml -``` - -For example, include the following `collector.yaml` in the function bundle to -export traces to X-Ray: - -```yaml -receivers: - otlp: - protocols: - http: - endpoint: "localhost:4318" -exporters: - awsxray: - region: "${AWS_REGION}" -service: - pipelines: - traces: - receivers: [otlp] - exporters: [awsxray] -``` - -Change the collector exporter to route spans to another platform. See the -[collector extension documentation](https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector) -for available components and layer deployment instructions. - -### Collector-only setup (advanced) - -Use the collector extension without a language auto-instrumentation layer when -you need to configure the OpenTelemetry provider in application code. Add only -the collector layer, leave `AWS_LAMBDA_EXEC_WRAPPER` unset, keep active tracing -enabled, and pass the application-owned provider to the plugin. Without an -ambient Lambda span, `Invocation` uses the remote backend span directly. - -```yaml -Layers: - - !Sub arn:aws:lambda:${AWS::Region}::layer:: -Environment: - Variables: - OPENTELEMETRY_COLLECTOR_CONFIG_URI: /var/task/collector.yaml -Tracing: Active -``` +Configure the exporter and processors once, then register the resulting factory +as in the handler examples above. The plugin flushes at invocation end but does +not take over provider shutdown. These examples send OTLP HTTP to a collector +listening on loopback port 4318. === "TypeScript" - `tracerProviderFactory` receives a function that creates the plugin's scoped - deterministic ID generator. Use it when constructing the provider. The - application owns provider registration and shutdown. + Install `@opentelemetry/exporter-trace-otlp-http`: ```bash npm install @opentelemetry/exporter-trace-otlp-http ``` + The provider factory receives the durable ID-generator factory; pass its + result to the provider constructor. + ```typescript - import { ExecutionOtelPlugin } from "@aws/durable-execution-sdk-js-otel"; + import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; - import { - BatchSpanProcessor, - NodeTracerProvider, - } from "@opentelemetry/sdk-trace-node"; + import { BatchSpanProcessor, NodeTracerProvider } from "@opentelemetry/sdk-trace-node"; - const plugin = new ExecutionOtelPlugin({ + const pluginFactory = createOtelPluginFactory({ tracerProviderFactory: (createIdGenerator) => { const provider = new NodeTracerProvider({ idGenerator: createIdGenerator(), - spanProcessors: [ - new BatchSpanProcessor( - new OTLPTraceExporter({ - url: "http://localhost:4318/v1/traces", - }), - ), - ], + spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter({ + url: "http://localhost:4318/v1/traces", + }))], }); provider.register(); return provider; @@ -653,270 +399,406 @@ Tracing: Active }); ``` - Substitute `InvocationOtelPlugin` to use the invocation-centered view. To - preserve a custom application ID generator for unrelated spans, pass it to - `createIdGenerator(applicationIdGenerator)`. + Set `mode: "invocation"` in the same options object to select the invocation view. === "Python" - Install the standalone dependencies and an OTLP HTTP exporter: - - ```bash - pip install "aws-durable-execution-sdk-python-otel[standalone]" \ - opentelemetry-exporter-otlp-proto-http - ``` - - Pass the application-owned provider through the shared configuration - object: + Install the `standalone` extra and `opentelemetry-exporter-otlp-proto-http`. ```python + from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor - from aws_durable_execution_sdk_python_otel import ( - ExecutionOtelPlugin, + create_otel_plugin_factory, OtelPluginConfig, ) provider = TracerProvider() - provider.add_span_processor( - BatchSpanProcessor( - OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces") - ) + provider.add_span_processor(BatchSpanProcessor( + OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces") + )) + trace.set_tracer_provider(provider) + plugin_factory = create_otel_plugin_factory( + OtelPluginConfig(tracer_provider=provider) ) + ``` - plugin = ExecutionOtelPlugin( - OtelPluginConfig(tracer_provider=provider) + For invocation view, replace the factory call with: + + ```python + plugin_factory = create_otel_plugin_factory( + OtelPluginConfig(tracer_provider=provider), mode="invocation" ) ``` - Substitute `InvocationOtelPlugin` to use the invocation-centered view. + `tracer_provider` is what the plugin uses, so `set_tracer_provider` is only + needed to point *other* instrumentation at the same provider. It takes effect + once per process and is ignored with a warning if a telemetry layer already + registered a global provider; omit it in that case. === "Java" - Add the OpenTelemetry OTLP exporter dependency, then pass a provider builder - to either plugin. The plugin wraps the builder's ID generator and builds the - application-owned provider. + Add `io.opentelemetry:opentelemetry-exporter-otlp` to your dependencies, + using OpenTelemetry 1.67.0 to match plugin 3.1.0: ```xml io.opentelemetry opentelemetry-exporter-otlp - ${opentelemetry.version} + 1.67.0 ``` + The factory wraps the provider builder's ID generator and sampler, then + builds the provider once. + ```java import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter; import io.opentelemetry.sdk.trace.SdkTracerProvider; import io.opentelemetry.sdk.trace.export.BatchSpanProcessor; - import software.amazon.lambda.durable.otel.ExecutionOtelPlugin; + import software.amazon.lambda.durable.otel.OtelPlugin; var exporter = OtlpHttpSpanExporter.builder() .setEndpoint("http://localhost:4318/v1/traces") .build(); - - var providerBuilder = SdkTracerProvider.builder() - .addSpanProcessor(BatchSpanProcessor.builder(exporter).build()); - - var plugin = new ExecutionOtelPlugin(providerBuilder); + var pluginFactory = OtelPlugin.factory( + OtelPlugin.Mode.EXECUTION, + SdkTracerProvider.builder() + .addSpanProcessor(BatchSpanProcessor.builder(exporter).build())); ``` - Substitute `InvocationOtelPlugin` to use the invocation-centered view. + Use `OtelPlugin.Mode.INVOCATION` as the first argument to select the invocation view. -## Configuration +### Plugin options === "TypeScript" - Both plugins accept an `OtelPluginConfig` object: + Pass `OtelPluginFactoryOptions` to `createOtelPluginFactory(options)`. These + options extend `OtelPluginConfig` with the view selection. - - **tracerProviderFactory** Creates an application-owned provider. Omit it - to use the global provider. - - **contextExtractor** Extracts upstream trace context. Defaults to - `xRayContextExtractor`. Use `w3cClientContextExtractor` for W3C - `traceparent` data in Lambda client context. - - **instrumentationName** Instrumentation scope name. Defaults to - `aws-durable-execution-sdk-js`. - - **workflowSpanName** Name of the execution-scoped `Workflow` span. - Defaults to `Workflow`. - - **enrichLogger** Adds `traceId`, `spanId`, and `otelTraceSampled` to each - durable log record. Defaults to `true`. + | Option | Default | Purpose | + | ----------------------- | ------------------------------ | ----------------------------------------------------------------- | + | `mode` | `"execution"` | Select `"execution"` or `"invocation"` | + | `tracerProviderFactory` | Global provider | Build an application-owned provider | + | `contextExtractor` | `xRayContextExtractor` | Extract this execution's upstream context | + | `instrumentationName` | `aws-durable-execution-sdk-js` | Instrumentation scope | + | `workflowSpanName` | `Workflow` | Workflow span name | + | `enrichLogger` | `true` | Add trace fields to durable logs | + | `flushTimeoutMillis` | `1000` | Total time budget for awaiting provider flushes at invocation end | - The application or telemetry layer owns exporters, propagators, resources, - sampling, and HTTP or AWS SDK instrumentation. + The default X-Ray extractor reads invocation-local `xRayTraceId` when + present, otherwise `_X_AMZN_TRACE_ID`. + A flush timeout stops waiting; it does not cancel exports or synchronous + provider work. === "Python" - Both plugins accept the same `OtelPluginConfig` dataclass: + Pass `OtelPluginConfig` to `create_otel_plugin_factory(config)`. Select the + invocation view with `create_otel_plugin_factory(config, mode="invocation")`; + `mode` is a keyword argument, separate from `OtelPluginConfig`. - - **tracer_provider** Application-owned provider. Omit it to use the global - provider. - - **context_extractor** Defaults to `xray_context_extractor`. Use - `w3c_client_context_extractor` for W3C `traceparent` data in Lambda - client context. - - **instrument_name** Instrumentation scope name. Defaults to - `aws-durable-execution-sdk-python`. - - **workflow_span_name** Name of the execution-scoped `Workflow` span. - Defaults to `Workflow`. - - **enrich_logger** Installs a root-logger filter that stamps trace context - onto log records. Defaults to `True`. + | Option | Default | Purpose | + | -------------------- | ---------------------------------- | ------------------------------------------------ | + | `tracer_provider` | Global provider | Use an application-owned SDK provider | + | `context_extractor` | `xray_context_extractor` | Extract upstream context from `_X_AMZN_TRACE_ID` | + | `instrument_name` | `aws-durable-execution-sdk-python` | Instrumentation scope | + | `workflow_span_name` | `Workflow` | Workflow span name | + | `enrich_logger` | `True` | Add trace fields through root logger handlers | - ```python - plugin = InvocationOtelPlugin( - OtelPluginConfig( - instrument_name="my-service", - enrich_logger=False, - ) - ) - ``` + Use the default X-Ray extractor or implement a custom extractor. === "Java" - Each plugin supports the following constructor forms: + `OtelPlugin` exposes `factory()`, `factory(mode)`, `factory(mode, config)`, + `factory(mode, providerBuilder)`, and `factory(mode, providerBuilder, config)`. + Use `OtelPlugin.Mode.EXECUTION` or `OtelPlugin.Mode.INVOCATION` for `mode`. + Only `factory()` omits the mode and defaults to execution. Build + `OtelPluginConfig` with: - ```java - new InvocationOtelPlugin(); - new InvocationOtelPlugin(otelPluginConfig); - new InvocationOtelPlugin(tracerProviderBuilder); - new InvocationOtelPlugin(tracerProviderBuilder, otelPluginConfig); - ``` + | Builder method | Default | Purpose | + | -------------------------- | -------------------------------- | ------------------------------ | + | `contextExtractor(...)` | `new XRayContextExtractor()` | Extract upstream trace context | + | `instrumentationName(...)` | `aws-durable-execution-sdk-java` | Instrumentation scope | + | `workflowSpanName(...)` | `Workflow` | Workflow span name | + | `enableMdc(...)` | `true` | Add trace fields to SLF4J MDC | - `ExecutionOtelPlugin` provides the same forms. The no-arg and config-only - constructors use the global provider. The builder constructors create an - application-owned provider. +A custom extractor must return the durable execution's own context consistently +across resumes. An unrelated or per-invocation trace ID splits the execution trace. - Build `OtelPluginConfig` with: +## Trace identity and sampling - - **contextExtractor(...)** Defaults to `new XRayContextExtractor()`. - - **enableMdc(...)** Adds `traceId`, `spanId`, and `otelTraceSampled` to the - SLF4J MDC. Defaults to `true`. - - **workflowSpanName(...)** Defaults to `Workflow`. - - **instrumentationName(...)** Defaults to - `aws-durable-execution-sdk-java`. +Both views keep one canonical trace ID per execution. A valid propagated X-Ray +`Root` supplies that ID, and a valid `Parent` becomes the remote execution +ancestor. Missing `Sampled` does not invalidate a complete remote parent. +`Workflow` has a stable execution-scoped span ID and parents to that ancestor. +`Invocation` uses an active ambient span, such as the Lambda span, only when it +belongs to the same trace; otherwise it parents to the execution ancestor. - ```java - var config = OtelPluginConfig.builder() - .instrumentationName("my-service") - .enableMdc(false) - .build(); +```text +Execution ancestor +|-- Workflow +|-- Ambient Lambda span #1 +| `-- Invocation #1 +`-- Ambient Lambda span #2 + `-- Invocation #2 +``` - var plugin = new InvocationOtelPlugin(config); - ``` +When no complete remote parent is available, the SDK uses a synthetic +`DurableExecutionRoot`. It retains a valid propagated trace ID, or derives one +from the execution ARN and stable execution start time. With a supported provider +and that start time, sampled invocations export a zero-duration root with stable +identity to preserve ancestry before the execution completes. Resumes can export +another copy. `Workflow` carries duration and outcome; the fallback root carries +neither. Remote backend spans are owned by the backend and are never exported +by the plugin. + +Configure a [supported sampling mode](#supported-sampling-modes) through the provider, +or through `OTEL_TRACES_SAMPLER` and +`OTEL_TRACES_SAMPLER_ARG` when using auto-configuration. Keep the plugin's sampler +integration installed so durable spans share the resolved decision while other +instrumentation retains its own policy. -Configure sampling through the provider. With ADOT, use the standard -`OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` environment variables. +=== "TypeScript" -## Load plugins dynamically + An explicit `Sampled=1` or `Sampled=0` on a valid extracted trace takes + precedence. Otherwise, the configured sampler evaluates its root policy + using the canonical trace ID. Missing `Sampled` therefore does not make a + `ParentBased` sampler automatically drop the trace. -The SDK can discover a plugin from a Lambda layer or installed package without -handler registration code. Dynamic loading uses default configuration and -therefore requires a global OpenTelemetry provider. + A tracer that hides its sampler cannot enforce a shared durable sampling + decision and retains terminal-only fallback-root export. + +=== "Python" + + Sampling follows the explicit backend decision, then the recording/sampled + state of a same-trace ambient span, then the configured sampler's root policy. + The plugin resolves the result once per invocation for its durable spans. + +=== "Java" + + Sampling follows the explicit backend decision, then a same-trace ambient + span, then the configured sampler. Retain `DurableSampler`, installed by the + provider-builder and agent-extension paths, as the final sampler. Replacing + it with a plain sampler removes the shared-decision guarantee. + +### Supported sampling modes + +These rules apply to TypeScript core 3.0.0 / OTel 2.1.0, Python core 3.0.2 / OTel 2.1.0, +and Java SDK/plugin 3.1.0, in both views. These versions include the unified +factory APIs used by this guide. + +- **Explicit upstream decision:** a valid `Sampled=0` or `Sampled=1` overrides the + local sampler for durable spans. Thus `Sampled=1` still exports durable spans + with local `always_off`, and `Sampled=0` drops them with local `always_on`. + Active tracing alone does not imply `Sampled=1`. +- **Otherwise:** use a deterministic policy based on the canonical trace ID, such + as `always_on`, `always_off`, or `traceidratio` with a fixed ratio. For + `traceidratio`, `OTEL_TRACES_SAMPLER_ARG=0.5` selects whole execution traces when + the local policy is consulted. + +Keep the policy, ratio, and any inherited upstream or ambient decision stable across +retries, resumes, and processes. Random, time-, load-, or mutable-state-dependent +policies are unsupported and can produce partial traces. The plugins do not +checkpoint sampling decisions; caching or application persistence does not make +another policy supported. + +The provider or auto-configuration layer resolves the environment settings, defaults, +and invalid values. They do not override an explicitly configured application +provider. Keep the plugin's sampler integration installed; other application and +ADOT spans retain their own policy. + +Sampling behavior may change in future SDK/plugin releases. +[Issue #295](https://github.com/aws/aws-durable-execution-docs/issues/295) proposes +letting the configured sampler decide whether to inherit upstream sampling, using +standard direct and `parentbased_*` semantics. That proposal is not implemented in +the versions listed above. Check the release notes and this section when upgrading. + +Sampling does not guarantee export or a complete parent tree: a durable span may be +sampled while its application parent is dropped, or its native X-Ray parent may be +absent from OTLP. See the [community layer setup](#community-language-layer) for +trace propagation and native X-Ray parents. + +## Span attributes and failures + +Optional attributes such as `durable.operation.name` and +`durable.operation.subtype` are emitted only when supplied. + +| Span | Main attributes | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Workflow | `durable.execution.arn`, `durable.execution.status` | +| Invocation | `durable.execution.arn`, `durable.invocation.status`, `durable.invocation.first` | +| Operation | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.operation.subtype`, `durable.operation.status` | +| Attempt | `durable.execution.arn`, `durable.operation.id`, `durable.operation.type`, `durable.operation.name`, `durable.attempt.number`, `durable.attempt.outcome` | + +Operation types include `STEP`, `WAIT`, `CONTEXT`, `CHAINED_INVOKE`, and `CALLBACK`. +A context has an operation span without a separate attempt span. An invocation +that will retry reports `durable.invocation.status=RETRYING` with OTel `UNSET`. +A caught operation failure can have an `ERROR` span while the enclosing invocation +and workflow succeed. Inspect operation status as well as exception events; +a backend can report failure without diagnostic details. === "TypeScript" - ```text - DURABLE_EXECUTION_PLUGINS=@aws/durable-execution-sdk-js-otel/otel-invocation + `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without + diagnostics. The plugins export newly observed external completion updates + for waits, callbacks, and invokes when their metadata is visible, even if replay + skips the operation's body. When completed child contexts would otherwise + prune that metadata, configure + `pluginsConfig.childOperationsDepth` as described in + [operation detail](telemetry/exporter.md#operation-detail). + +=== "Python" + + `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without an + error object. Exception events preserve the checkpointed type, message, and + stack trace; missing fields are omitted. No exception event is emitted when + both type and message are absent. Terminal operation and attempt exception + events use the span end timestamp. + + Completion spans can be missing when replay returns a cached parent result + without visiting its children, even if their operation metadata is still + visible. Pruned operation metadata cannot be reconstructed. + +=== "Java" + + `FAILED`, `TIMED_OUT`, and `STOPPED` operations set `ERROR` even without + diagnostics. Missing details do not produce an exception event. Execution-view + operation exceptions use the known backend end timestamp when available; + invocation-view exceptions retain normal OpenTelemetry event timing. + Since SDK/plugin 3.0.1, operation exception events report the stored original + error type when available, including errors from another language, while + retaining the throwable's message and stack diagnostics. + +## Correlate logs and verify traces + +Log correlation is enabled by default. `traceId` identifies the execution trace, +`spanId` identifies the active span, and `otelTraceSampled` carries its sampled +flag. Application-created spans on that trace can become the active context too. + +=== "TypeScript" + + `enrichLogger` defaults to `true`. The default logger includes trace fields + when you call `context.logger.info()` or another log method. + + For a custom Powertools logger, use `createPowertoolsLogger` to include the + same fields. Install Powertools alongside the SDK and plugin: + + ```bash + npm install @aws-lambda-powertools/logger + ``` + + With the OpenTelemetry provider configured as shown above: + + ```typescript + --8<-- "examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts" ``` - Use `@aws/durable-execution-sdk-js-otel/otel-execution` for the - workflow-centered plugin. + The resulting JSON record includes these fields; trace and span IDs vary: + + ```json + { + "message": "Processing order", + "traceId": "1234567890abcdef1234567890abcdef", + "spanId": "1234567890abcdef", + "otelTraceSampled": true + } + ``` + + The adapter reads the current trace fields on each log call. In execution + view, handler code between operations uses `Workflow`; invocation view uses + `Invocation`. Step bodies use their attempt span. If you write your own + adapter, call `getLogEnrichment()` from its `DurableLoggingContext` on each + log call and include the returned fields in the output. See + [custom loggers](logging.md#custom-logger). === "Python" + `enrich_logger` defaults to `True`. Its filter adds trace fields to log + records processed by the root logger's handlers. For standard-library text + logging, configure the formatter to print those fields: + + ```python + --8<-- "examples/python/sdk-reference/observability/opentelemetry/log-correlation.py" + ``` + + Configure the OpenTelemetry provider as shown above. The example keeps + existing handlers and sets their formatter before the first invocation. + Initialization logs and threads without invocation context have no trace + fields, so the formatter supplies `-` for missing values: + ```text - DURABLE_EXECUTION_PLUGINS=otel-invocation + Function initialized traceId=- spanId=- otelTraceSampled=- + Processing request traceId=1234567890abcdef1234567890abcdef spanId=1234567890abcdef otelTraceSampled=True ``` - Use `otel-execution` for the workflow-centered plugin. + Trace and span IDs vary. Handler code uses `Invocation`, step bodies use + their attempt, and child-context bodies use their context span. === "Java" - ```text - DURABLE_EXECUTION_PLUGINS=otel-invocation + `enableMdc` defaults to `true`. The plugin puts the trace fields in SLF4J's + mapped diagnostic context (MDC). Your logging layout must include those + fields to print them. + + For text logging with Logback, add its SLF4J provider to your `pom.xml`: + + ```xml + + ch.qos.logback + logback-classic + 1.6.6 + runtime + ``` - Use `otel-execution` for the workflow-centered plugin. Put the plugin JAR - under `java/lib` in the layer so it is on the Lambda class path. + Save this configuration as `src/main/resources/logback.xml`. If you already + have a Logback configuration, add the three `%X` fields to its existing + encoder pattern: -## Correlate logs with traces + ```xml title="src/main/resources/logback.xml" + --8<-- "examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml" + ``` -The plugin stamps active trace and span IDs onto log records. Logs from every -reinvocation of one execution use the same canonical `traceId`. The `spanId` -identifies the active invocation, operation, or attempt, and -`otelTraceSampled` reflects the effective sampled flag. See -[Logging](logging.md) for the SDK logger. + With the OpenTelemetry provider configured as shown above, log through + the durable context: -=== "TypeScript" + ```java + --8<-- "examples/java/sdk-reference/observability/opentelemetry/log-correlation.java" + ``` - With `enrichLogger` enabled, the plugin adds `traceId`, `spanId`, and - `otelTraceSampled` to each durable log record through the - `enrichLogContext()` hook described in - [Logging from a plugin](plugins.md#logging-from-a-plugin). + The `%X` expressions read the current MDC fields. Their defaults print `-` + when a field is absent. With the default execution view in SDK/plugin + 3.1.0, the plugin sets only `traceId` between operations. The step log + includes all three fields and identifies its attempt span: -=== "Python" + ```text + Function initialized traceId=- spanId=- otelTraceSampled=- + Processing request traceId=1234567890abcdef1234567890abcdef spanId=- otelTraceSampled=- + Processing step traceId=1234567890abcdef1234567890abcdef spanId=1234567890abcdef otelTraceSampled=true + ``` - With `enrich_logger=True`, the plugin installs a filter on root logger - handlers. It adds `traceId`, `spanId`, and `otelTraceSampled` when a valid - span is active. Treat these fields as optional in your formatter. + Trace and span IDs vary. In CloudWatch, use **Group by nodes** to inspect + the hierarchy; the ungrouped X-Ray Segments Timeline cannot nest OTLP spans + under the Lambda platform segment. If spans are missing when using a Java + agent, check `OTEL_JAVAAGENT_EXTENSIONS`. -=== "Java" +Test with an execution that waits and resumes. Confirm that its spans share a +trace ID, invocations appear as they finish, completed steps do not run again, +and logs identify the active span. `Workflow` appears after terminal completion. +Span links correlate the two views but do not change parent-child relationships; +CloudWatch does not currently visualize these links. - With `enableMdc(true)`, the plugin puts `traceId`, `spanId`, and - `otelTraceSampled` into the SLF4J MDC. Configure your logging framework to - include MDC fields in its output. - -## Verify - -Invoke a durable function that includes a wait or resume so it runs across more -than one Lambda invocation. In CloudWatch Traces, expect one canonical trace ID -for every span carrying the execution ARN: - -- The durable backend server span is the common ancestor when CloudWatch - returns it. `Workflow` is its direct child. -- Each `Invocation` is a child of a same-trace ambient Lambda span, or a direct - child of the backend server span when no valid ambient span exists. -- `ExecutionOtelPlugin` operations are children of `Workflow` and link only to - the current invocation when each span is created. -- `InvocationOtelPlugin` operation segments are children of `Invocation` and - link to `Workflow`. A resumed wait or replayed operation also links to its - initial logical operation span. -- Log records retain the canonical `traceId`. Their `spanId` values match the - active invocation, operation, or attempt. - -Some backend queries omit the remote backend span. In that case, `Workflow` and -directly parented `Invocation` spans still contain its non-null parent span ID. -Do not interpret the unresolved parent as a parentless Workflow span. - -For Java, use the CloudWatch **Group by nodes** view to inspect the hierarchy. -The ungrouped X-Ray Segments Timeline cannot attach OTLP-exported spans beneath -the Lambda platform segment. - -When you use the community collector layer, enable CloudWatch Transaction -Search in your account for traces to appear. - -If no plugin spans appear: - -- Confirm a provider with a span processor and exporter is registered. -- Check for a warning that the global SDK provider or its deterministic ID - integration was unavailable at invocation start. -- With ADOT, confirm `AWS_LAMBDA_EXEC_WRAPPER` and active tracing are enabled. -- With the community collector, confirm the layer and - `OPENTELEMETRY_COLLECTOR_CONFIG_URI` are configured. -- For Java ADOT, confirm the plugin JAR is listed in - `OTEL_JAVAAGENT_EXTENSIONS`. -- Check `_X_AMZN_TRACE_ID` and provider sampling configuration. An explicit - `Sampled=0` suppresses export, and `ParentBased` treats a missing sampled bit - as not sampled. - -If spans carrying one execution ARN use different trace IDs, confirm active -tracing is enabled and that every invocation uses the same current plugin and -context extractor. Missing or invalid remote parent data should still produce -one canonical fallback trace, not one trace per invocation. +If spans are missing, check provider warnings, exporter configuration, the layer's +execution wrapper, and [sampling](#supported-sampling-modes). +`Workflow` requires a terminal invocation to export it. An unresolved parent can +belong to the remote backend or an undelivered fallback root. ## See also - [Plugins](plugins.md) -- [Durable Execution Telemetry](telemetry/index.md) - [Logging](logging.md) +- [Durable Execution Telemetry](telemetry/index.md) - [Steps](../operations/step.md) +- [Feedback and discussion](https://github.com/aws/aws-durable-execution-docs/discussions/206) diff --git a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java index c1541cbf..6ee6922e 100644 --- a/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java +++ b/examples/java/sdk-reference/observability/opentelemetry/execution-plugin.java @@ -3,16 +3,16 @@ import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.DurableContext; import software.amazon.lambda.durable.DurableHandler; -import software.amazon.lambda.durable.otel.ExecutionOtelPlugin; +import software.amazon.lambda.durable.otel.OtelPlugin; public class ExecutionOtelExample extends DurableHandler, String> { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new ExecutionOtelPlugin()).build(); + return DurableConfig.builder().withPlugins(OtelPlugin.factory()).build(); } @Override - protected String handleRequest(Map event, DurableContext context) { + public String handleRequest(Map event, DurableContext context) { String data = context.step("fetch-data", String.class, ctx -> event.get("id")); context.wait("cooldown", Duration.ofSeconds(5)); return context.step("process", String.class, ctx -> "processed " + data); diff --git a/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java b/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java index 6ab7ec2c..2949cb3c 100644 --- a/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java +++ b/examples/java/sdk-reference/observability/opentelemetry/invocation-plugin.java @@ -1,16 +1,16 @@ import software.amazon.lambda.durable.DurableConfig; import software.amazon.lambda.durable.DurableContext; import software.amazon.lambda.durable.DurableHandler; -import software.amazon.lambda.durable.otel.InvocationOtelPlugin; +import software.amazon.lambda.durable.otel.OtelPlugin; public class InvocationOtelExample extends DurableHandler { @Override protected DurableConfig createConfiguration() { - return DurableConfig.builder().withPlugins(new InvocationOtelPlugin()).build(); + return DurableConfig.builder().withPlugins(OtelPlugin.factory(OtelPlugin.Mode.INVOCATION)).build(); } @Override - protected String handleRequest(Object event, DurableContext context) { + public String handleRequest(Object event, DurableContext context) { return context.step("process", String.class, ctx -> "done"); } } diff --git a/examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml b/examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml new file mode 100644 index 00000000..165e6265 --- /dev/null +++ b/examples/java/sdk-reference/observability/opentelemetry/log-correlation-logback.xml @@ -0,0 +1,10 @@ + + + + %msg traceId=%X{traceId:--} spanId=%X{spanId:--} otelTraceSampled=%X{otelTraceSampled:--}%n + + + + + + diff --git a/examples/java/sdk-reference/observability/opentelemetry/log-correlation.java b/examples/java/sdk-reference/observability/opentelemetry/log-correlation.java new file mode 100644 index 00000000..f2a8cab7 --- /dev/null +++ b/examples/java/sdk-reference/observability/opentelemetry/log-correlation.java @@ -0,0 +1,26 @@ +import java.util.Map; +import org.slf4j.LoggerFactory; +import software.amazon.lambda.durable.DurableConfig; +import software.amazon.lambda.durable.DurableContext; +import software.amazon.lambda.durable.DurableHandler; +import software.amazon.lambda.durable.otel.OtelPlugin; + +public class LogCorrelationExample extends DurableHandler, String> { + static { + LoggerFactory.getLogger(LogCorrelationExample.class).info("Function initialized"); + } + + @Override + protected DurableConfig createConfiguration() { + return DurableConfig.builder().withPlugins(OtelPlugin.factory()).build(); + } + + @Override + public String handleRequest(Map event, DurableContext context) { + context.getLogger().info("Processing request"); + return context.step("process", String.class, stepContext -> { + stepContext.getLogger().info("Processing step"); + return "done"; + }); + } +} diff --git a/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py b/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py index 0a1b7327..33329994 100644 --- a/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py +++ b/examples/python/sdk-reference/observability/opentelemetry/execution-plugin.py @@ -1,9 +1,9 @@ from aws_durable_execution_sdk_python import DurableContext, durable_execution from aws_durable_execution_sdk_python.config import Duration -from aws_durable_execution_sdk_python_otel import ExecutionOtelPlugin +from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory -@durable_execution(plugins=[ExecutionOtelPlugin()]) +@durable_execution(plugins=[create_otel_plugin_factory()]) def handler(event: dict, context: DurableContext) -> str: data = context.step(lambda ctx: event["id"], name="fetch-data") context.wait(Duration.from_seconds(5), name="cooldown") diff --git a/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py b/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py index 0d6d5f34..502fabfc 100644 --- a/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py +++ b/examples/python/sdk-reference/observability/opentelemetry/invocation-plugin.py @@ -1,7 +1,7 @@ from aws_durable_execution_sdk_python import DurableContext, durable_execution -from aws_durable_execution_sdk_python_otel import InvocationOtelPlugin +from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory -@durable_execution(plugins=[InvocationOtelPlugin()]) +@durable_execution(plugins=[create_otel_plugin_factory(mode="invocation")]) def handler(event: dict, context: DurableContext) -> str: return context.step(lambda ctx: "done", name="process") diff --git a/examples/python/sdk-reference/observability/opentelemetry/log-correlation.py b/examples/python/sdk-reference/observability/opentelemetry/log-correlation.py new file mode 100644 index 00000000..8ecb6b9e --- /dev/null +++ b/examples/python/sdk-reference/observability/opentelemetry/log-correlation.py @@ -0,0 +1,23 @@ +import logging + +from aws_durable_execution_sdk_python import DurableContext, durable_execution +from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory + + +root_logger = logging.getLogger() +root_logger.setLevel(logging.INFO) +formatter = logging.Formatter( + "%(message)s traceId=%(traceId)s spanId=%(spanId)s " + "otelTraceSampled=%(otelTraceSampled)s", + defaults={"traceId": "-", "spanId": "-", "otelTraceSampled": "-"}, +) +for log_handler in root_logger.handlers: + log_handler.setFormatter(formatter) + +root_logger.info("Function initialized") + + +@durable_execution(plugins=[create_otel_plugin_factory()]) +def handler(event: dict, context: DurableContext) -> str: + context.logger.info("Processing request") + return "done" diff --git a/examples/typescript/sdk-reference/observability/custom-logger.ts b/examples/typescript/sdk-reference/observability/custom-logger.ts index e6d76ea8..4cb86180 100644 --- a/examples/typescript/sdk-reference/observability/custom-logger.ts +++ b/examples/typescript/sdk-reference/observability/custom-logger.ts @@ -1,9 +1,13 @@ -import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; +import { + createPowertoolsLogger, + DurableContext, + withDurableExecution, +} from "@aws/durable-execution-sdk-js"; import { Logger } from "@aws-lambda-powertools/logger"; const powertoolsLogger = new Logger({ serviceName: "my-service" }); export const handler = withDurableExecution(async (event, context: DurableContext) => { - context.configureLogger({ customLogger: powertoolsLogger }); + context.configureLogger({ customLogger: createPowertoolsLogger(powertoolsLogger) }); context.logger.info("Using Powertools logger"); }); diff --git a/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts b/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts index 07aef2fd..90d25c0f 100644 --- a/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts +++ b/examples/typescript/sdk-reference/observability/opentelemetry/execution-plugin.ts @@ -1,5 +1,5 @@ import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; -import { ExecutionOtelPlugin } from "@aws/durable-execution-sdk-js-otel"; +import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; export const handler = withDurableExecution( async (event: { id: string }, context: DurableContext) => { @@ -7,5 +7,5 @@ export const handler = withDurableExecution( await context.wait("cooldown", { seconds: 5 }); return await context.step("process", async () => `processed ${data}`); }, - { plugins: [new ExecutionOtelPlugin()] }, + { plugins: [createOtelPluginFactory()] }, ); diff --git a/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts b/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts index 3d4a8927..336d2f85 100644 --- a/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts +++ b/examples/typescript/sdk-reference/observability/opentelemetry/invocation-plugin.ts @@ -1,9 +1,9 @@ import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; -import { InvocationOtelPlugin } from "@aws/durable-execution-sdk-js-otel"; +import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; export const handler = withDurableExecution( async (event, context: DurableContext) => { return await context.step("process", async () => "done"); }, - { plugins: [new InvocationOtelPlugin()] }, + { plugins: [createOtelPluginFactory({ mode: "invocation" })] }, ); diff --git a/examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts b/examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts new file mode 100644 index 00000000..75fd0f0f --- /dev/null +++ b/examples/typescript/sdk-reference/observability/opentelemetry/log-correlation.ts @@ -0,0 +1,18 @@ +import { Logger } from "@aws-lambda-powertools/logger"; +import { + createPowertoolsLogger, + DurableContext, + withDurableExecution, +} from "@aws/durable-execution-sdk-js"; +import { createOtelPluginFactory } from "@aws/durable-execution-sdk-js-otel"; + +const logger = createPowertoolsLogger(new Logger({ serviceName: "orders" })); + +export const handler = withDurableExecution( + async (_event: unknown, context: DurableContext) => { + context.configureLogger({ customLogger: logger }); + context.logger.info("Processing order"); + return "processed"; + }, + { plugins: [createOtelPluginFactory()] }, +); diff --git a/examples/typescript/sdk-reference/observability/powertools-logger.ts b/examples/typescript/sdk-reference/observability/powertools-logger.ts index 88a94ee2..3e248f53 100644 --- a/examples/typescript/sdk-reference/observability/powertools-logger.ts +++ b/examples/typescript/sdk-reference/observability/powertools-logger.ts @@ -1,9 +1,13 @@ -import { Logger } from '@aws-lambda-powertools/logger'; -import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js"; +import { Logger } from "@aws-lambda-powertools/logger"; +import { + createPowertoolsLogger, + DurableContext, + withDurableExecution, +} from "@aws/durable-execution-sdk-js"; -const powertoolsLogger = new Logger({ serviceName: 'my-service' }); +const powertoolsLogger = new Logger({ serviceName: "my-service" }); export const handler = withDurableExecution(async (event, context: DurableContext) => { - context.configureLogger({ customLogger: powertoolsLogger }); - context.logger.info('Running handler'); + context.configureLogger({ customLogger: createPowertoolsLogger(powertoolsLogger) }); + context.logger.info("Running handler"); });