Skip to content

[Feature]: Switch existing S3 OTel tests to PassThrough and validate complete trace hierarchies #180

Open

Description

@zhongkechen

Problem / motivation

The existing S3 collector tests can pass while referenced parent spans are absent from the OTLP capture. That does not establish whether users can obtain a complete trace hierarchy through an OTLP-only integration.

The remaining validation work in docs #297 should be implemented by switching the existing S3 collector test configuration to Lambda PassThrough, exporting a real caller span into the same capture, and requiring complete parent relationships for sampled traces. This replaces the current S3 configuration; it is not a separate optional scenario alongside S3 tests that continue accepting missing parents.

With Active Tracing, a valid parent can identify a native Lambda segment exported only to X-Ray. Docs PR #302 explains that limitation, but the proposed caller-to-OTLP plus PassThrough setup remains unvalidated end to end.

Proposed solution

Use the existing backend split to test both configurations:

Existing test configuration Target behavior
ADOT / X-Ray Keep the current Active Tracing configuration and assertions unchanged.
Community instrumentation / S3 collector Switch the existing tests to PassThrough, export the real caller and application ancestors into the same S3-backed OTLP capture, and enforce complete parent hierarchies for sampled traces.

Apply the S3 change to the existing hosted short and long-running OTel suites, both Execution and Invocation views, and the JavaScript, Python, and Java integrations. Preserve their existing operation, replay, lifecycle, and link assertions while adding ancestry completeness.

This repository owns the shared runner/caller coordination, requirements, collector retrieval, validators, and reusable workflows. SDK repositories own the language-specific handlers, instrumentation, and deployment templates. Select tracing configuration and validation rules by the test configuration so the S3 migration does not change X-Ray behavior.

Acceptance criteria:

  • Switch the existing S3 collector deployments to TracingConfig.Mode=PassThrough, retaining the OTel layer/provider and durable plugin. Apply consistent propagation and tracing configuration to downstream Lambda/durable invokes. Keep existing ADOT/X-Ray deployments and assertions unchanged.
  • Have the real test caller create a span and, for sampled cases, export it to the same logical S3-backed OTLP capture as the called functions. Propagate a valid X-Amzn-Trace-Id derived from its trace ID, span ID, and sampling decision. Verify the actual invocation metadata sent by the selected Lambda client; generating an arbitrary parent ID is insufficient.
  • Retrieve the complete relevant traces from the capture, including caller, vendor, and application spans without durable.execution.arn and spans emitted by chained functions. Ensure all relevant collector outputs and caller exports are available before evaluating completeness.
  • For sampled S3 traces, require every non-root span's parent ID to resolve to an actual exported span in the same trace, with only the scenario's expected roots permitted. Assert the expected caller-to-function and durable-operation relationships, detect cycles and unexpected disconnected roots, and preserve the existing view-specific link checks. Trace-ID equality alone is not sufficient, and span links do not substitute for parent edges.
  • Remove or override unresolved-parent allowances for the S3 configuration. A missing parent must fail the S3 test with useful diagnostics. Do not satisfy the assertion by importing X-Ray-only ancestors, rewriting SDK identities, fabricating spans, or dropping offending spans from the queried set. If PassThrough still exposes unavailable native ancestors, report the failing configuration and track the required fix.
  • Preserve coverage of both views across the initial invocation, at least two resumes, and at least one chained durable invoke, including the existing long-running cases. Verify actual runtime context and stable relationships across those boundaries rather than assuming a topology from the tracing setting.
  • Cover Sampled=1, Sampled=0, and missing/invalid upstream context using the contract implemented by the tested SDK versions. Validate expected suppression for unsampled cases and the defined synthetic-root behavior for fallback cases; do not require spans that should be unsampled. Keep this separate from the proposed sampling-semantics changes in docs #295.
  • Run the updated S3 configuration for JavaScript, Python, and Java and report results alongside the unchanged X-Ray configuration. Record exact durable SDK/plugin, Lambda client, OTel provider, layer, runtime, and collector versions, configuration, tested triggers, and reproducible evidence. Identify failures or untested combinations explicitly.
  • Keep raw OTLP/S3 validation separate from vendor backend ingestion/query and UI validation. Exercise the Dash0 candidate manually with DASH0_XRAY_TRACES_ENABLED=false; a passing S3 hierarchy does not qualify Dash0 backend/UI behavior. Vendor credentials are not required for hosted S3 or X-Ray CI.
  • Supply results to docs #297 for the runnable procedure, prerequisites, tested versions, and tradeoffs. Do not present the setup as supported until the relevant validation passes. Document that PassThrough gives up native Lambda X-Ray platform tracing.

Complete ancestry is a requirement of the controlled, sampled S3 test configuration, where the test owns and exports its caller and application ancestors. It is not a universal requirement that SDKs synthesize or export externally owned spans. The existing external-parent contract and X-Ray coverage remain intact.

Alternatives considered

Keeping the existing S3 tests permissive and adding only an optional PassThrough scenario would leave the main S3 path unable to catch incomplete hierarchies. This issue instead upgrades the existing S3 path. Manual captures alone do not provide cross-SDK regression coverage, and collector results alone do not establish vendor ingestion or UI behavior.

Additional context

  • Docs #297 remains responsible for publishing the validated procedure, runnable user example, prerequisites, tradeoffs, tested-version scope, and package README links.
  • Conformance #99 describes the shared external-parent contract.
  • Hosted workflows currently run ADOT/X-Ray and the S3 collector; Dash0 is available for manual runs.
  • Existing JavaScript evidence uses SDK source 721217ae and Dash0 Node layer v26 with Active Tracing. It establishes the X-Ray/OTLP export boundary, not the proposed PassThrough configuration.

Activity

  1. changed the title [-][Feature]: Validate caller-to-OTLP propagation with Lambda PassThrough across durable resumes and chained invokes[/-] [+][Feature]: Switch existing S3 OTel tests to PassThrough and validate complete trace hierarchies[/+] on Nov 3, 2026
  2. SilanHe commented on Nov 16, 2026

    @SilanHe
    Contributor

    Triaged: valid. Priority: low. It depends on aws/aws-durable-execution-docs#297 and on changes in all three SDKs, so it needs a decision on owner and order first.

    AI-assisted comment.

  3. added
    needs-decisionStill needs a decision from maintainers. Not ready to work on yet.
    and removed
    needs-triageAwaiting operator triage
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestneeds-decisionStill needs a decision from maintainers. Not ready to work on yet.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions