Skip to content

feat(sdk): add distributed map operation - #1227

Draft
nvasiu wants to merge 1 commit into
mainfrom
feat/distributed-map-operation
Draft

nvasiu wants to merge 1 commit into
mainfrom
feat/distributed-map-operation

Conversation

@nvasiu

@nvasiu nvasiu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

N/A

Description

Adds the distributed map operation (ctx.distributedMap) to the JavaScript/TypeScript SDK:
A map run processes a bounded dataset in parallel. A customer starts a map run from a durable function, naming a source to read items from, a processor function to invoke per batch, and concurrency, retry, and failure settings. The service reads items from the source, groups them into batches, invokes the processor for each batch, retries failures, tracks progress, routes successful results and failed items to destinations, and reports completion.

Changes

src/types/distributed-map.ts

  • DistributedMapSummary: what ctx.distributedMap returns, describes the run's overall outcome: status, sourceExhausted, error, the item counts (successCount, failureCount, readCount, totalCount) and throwIfError().
  • DistributedMapResult: returned from ctx.distributedMap when result collection is enabled. Contains individual map run item outcomes.
  • DistributedMapResultItem: represents a single item's result or error
  • DistributedMapConfig: optional settings for distributed map
  • DistributedMapSource: factories for where map run items come from: an inline array (a plain array is also accepted directly), S3 objects and listed objects in JSON Lines, JSON array or CSV format, or a custom reader function
  • DistributedMapProcessor: factories for the Lambda that processes items, reportBatchOutcome and reportItemOutcomes, named after what the processor reports back
  • DistributedMapProcessorRetryConfig: configures how failing items are retried, passed as the processor's retry option
  • DistributedMapCompletionConfig: defines item failure thresholds for marking the overall map run failed
  • SuccessDestination, FailureDestination, DistributedMapDestinationConfig, DistributedMapDestination: for routing successful and failed item records to S3
  • DistributedMapItemContext, DistributedMapBatchContext, DistributedMapReaderContext: the run ARN and item ids passed to the customer's processor and reader functions

src/types/durable-context.ts and src/context/durable-context/durable-context.ts

  • ctx.distributedMap: declares the entry point on the DurableContext interface and wires it up in the context implementation

src/distributed-map-helpers.ts

  • Authoring wrappers: let a customer write a plain function and have it work as a processor Lambda without hand-writing the item or batch protocol, including durable-execution variants and a reader. The durable variants accept a durableExecutionConfig that is passed to withDurableExecution. Each wrapper passes the run ARN and item ids to the customer's function as a last argument, and rejects an event with a missing or malformed run ARN.

src/handlers/distributed-map-handler/distributed-map-handler.ts

  • createDistributedMapHandler: the operation handler behind ctx.distributedMap. It builds the wire request (source, processor, destination, completion config), checkpoints the START, and returns a DurablePromise that suspends while the run executes. The promise resolves with the summary or result for every terminal status, including FAILED, TIMED_OUT and STOPPED. A configuration error terminates the execution with CONFIG_VALIDATION_ERROR, the same as map and parallel. The handler throws DistributedMapError only when the backend's terminal operation is missing required details.

src/handlers/distributed-map-handler/distributed-map-result.ts

  • DistributedMapSummaryImpl and DistributedMapResultImpl: the summary and result classes, following BatchResultImpl

src/handlers/distributed-map-handler/distributed-map-serdes-context.ts

  • Parses the run ARN, and gives each serialized value its own serdes context, so a custom serdes that stores values externally keeps them apart

src/types/wire/wire-model.ts

  • The wire shapes sent to and returned by the backend for the operation

src/types/wire/wire-enums.ts and src/types/core.ts

  • Add the DISTRIBUTED_MAP operation type and subtype

src/errors/durable-error/durable-error.ts

  • DistributedMapError: the error thrown by throwIfError() when a run does not fully succeed, and when the backend sends invalid terminal details
  • DistributedMapItemError: the error recorded for a single failed item

src/handlers/concurrent-execution-handler/batch-result.ts

  • Add the distributed map errors to the known error types, so they keep their type inside map and parallel results

src/utils/operation/operation.ts

  • Plugin hooks read a failed distributed map run's error, which the backend records under the distributed map details

src/utils/serdes/serdes.ts

  • Document that a serdes payload must deserialize without its original context, because the function that reads a value can differ from the one that wrote it

src/index.ts and src/types/index.ts

  • Export the distributed-map types as public API

packages/aws-durable-execution-sdk-js-testing

  • The local and cloud test runners throw a "not supported" error when they reach a distributed map operation. The testing README notes this limitation.

packages/aws-durable-execution-sdk-js-otel/src/external-completions.ts

  • Add DISTRIBUTED_MAP to the external operation types, so it is skipped on replay like WAIT, INVOKE and CALLBACK

Tests

src/types/distributed-map.test.ts

  • Factory, type, and validation unit tests: source, processor, and destination factories, completion and processor config validation

src/handlers/distributed-map-handler/distributed-map-handler.test.ts

  • Core handler unit tests: wire building, summary building, required backend fields, collected-result access

src/handlers/distributed-map-handler/distributed-map-handler-two-phase.test.ts

  • Two-phase suspend / resume tests, mocking backend responses, including configuration errors terminating the execution

src/handlers/distributed-map-handler/distributed-map-handler.composed.test.ts, .status.composed.test.ts and .invalid-config.composed.test.ts

  • Tests for distributed map running inside child contexts and alongside other operations, waiting through status changes, and invalid configurations ending the execution

src/handlers/distributed-map-handler/distributed-map-handler.plugin.test.ts

  • Plugin hook tests: onOperationStart, onOperationChange, onInvocationStart and onOperationEnd, including error attachment on failure

src/handlers/distributed-map-handler/distributed-map-result.test.ts and distributed-map-serdes-context.test.ts

  • Result class tests, run ARN parsing and per-value serdes contexts

src/distributed-map-helpers.test.ts and distributed-map-helpers.composed.test.ts

  • Authoring wrapper tests: checking that they process items, report failures, reject bad inputs, pass the run ARN and item ids, and pass the durable execution config through

src/index.test.ts

  • Updates the public export contract to include the distributed map surface

Error, batch result, operation, testing library and OTel tests

  • Distributed map error round trips, error types in batch results, hook errors, the test runners' unsupported-operation errors, and the OTel external operation list

TODO

  • When the model changes and distributed map implementation are complete in the durable service backend, these SDK changes need to be verified against them.
  • The AWS SDK parity test (wire-model.aws-sdk-parity.test.ts) and the testing package's type check stay red until @aws-sdk/client-lambda ships the DistributedMap shapes.
  • Add distributed map to the local emulator.
  • Add distributed map examples to the examples package.
  • Add a distributed map section to AGENTS.md.

Demo/Screenshots

N/A

Design

See Rules for every change.

  • Design note (rule 3), for a new feature, a new option or a public API change: Feature planned internally.

  • Simpler options considered (rule 2), if the change adds logic: N/A

  • Net change in source lines and in public API members:

    • About +2800 lines in src and +4100 in tests.
    • New public members: ctx.distributedMap, the DistributedMapSource, DistributedMapProcessor and DistributedMapDestination factories, 5 authoring wrappers, the config, result and context types, and 2 error types.

Assumptions

Each assumption about the service, Lambda, the runtime or another library, with its evidence, or marked "unverified" (rule 6):

  • A distributed map operation ends in one of SUCCEEDED, FAILED, STOPPED or TIMED_OUT. Unverified.
  • A terminal operation's DistributedMapDetails carries SuccessCount, FailureCount and ReadCount, carries TotalCount once the run succeeded, carries SourceExhausted once DistributedMapRunArn is present, and reports a run failure as Error there. Unverified.
  • A run ARN is the parent durable execution ARN, then /distributed-map-run/, then a run id containing no further /. Unverified.
  • Every collected result in Results carries ItemId and a Status of SUCCEEDED or FAILED, with the item's value in Output as a string the customer's serdes decodes. Unverified.
  • A processor invocation carries the run ARN and a records list, each record holding a non-empty itemId and a body the customer's serdes decodes. Unverified.
  • The service matches a reported outcome to its item by the itemId the processor returns, and accepts the response shapes the wrappers return. Unverified.
  • A reader invocation carries the run ARN, an integer maxItemsPerPage, and state as the reader last returned it. Unverified.
  • The service accepts the source, processor, completion and destination configuration the checkpoint sends, with MaxRetryAttempts of -1 meaning unlimited attempts. Unverified.
  • The service's limits are the ones the SDK validates against: a destination key prefix of 900 UTF-8 bytes, a batch size of 10000 items, a 1 MB inline source, a 32 KB reader state, and a map run timeout of 90 days. Unverified.

Decisions not covered by the issue or the design note

One line each, with the reason (rule 7):

  • An invalid config ends the execution with CONFIG_VALIDATION_ERROR instead of throwing, to match map and parallel.
  • Config is validated only on start, not on replay, to match invoke.
  • A missing count or item id throws DistributedMapError rather than substituting a value the service never sent.
  • Item results use a result field and failed items carry Error instances, to match map and parallel.
  • maxRetryAttempts takes "unlimited" rather than the wire sentinel, so a caller never writes a magic number.
  • The S3 source factories are grouped under DistributedMapSource.s3, so each factory name states the source format.
  • A destination factory returns one half of DistributedMapDestinationConfig, which holds onSuccess and onFailure.
  • The destination prefix check is 900 UTF-8 bytes, so a multi-byte prefix is rejected at the service's real limit.
  • maxConcurrency is checked only as an integer greater than zero, so a service-side limit change needs no SDK release.
  • toleratedFailurePercentage accepts fractions, because the service model allows them.
  • A processor record with no item id fails the invocation before any item runs, because an outcome can't be reported for an item that wasn't named.
  • The durable wrappers take a durableExecutionConfig that is passed to withDurableExecution, so a durable processor can register plugins or supply its own client.
  • Items, Output and the record body are opaque strings, so any serdes works without an SDK change.
  • The processor and reader contexts have no logger, to avoid an SDK-wide logger change.
  • Distributed map is exported from the package root, like map and parallel.

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change
  • Every new branch has a test that fails without it (rule 4)
  • For a bug fix: the first commit adds a test that fails on the current code (rule 5)
  • Every edited test expectation is listed in the description, with the reason (rule 5)

Testing

Unit Tests

Have unit tests been written for these changes?

Yes (see above for tests added). All pass except the AWS SDK parity test, which fails until @aws-sdk/client-lambda adds the distributed map types.

Integration Tests

Have integration tests been written for these changes?

TODO, the test runners don't support distributed map yet.

Examples

Has a new example been added for the change? (if applicable)

TODO

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@nvasiu
nvasiu force-pushed the feat/distributed-map-operation branch from bb8529f to 12948d8 Compare November 19, 2026 21:03
Add ctx.distributedMap, a durable operation that runs a processor Lambda
over a large source of items, as a run managed by the service, and
resumes the execution with the result when the run ends.

- Sources: inline items, S3 objects and prefixes (JSON Lines, JSON
  array, CSV) and a reader function, built with DistributedMapSource
- Processors: DistributedMapProcessor.reportBatchOutcome and
  reportItemOutcomes, with batching, retries and an optional durable
  processor
- Destinations: optional S3 locations for successful and failed items,
  built with DistributedMapDestination
- Run config: completion thresholds, timeout and optional collection of
  each item's result
- Results: a summary with the run status, counts and error, or a result
  with each item's outcome when results are collected
- Authoring wrappers: createDistributedMap* turn plain item, batch and
  page functions into processor and reader Lambdas, and pass them the
  run ARN and item ids
@nvasiu
nvasiu force-pushed the feat/distributed-map-operation branch from 12948d8 to bd2a896 Compare November 19, 2026 21:59
@nvasiu

nvasiu commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Update to improve the authoring helpers, following review feedback:

Helpers instead of wrappers

  • Replaced the five createDistributedMap* wrappers with processDistributedMapItems, processDistributedMapBatch and readDistributedMapPage, which customers call inside their own handler, so they keep control of the handler.
  • Durable processors use withDurableExecution directly, so their config is passed there like any durable function.
  • The item and batch helpers work in both non-durable and durable handlers. In a durable handler, the item helper runs items with a FLAT ctx.map by default, with a nesting option.

Event types

  • Exported DistributedMapProcessorEvent, DistributedMapProcessorRecord and DistributedMapReaderEvent, and typed the helpers' event with them, so customers don't have to pass unknown.

Documentation

  • Each helper's docs say where to call it, which factory takes the Lambda's name, and that the handler must return the helper's result.

Tests

  • Updated the tests to the helpers, and added tests for both item forms, FLAT and nested modes, and the helper and event types.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant