Repository navigation
Conversation
nvasiu
force-pushed
the
feat/distributed-map-operation
branch
from
November 19, 2026 21:03
bb8529f to
12948d8
Compare
Add ctx.distributedMap, a durable operation that runs a processor Lambda over a large source of items, as a run managed by the service, and resumes the execution with the result when the run ends. - Sources: inline items, S3 objects and prefixes (JSON Lines, JSON array, CSV) and a reader function, built with DistributedMapSource - Processors: DistributedMapProcessor.reportBatchOutcome and reportItemOutcomes, with batching, retries and an optional durable processor - Destinations: optional S3 locations for successful and failed items, built with DistributedMapDestination - Run config: completion thresholds, timeout and optional collection of each item's result - Results: a summary with the run status, counts and error, or a result with each item's outcome when results are collected - Authoring wrappers: createDistributedMap* turn plain item, batch and page functions into processor and reader Lambdas, and pass them the run ARN and item ids
nvasiu
force-pushed
the
feat/distributed-map-operation
branch
from
November 19, 2026 21:59
12948d8 to
bd2a896
Compare
Contributor
Author
|
Update to improve the authoring helpers, following review feedback: Helpers instead of wrappers
Event types
Documentation
Tests
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Issue Link, if available
N/A
Description
Adds the distributed map operation (
ctx.distributedMap) to the JavaScript/TypeScript SDK:A map run processes a bounded dataset in parallel. A customer starts a map run from a durable function, naming a source to read items from, a processor function to invoke per batch, and concurrency, retry, and failure settings. The service reads items from the source, groups them into batches, invokes the processor for each batch, retries failures, tracks progress, routes successful results and failed items to destinations, and reports completion.
Changes
src/types/distributed-map.tsDistributedMapSummary: whatctx.distributedMapreturns, describes the run's overall outcome: status,sourceExhausted,error, the item counts (successCount,failureCount,readCount,totalCount) andthrowIfError().DistributedMapResult: returned fromctx.distributedMapwhen result collection is enabled. Contains individual map run item outcomes.DistributedMapResultItem: represents a single item's result or errorDistributedMapConfig: optional settings for distributed mapDistributedMapSource: factories for where map run items come from: an inline array (a plain array is also accepted directly), S3 objects and listed objects in JSON Lines, JSON array or CSV format, or a custom reader functionDistributedMapProcessor: factories for the Lambda that processes items,reportBatchOutcomeandreportItemOutcomes, named after what the processor reports backDistributedMapProcessorRetryConfig: configures how failing items are retried, passed as the processor'sretryoptionDistributedMapCompletionConfig: defines item failure thresholds for marking the overall map run failedSuccessDestination,FailureDestination,DistributedMapDestinationConfig,DistributedMapDestination: for routing successful and failed item records to S3DistributedMapItemContext,DistributedMapBatchContext,DistributedMapReaderContext: the run ARN and item ids passed to the customer's processor and reader functionssrc/types/durable-context.tsandsrc/context/durable-context/durable-context.tsctx.distributedMap: declares the entry point on theDurableContextinterface and wires it up in the context implementationsrc/distributed-map-helpers.tsdurableExecutionConfigthat is passed towithDurableExecution. Each wrapper passes the run ARN and item ids to the customer's function as a last argument, and rejects an event with a missing or malformed run ARN.src/handlers/distributed-map-handler/distributed-map-handler.tscreateDistributedMapHandler: the operation handler behindctx.distributedMap. It builds the wire request (source, processor, destination, completion config), checkpoints the START, and returns aDurablePromisethat suspends while the run executes. The promise resolves with the summary or result for every terminal status, includingFAILED,TIMED_OUTandSTOPPED. A configuration error terminates the execution withCONFIG_VALIDATION_ERROR, the same asmapandparallel. The handler throwsDistributedMapErroronly when the backend's terminal operation is missing required details.src/handlers/distributed-map-handler/distributed-map-result.tsDistributedMapSummaryImplandDistributedMapResultImpl: the summary and result classes, followingBatchResultImplsrc/handlers/distributed-map-handler/distributed-map-serdes-context.tssrc/types/wire/wire-model.tssrc/types/wire/wire-enums.tsandsrc/types/core.tsDISTRIBUTED_MAPoperation type and subtypesrc/errors/durable-error/durable-error.tsDistributedMapError: the error thrown bythrowIfError()when a run does not fully succeed, and when the backend sends invalid terminal detailsDistributedMapItemError: the error recorded for a single failed itemsrc/handlers/concurrent-execution-handler/batch-result.tsmapandparallelresultssrc/utils/operation/operation.tssrc/utils/serdes/serdes.tssrc/index.tsandsrc/types/index.tspackages/aws-durable-execution-sdk-js-testingpackages/aws-durable-execution-sdk-js-otel/src/external-completions.tsDISTRIBUTED_MAPto the external operation types, so it is skipped on replay likeWAIT,INVOKEandCALLBACKTests
src/types/distributed-map.test.tssrc/handlers/distributed-map-handler/distributed-map-handler.test.tssrc/handlers/distributed-map-handler/distributed-map-handler-two-phase.test.tssrc/handlers/distributed-map-handler/distributed-map-handler.composed.test.ts,.status.composed.test.tsand.invalid-config.composed.test.tssrc/handlers/distributed-map-handler/distributed-map-handler.plugin.test.tssrc/handlers/distributed-map-handler/distributed-map-result.test.tsanddistributed-map-serdes-context.test.tssrc/distributed-map-helpers.test.tsanddistributed-map-helpers.composed.test.tssrc/index.test.tsError, batch result, operation, testing library and OTel tests
TODO
wire-model.aws-sdk-parity.test.ts) and the testing package's type check stay red until@aws-sdk/client-lambdaships the DistributedMap shapes.AGENTS.md.Demo/Screenshots
N/A
Design
See Rules for every change.
Design note (rule 3), for a new feature, a new option or a public API change: Feature planned internally.
Simpler options considered (rule 2), if the change adds logic: N/A
Net change in source lines and in public API members:
+2800lines insrcand+4100in tests.ctx.distributedMap, theDistributedMapSource,DistributedMapProcessorandDistributedMapDestinationfactories, 5 authoring wrappers, the config, result and context types, and 2 error types.Assumptions
Each assumption about the service, Lambda, the runtime or another library, with its evidence, or marked "unverified" (rule 6):
SUCCEEDED,FAILED,STOPPEDorTIMED_OUT. Unverified.DistributedMapDetailscarriesSuccessCount,FailureCountandReadCount, carriesTotalCountonce the run succeeded, carriesSourceExhaustedonceDistributedMapRunArnis present, and reports a run failure asErrorthere. Unverified./distributed-map-run/, then a run id containing no further/. Unverified.ResultscarriesItemIdand aStatusofSUCCEEDEDorFAILED, with the item's value inOutputas a string the customer's serdes decodes. Unverified.recordslist, each record holding a non-emptyitemIdand abodythe customer's serdes decodes. Unverified.itemIdthe processor returns, and accepts the response shapes the wrappers return. Unverified.maxItemsPerPage, andstateas the reader last returned it. Unverified.MaxRetryAttemptsof-1meaning unlimited attempts. Unverified.Decisions not covered by the issue or the design note
One line each, with the reason (rule 7):
CONFIG_VALIDATION_ERRORinstead of throwing, to matchmapandparallel.invoke.DistributedMapErrorrather than substituting a value the service never sent.resultfield and failed items carryErrorinstances, to matchmapandparallel.maxRetryAttemptstakes"unlimited"rather than the wire sentinel, so a caller never writes a magic number.DistributedMapSource.s3, so each factory name states the source format.DistributedMapDestinationConfig, which holdsonSuccessandonFailure.maxConcurrencyis checked only as an integer greater than zero, so a service-side limit change needs no SDK release.toleratedFailurePercentageaccepts fractions, because the service model allows them.durableExecutionConfigthat is passed towithDurableExecution, so a durable processor can register plugins or supply its own client.Items,Outputand the record body are opaque strings, so any serdes works without an SDK change.mapandparallel.Checklist
Testing
Unit Tests
Have unit tests been written for these changes?
Yes (see above for tests added). All pass except the AWS SDK parity test, which fails until
@aws-sdk/client-lambdaadds the distributed map types.Integration Tests
Have integration tests been written for these changes?
TODO, the test runners don't support distributed map yet.
Examples
Has a new example been added for the change? (if applicable)
TODO
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.