Skip to content

feat(sdk): add distributed map operation config/types - #889

Draft
nvasiu wants to merge 1 commit into
mainfrom
feat/distributed-map-operation
Draft

nvasiu wants to merge 1 commit into
mainfrom
feat/distributed-map-operation

Conversation

@nvasiu

@nvasiu nvasiu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

N/A

Description

This is the first of two PRs. Nothing in this PR uses the new Lambda model, so it can merge before the model is released. The second PR, #901, adds ctx.distributedMap and the operation that starts a run and reads its outcome, which needs the new model.

Adds the types the distributed map operation is built from to the Java SDK: the config types a customer constructs to describe a map run, the result types it returns, and the helpers a processor or reader Lambda calls in its handler.

A map run processes a bounded dataset in parallel. A customer starts a map run from a durable function, naming a source to read items from, a processor function to invoke per batch, and concurrency, retry, and failure settings. The service reads items from the source, groups them into batches, invokes the processor for each batch, retries failures, tracks progress, routes successful results and failed items to destinations, and reports completion.

config/

The input types a customer constructs to describe a map run.

  • DistributedMapConfig: optional settings for distributed map. Its CompletionConfig defines item failure thresholds for marking the overall map run failed.
  • DistributedMapSource: factories naming where map run items come from. inline, s3JsonLines, s3JsonArray, s3Csv, s3Objects, s3FlattenedJsonLines, s3FlattenedJsonArray, s3FlattenedCsv and reader.
  • DistributedMapProcessor: describes the Lambda that processes items. The reportBatchOutcome and reportItemOutcomes factories pick how the processor reports outcomes back. The retry option takes a DistributedMapProcessorRetryConfig holding maxRetryAttempts and maxRetryDuration, with DistributedMapProcessorRetryConfig.UNLIMITED for an unbounded attempt count.
  • DistributedMapDestination: the s3Successes and s3Failures factories for routing item records to S3.
  • Each config validates its own arguments on construction: a destination prefix of at most 900 UTF-8 bytes, a batch size between 1 and 10000, a timeout of at most 90 days, and retry attempts that are non-negative or UNLIMITED.

model/

The result types a customer receives back from a map run.

  • DistributedMapSummary: describes the run's overall outcome.
  • DistributedMapResult: contains individual map run item outcomes.
  • DistributedMapError: the error on a run or on one item.
  • DistributedMapStatus: the run's terminal status.

dmap/DistributedMapHelpers.java

Authoring helpers for the processor and reader Lambdas, which a customer calls inside their own handler, so they write a plain function rather than the item or batch protocol.

  • processDistributedMapItems, processDistributedMapBatch and readDistributedMapPage.
  • The item and batch helpers work in both non-durable and durable handlers. In a durable handler, the item helper runs items with a FLAT ctx.map by default, with a nesting option.
  • The customer's function receives the run ARN, and the item ids where they apply, through ItemContext, BatchContext and ReaderContext, passed as the last argument.
  • The helpers validate the invocation input and reject a record that carries no item id, since every outcome is reported against an item id.
  • processDistributedMapItems writes one ERROR log line for each failed item, with distributedMapRunArn, itemId and the error it reports.

dmap/

The records the helpers read and return.

  • ProcessorEvent, ProcessorRecord and ReaderEvent: the events the service sends to a processor or reader Lambda.
  • ItemHandlerResponse, ItemResult, ItemFailure and ReaderResponse: the responses the helpers build, rather than hand-built maps.
  • ItemContext, BatchContext and ReaderContext: the run ARN and item ids passed to the customer's function.

execution/DurableExecutor.java

  • A processor's invocation input, result and error are read and written with the default JSON serdes, because the service defines their format. Durable operations inside the processor still use the config's serdes.

exception/DistributedMapException.java

The error type a customer catches.

  • DistributedMapException: thrown by throwIfError() when a run or an item fails.

Future tasks

  • Add ctx.distributedMap and the distributed map operation, in #901 .
  • Add distributed map to LocalDurableTestRunner (in the testing package). When this is done, we can add full end to end tests using the runner.
  • Add distributed map examples and a docs page.

Demo/Screenshots

N/A

Design

See Rules for every change.

  • Design note (rule 3), for a new feature, a new option or a public API change: Feature planned internally.
  • Simpler options considered (rule 2), if the change adds logic: N/A
  • Root cause (rule 5), for a bug fix: N/A
  • Net change in source lines and in public API members:
    • +1899 lines in src/main and +2521 in src/test.
    • New public members: 5 config classes in config/ with their nested types and factories, 4 result types in model/, the 3 helpers in DistributedMapHelpers with their 2 functional interfaces, 11 records in dmap/, and 1 exception type.

Assumptions

Each assumption about the service, Lambda, the runtime or another library, with its evidence, or marked "unverified" (rule 6):

  • A processor invocation payload carries distributedMapRunArn and a records list, each record holding a non-empty string itemId and a body the customer's serdes decodes. Unverified.
  • The service matches a reported outcome back to its item by the itemId the processor echoes, and accepts the batch-failure and item-outcome response shapes the helpers return. Unverified.
  • A reader invocation payload carries distributedMapRunArn, an integer maxItemsPerPage, and state as the reader last returned it. Unverified.
  • The service's limits are the ones the configs validate against: a destination key prefix of 900 UTF-8 bytes, a batch size of 10000 items, and a map run timeout of 90 days. The ceiling on concurrent processor invocations is the service's alone, since the SDK enforces none. Unverified.

Decisions not covered by the issue or the design note

One line each, with the reason (rule 7):

  • An invalid configuration throws IllegalArgumentException, because the other operations' configuration checks throw it.
  • DistributedMapException stays outside the DurableOperationException hierarchy, because it reports a run's aggregate result rather than one operation's failure.
  • DurableExecutor picks the default JSON serdes when the input type is ProcessorEvent, so a customer serdes cannot change the format the service defines, with no new public method or DurableConfig option.
  • The source factories are flat on DistributedMapSource, named s3JsonLines and so on, so the factory name states the source kind in one call.
  • The destination prefix check is 900 UTF-8 bytes, which is the lower of the service's character and byte limits, so a multi-byte prefix is rejected at the real limit.
  • No upper bound on concurrency is enforced in the SDK, so a service-side limit change needs no SDK release.
  • A processor record with no item id fails the invocation before any item handler runs, rather than failing that record alone, because an outcome cannot be reported for an item that was not named.
  • The context records are passed as the last argument, because Java's other operations pass their context last. The durable item helper keeps DurableContext first.
  • The durable item helper runs items with FLAT nesting by default, because it uses fewer operations. ctx.map's own default stays NESTED.
  • A null reader item is sent as "null", and a non-null value that serializes to null is rejected, because the service requires every item to be a string.
  • The record body is an opaque string, so a customer serdes other than JSON works without an SDK change.

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change
  • Every new branch has a test that fails without it (rule 4)
  • For a bug fix: the description or commit order shows the regression test failing before the fix and passing after it (rule 5)
  • Every edited test expectation is listed in the description, with the reason (rule 5)

Testing

Unit Tests

Have unit tests been written for these changes?

Yes. Tests are split by layer so each sits alongside its class.

  • config/DistributedMap*Test: config and argument validation.
  • model/DistributedMapSummaryTest and model/DistributedMapResultTest: the result types and throwIfError().
  • dmap/DistributedMapHelpersTest and dmap/DistributedMapDurableHelpersTest: helper tests, checking that they process items, report failures, pass the run ARN and item ids, reject bad inputs, run in FLAT and nested modes, and write the error log line.
  • sdk/pom.xml adds the test-only aws-lambda-java-serialization dependency, so a test checks that the Lambda runtime's own serializer reads the events. No existing test expectation was edited.

Integration Tests

Have integration tests been written for these changes?

No. LocalDurableTestRunner does not support distributed map yet (see Future tasks).

Examples

Has a new example been added for the change? (if applicable)

No. Examples will follow with runner support (see Future tasks).

@zhongkechen

Copy link
Copy Markdown
Contributor

@nvasiu could you split this PR into smaller ones so that the changes don't rely on new models can be merged early?

@nvasiu
nvasiu force-pushed the feat/distributed-map-operation branch from 264fc4e to b15a796 Compare November 20, 2026 18:27
@nvasiu

nvasiu commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Update to improve the authoring helpers, following review feedback:

Helpers instead of wrappers

  • Replaced the createDistributedMap* wrappers in DistributedMapHandlers with processDistributedMapItems, processDistributedMapBatch and readDistributedMapPage in DistributedMapHelpers, which customers call inside their own handler, so they keep control of the handler.
  • Durable processors extend DurableHandler directly, so their config is passed there like any durable function.
  • The item and batch helpers work in both non-durable and durable handlers. In a durable handler, the item helper runs items with a FLAT ctx.map by default, with a nesting option.
  • The helpers take the existing public ProcessorEvent and ReaderEvent types, so customers type their handler's event with them.

Per-item error log

  • processDistributedMapItems now writes one ERROR log line for each failed item, with distributedMapRunArn, itemId and the error it returns in batchItemFailures. This means a handler that only throws still leaves a log line with the item's ID.
  • A non-durable handler logs with an SLF4J logger, and a durable handler logs with context.getLogger(). The fields are added to the line through MDC.

Documentation

  • Each helper's javadoc says where to call it, which factory takes the Lambda's name, and that the handler must return the helper's result.

Tests

  • Updated the tests to the helpers, and added tests for both item forms, FLAT and nested modes, a step before the items, the batch helper in a durable handler, and the error log line.

Also rebased onto the latest main.

Add the types a distributed map run is built from: the source,
processor, completion and destination config, the summary and result
types, and the helpers a processor or reader Lambda calls in its handler.
@nvasiu
nvasiu force-pushed the feat/distributed-map-operation branch from b15a796 to 8ecc0f1 Compare November 20, 2026 21:15
@nvasiu

nvasiu commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

@zhongkechen Split up this PR into 2. Now this PR only contains all the changes that can merge without the new model. I made a second PR #901 containing the rest of the Dmap operation changes that depend on the model.

@nvasiu nvasiu changed the title feat(sdk): add distributed map operation feat(sdk): add distributed map operation config/types Nov 20, 2026

// Lambda response size limit is 6MB minus small epsilon for envelope
private static final int LAMBDA_RESPONSE_SIZE_LIMIT = 6 * 1024 * 1024 - 50;
private static final SerDes JSON = new JacksonSerDes();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need NON_NULL here

var records = event.records();
var bodies = new ArrayList<String>(records.size());
for (var record : records) {
bodies.add(record.body());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

handle null body properly

* @param distributedMapRunArn the ARN of the distributed map run this batch belongs to
* @param records the items in this batch
*/
public record ProcessorEvent(String distributedMapRunArn, List<ProcessorRecord> records) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

handle null record in records


/**
* Thrown by throwIfError when a distributed map run did not fully succeed, or when an item failed on a run that
* otherwise succeeded. It reports an aggregate run result and is not a single operation failure, so it stays out of the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't a single run always mapping to a single dmap operation?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants