Skip to content

feat(sdk): add distributed map operation to context - #901

Draft
nvasiu wants to merge 2 commits into
mainfrom
feat/ctx-distributed-map
Draft

nvasiu wants to merge 2 commits into
mainfrom
feat/ctx-distributed-map

Conversation

@nvasiu

@nvasiu nvasiu commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

N/A

Description

This is the second of two PRs, and builds on the config, result and helper types in #889. It uses the new distributed map shapes in the Lambda model, so it can only merge once that model is released.

Adds the distributed map operation (ctx.distributedMap) to the Java SDK:
A map run processes a bounded dataset in parallel. A customer starts a map run from a durable function, naming a source to read items from, a processor function to invoke per batch, and concurrency, retry, and failure settings. The service reads items from the source, groups them into batches, invokes the processor for each batch, retries failures, tracks progress, routes successful results and failed items to destinations, and reports completion.

DurableContext.java, context/DurableContextImpl.java

The customer-facing entry point on the durable context.

  • ctx.distributedMap: the method a customer calls to run a distributed map. Overloads that take a result type return DistributedMapResult, and the others return DistributedMapSummary.

operation/DistributedMapOperation.java

The operation that drives the run against the durable execution runtime.

  • Suspends the caller's function while the run executes and resumes it with the finished outcome.
  • Every terminal state resolves, and throwIfError() opts into throwing.
  • All translation between the config types and the service shapes happens here.
  • The item counts and each collected result's item id are required. A terminal operation that omits a count, or a result that omits its item id, throws an error naming the missing field rather than substituting a value the service never sent.

execution/CheckpointManager.java

  • The checkpoint size estimate counts the distributed map options, so checkpoint batches stay within the limit.

model/OperationSubType.java, operation/BaseDurableOperation.java, operation/ChildContextOperation.java

Operation type registration.

  • Adds DISTRIBUTED_MAP to OperationSubType, and passes a failed run's error to plugins.

Future tasks

  • Add distributed map to LocalDurableTestRunner (in the testing package). When this is done, we can add full end to end tests using the runner.
  • Add distributed map examples and a docs page.

Demo/Screenshots

N/A

Design

See Rules for every change.

  • Design note (rule 3), for a new feature, a new option or a public API change: Feature planned internally.
  • Simpler options considered (rule 2), if the change adds logic: N/A
  • Root cause (rule 5), for a bug fix: N/A
  • Net change in source lines and in public API members:
    • +780 lines in src/main and +1539 in src/test.
    • New public members: the ctx.distributedMap overloads and the DISTRIBUTED_MAP value of OperationSubType.

Assumptions

Each assumption about the service, Lambda, the runtime or another library, with its evidence, or marked "unverified" (rule 6):

  • A distributed map operation ends in one of SUCCEEDED, FAILED, STOPPED or TIMED_OUT. Unverified.
  • A terminal operation's DistributedMapDetails carries SuccessCount, FailureCount and ReadCount, carries TotalCount once the run succeeded, and reports a run failure as Error there rather than as an operation-level error. Unverified.
  • Every collected result in Results carries ItemId and a Status of SUCCEEDED or FAILED, with the item's value in Output as a string the customer's serdes decodes. Unverified.
  • The service accepts the source, processor, completion and destination configuration the checkpoint sends, with MaxRetryAttempts of -1 meaning unlimited attempts and MaxRetryDurationSeconds a whole number of seconds. Unverified.

Decisions not covered by the issue or the design note

One line each, with the reason (rule 7):

  • A missing count or item id throws, because a value the service never sent cannot be substituted safely.
  • A null inline item is sent as "null", and a non-null value that serializes to null is rejected, because the service requires every item to be a string.
  • Items and Output are opaque strings, so a customer serdes other than JSON works without an SDK change.

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change
  • Every new branch has a test that fails without it (rule 4)
  • For a bug fix: the description or commit order shows the regression test failing before the fix and passing after it (rule 5)
  • Every edited test expectation is listed in the description, with the reason (rule 5)

Testing

Unit Tests

Have unit tests been written for these changes?

Yes. Tests are split by layer so each sits alongside its class.

  • operation/DistributedMapOperationTest and operation/DistributedMapOperationTranslationTest: operation behaviour and the translation to and from the service shapes.
  • context/DurableContextDistributedMapTest: the ctx.distributedMap surface.
  • execution/CheckpointManagerTest: new tests for the size estimate. No existing test expectation was edited.

Integration Tests

Have integration tests been written for these changes?

No. LocalDurableTestRunner does not support distributed map yet (see Future tasks).

Examples

Has a new example been added for the change? (if applicable)

No. Examples will follow with runner support (see Future tasks).

Add the types a distributed map run is built from: the source,
processor, completion and destination config, the summary and result
types, and the helpers a processor or reader Lambda calls in its handler.
Add ctx.distributedMap, which starts a distributed map run, waits for it
to finish and returns its summary or per-item result. Builds on the
distributed map config, result and helper types.
@nvasiu nvasiu changed the title Feat/ctx distributed map feat(sdk): add distributed map operation to context Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant