Skip to content

feat(step): pause the execution from a retry decision - #190

Merged
ParidelPooya merged 3 commits into
mainfrom
feat/step-pause
Nov 19, 2026
Merged

ParidelPooya merged 3 commits into
mainfrom
feat/step-pause

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Description of changes

A retry decision can now pause the durable execution. An operator can then check the cause of the failures and resume the execution with the Lambda ResumeDurableExecution API.

Design note

The design was agreed in the JavaScript SDK (aws-durable-execution-sdk-js#1253) and the Python SDK (aws-durable-execution-sdk-python#961). This PR ports it to Go.

Problem. A step that keeps failing retries until its retry strategy stops. An operator has no way to stop it after a few failures, check the cause and then let it continue.

Cause. The SDK has no link between step retries and the Lambda pause API. Lambda added PauseDurableExecution and ResumeDurableExecution on 2026-11-17.

Behavior.

  1. RetryDecision gets a Pause bool field. Its zero value is false, so existing strategies and decision literals work as before.
  2. RetryConfig gets PauseEvery int. The strategy that NewRetryStrategy builds sets Pause on a retry after attempt n when n % PauseEvery == 0. So PauseEvery: 5 pauses after attempts 5, 10, 15 and so on. The delay and the jitter do not change. LinearRetryConfig gets no option.
  3. A step pauses only when its function failed in this invocation and the decision sets Retry and Pause. The step then:
    1. sends the normal RETRY checkpoint and waits until Lambda accepts it;
    2. calls PauseDurableExecution with the execution ARN, a ClientToken and the Reason Step "<name or wire id>" failed attempt <n>;
    3. halts the checkpointer with the suspension cause, so the invocation ends with PENDING. It does not wait for the retry timer.
      After the resume, the next attempt runs normally.
  4. The interrupted AtMostOncePerRetry replay path also calls the strategy. It ignores Pause.
  5. Pause never changes the attempt count. With Retry false, the SDK ignores Pause and the step fails as usual.
  6. When the pause call fails, the step logs a warning through the operation logger and retries as usual. The warning names the error code (or the Go type) and the message. For ResourceConflictException it says "The execution has closed (for example, stopped) or is already paused." Every other error says a missing lambda:PauseDurableExecution permission is one possible cause.
  7. ExecutionPauser is an optional interface. The step checks for it at run time, so a custom ExecutionClient keeps compiling and retries as usual. The default Lambda client implements it.
  8. The SDK does not validate PauseEvery. The doc comment states the expected values.

Why the retry is saved before the pause. Once a pause takes effect, Lambda saves one more checkpoint and then accepts no more. So the step saves RETRY first and asks to pause after that. The checkpointer sends calls in queue order, so an earlier START queued without waiting is also saved before RETRY returns. If Lambda answers RETRY without a token, the existing tokenless-response path halts the checkpointer, and the step requests no pause.

Abandoned steps. A step in a Parallel or Map branch that the batch abandoned after early completion still records its RETRY in Go. It does not pause: the step checks the abandon handle before the pause call. Its wait for the retry then returns at once, so it does not hang.

Other branches. Steps already running in other branches finish their current attempt. The SDK sends no checkpoint after the pause request, so an attempt that finishes after it runs again after the resume.

Decisions not covered by the agreed design

  • Pause is a plain bool field on the struct. The [0]func() guard already blocks unkeyed literals, so adding the field breaks no caller.
  • The pause decision is passed into settleStepFailure as mayPause. Only the body-failure path passes true.
  • The step halts the checkpointer after a successful pause call, with the same cause as a tokenless response. So user code that ignores the step's error cannot record more state, and the invocation ends PENDING.
  • ExecutionPauser.PauseExecution takes PauseExecutionInput{ExecutionArn, ClientToken, Reason} and returns an empty PauseExecutionOutput, matching the style of the other client methods. ExecutionTimeoutPaused is not exposed; the API default applies.
  • ClientToken: the SHA-256 of pause:<execution ARN>:<wire step id>:<attempt>, as 64 lowercase hex characters, as in the JavaScript SDK.
  • The Reason uses the step name, or the wire (hashed) id when the step has no name. It is cut to 1024 characters, as in the Python SDK.
  • The warning uses the smithy APIError code when there is one, and the Go type name otherwise.
  • durable.Retry ignores Pause, because it does not run as a step. The WaitForCallback submitter honors it, because it runs as a step. Both are documented.
  • PauseEvery: 0 never pauses. A negative value pauses on the multiples of its absolute value. Not validated, as agreed.
  • The service/lambda requirement goes from v1.99.0 to v1.117.0, the first version with PauseDurableExecution. go mod tidy also moves aws-sdk-go-v2 to v1.48.0 and smithy-go to v1.28.9 in the root, telemetry, conformance and examples modules. The release notes name all three bumps.
  • durabletest: the in-memory client implements ExecutionPauser. It marks the execution paused without waiting for the invocation to end, answers the next checkpoint without a token, and returns ResourceConflictException for a pause with another token, a pause of an ended execution, or a resume of an execution that is not paused. LocalRunner gains Paused(), ResumeExecution() and ErrExecutionPaused, which Run and RunUntilComplete return while the execution is paused. ResumeExecution resumes only the runner's own execution. A registered durable target that pauses stays PENDING; the cloud follow-up covers resuming it.
  • The local runner records no ExecutionPaused or ExecutionResumed history events. The history-to-operation fold ignores unknown event types, so the new client version needs no mapping.
  • New example step-pause-every, local only. It is not in build.sh or template.yaml, so the cloud tests skip it. It is in parity-map.txt against the JavaScript step/pause-every example.

Assumptions

  • Confirmed by a live test against Lambda: after the PauseDurableExecution call succeeds, Lambda holds the next invocation until the execution is resumed. So the pausing attempt needs no special delay or jitter.
  • Confirmed by the docs: the pause is asynchronous, and the status does not change during the call. See Pausing and resuming durable executions.
  • Unverified: a function's execution role can pause its own running execution. The docs say the action applies to the durable execution ARN.
  • Unverified: when the pause is requested after RETRY is saved, the invocation returns PENDING, and Lambda then moves the execution to PAUSED.
  • Unverified: durabletest.CloudRunner keeps polling a PAUSED execution until its own timeout, because PAUSED is not a terminal status. The cloud follow-up will handle it.

Tests

  • go build ./..., go vet ./..., golangci-lint run ./... and golangci-lint fmt --diff ./... pass in every module. The release tool's check and consumer commands pass.
  • Root module, go test -race ./...: 2377 tests and subtests pass, 1 skipped. TestDefaultClientRequestTimeouts was skipped locally. It fails on main too in my environment, because the shell's AWS profile has no shared config.
  • telemetry, conformance, examples and analysis: scripts/ci-local.sh <module> passes.
  • New tests in durable/pause_test.go: pause on Pause: true (RETRY then pause, no later checkpoint), no pause on false or absent Pause (attempt 2 runs in the same invocation), no pause when not retrying, the interrupted path ignores Pause, a tokenless RETRY answer requests no pause, an abandoned Parallel branch does not pause and does not hang (it passes 300 of 300 runs with go test -race -count=300), pause failures log the warning and retry as usual (four error kinds: a service error, a plain error, an API error without a code that reports its Go type, and ResourceConflictException with its text), a client without ExecutionPauser retries as usual, ClientToken stable and different per ARN, step and attempt (64 hex), Reason for an unnamed step and truncation, PauseEvery on multiples only (4 no, 5 yes, 6 no, 10 yes, 15 yes), the full jitter kept on the pausing attempt, and no pause from ExponentialBackoff or a config without the option.
  • durable/lambda_client_adapter_test.go: the adapter maps the fields, leaves ExecutionTimeoutPaused unset, and returns the service error.
  • durabletest: pause after attempts 2 and 4 with PauseEvery: 2, Run refused while paused, resume, success on attempt 5; RunUntilComplete returns ErrExecutionPaused while paused; Reset clears the pause; a registered durable target that pauses stays PENDING and is not invoked again; the WaitForCallback submitter pauses; resume of an execution that is not paused conflicts; the in-memory client's idempotent token, a repeated pause without a token conflicts, the other conflicts and the tokenless answer.
  • Mutation check: 25 mutations of the new branches. Each one fails at least one test. They include: drop the halt, ignore Pause, ignore mayPause, drop the abandon check, treat a failed pause as success, drop the warning, drop the conflict hint, drop the empty-code check on the error name, pause on every retry, pause before the RETRY checkpoint, drop truncation, drop the ARN from the token, use the raw step id in the token, drop the ClientToken in the adapter, drop the local tokenless answer, turn off the local idempotent token, make an empty token idempotent, let a pause of an ended execution succeed, let the runner invoke while paused, let Run or RunUntilComplete ignore the pause, keep the pause across Reset, let a resume of a running execution succeed, and drop the PENDING answer for a paused registered target. Removing PauseEvery from the example fails its test.
  • Integration tests were not run.

Follow-up: cloud integration test

The example runs only on the local runner. A cloud run needs three more changes in a follow-up PR:

  • The example function role gets lambda:PauseDurableExecution.
  • durabletest.CloudRunner gets a way to wait for PAUSED and call ResumeDurableExecution.
  • The CI credentials get lambda:ResumeDurableExecution.

That test also checks the unverified assumptions above against the real service.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

A retry decision can now pause the durable execution, so an operator
can check the cause of the failures and resume it with the Lambda
ResumeDurableExecution API. The feature is experimental.

RetryDecision gets a Pause field and RetryConfig gets PauseEvery. A
step pauses only when its function failed in this invocation and the
decision sets Retry and Pause. The step records RETRY first, then calls
PauseDurableExecution, then ends the invocation with PENDING.

ExecutionPauser is an optional interface, so custom clients keep
compiling. The default client implements it, and the local runner
models pause and resume. The Lambda client goes to v1.117.0, the first
version with PauseDurableExecution.
The pause feature is stable. Remove its experimental markers from the
README, the release notes, the godoc and the example. Fix the
RetryDecision.Pause doc: only a failed pause call logs a warning. A
client without ExecutionPauser retries as usual with no warning.
ParidelPooya added a commit to aws/aws-durable-execution-docs that referenced this pull request Nov 19, 2026
Add Go and Rust tabs to the "Pause after a failed attempt" section of
the retries page: pause every N attempts, pause from a custom strategy,
pause instead of failing, and pauses and other branches.

Document Go RetryConfig.PauseEvery and RetryDecision.Pause, and Rust
RetryStrategyConfigBuilder::pause_every and RetryDecision::Retry pause
with RetryDecision::with_pause, in the signature sections.

The Go examples build and pass go vet against
aws/aws-durable-execution-sdk-go#190. The Rust examples pass rustfmt and
clippy against aws/aws-durable-execution-sdk-rust#214.
TestAbandonedStepDoesNotPause failed about 4% of runs under -race. The
fast branch could return before the slow branch started its step, so the
batch abandoned the slow branch before any RETRY was recorded. The fast
branch now waits until the slow step body has started. The handler also
runs on the test goroutine, so t.Fatalf is safe.

New tests cover these pause branches:
- the Go type name in the warning for an API error without a code;
- RunUntilComplete returning ErrExecutionPaused while paused;
- Reset clearing the paused state;
- a registered durable target that pauses reporting PENDING;
- a repeated pause without a client token returning a conflict.

The ResumeExecution doc now says it resumes only the runner's own
execution. The release notes now name the aws-sdk-go-v2 and smithy-go
bumps.

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

Comment thread durable/durabletest/pause.go Resolved
@ParidelPooya
ParidelPooya marked this pull request as ready for review November 19, 2026 17:44
@ParidelPooya
ParidelPooya merged commit 258bceb into main Nov 19, 2026
30 checks passed
ParidelPooya added a commit to aws/aws-durable-execution-docs that referenced this pull request Nov 19, 2026
* docs: pause the execution from a retry decision

- Add "Pause after a failed attempt" to the retries page, with
  TypeScript and Python examples for pauseEvery / pause_every and
  for a custom strategy that sets pause
- Add a redrive tip: pause instead of failing, fix the cause, then
  resume the execution to run the failed step again
- Add pause to the RetryDecision signatures and pauseEvery /
  pause_every to the RetryStrategyConfig signatures
- Add "Pause and resume an execution" to the manage executions page
- Link the wait-based pause and resume page to the new sections

* docs: drop experimental from pause and list Go and Rust support

The pause option is not experimental. The SDK support note now lists
the TypeScript, Python, Go and Rust SDKs. The Java and C# SDKs do not
support it yet.

* docs: add Go and Rust tabs for pause

Add Go and Rust tabs to the "Pause after a failed attempt" section of
the retries page: pause every N attempts, pause from a custom strategy,
pause instead of failing, and pauses and other branches.

Document Go RetryConfig.PauseEvery and RetryDecision.Pause, and Rust
RetryStrategyConfigBuilder::pause_every and RetryDecision::Retry pause
with RetryDecision::with_pause, in the signature sections.

The Go examples build and pass go vet against
aws/aws-durable-execution-sdk-go#190. The Rust examples pass rustfmt and
clippy against aws/aws-durable-execution-sdk-rust#214.

* docs: match the pause SDK support note to other pages

List only the SDKs that do not have the feature, as the plugins page
does.

* docs: address review on pause from retry

Use language-neutral names for the callback submitter and the retry block.
Describe how a TypeScript step in another branch behaves when a pause ends
the invocation. Add kms:Decrypt to the permissions the resume caller needs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants