Repository navigation
feat(step): pause the execution from a retry decision - #190
Merged
Merged
Conversation
A retry decision can now pause the durable execution, so an operator can check the cause of the failures and resume it with the Lambda ResumeDurableExecution API. The feature is experimental. RetryDecision gets a Pause field and RetryConfig gets PauseEvery. A step pauses only when its function failed in this invocation and the decision sets Retry and Pause. The step records RETRY first, then calls PauseDurableExecution, then ends the invocation with PENDING. ExecutionPauser is an optional interface, so custom clients keep compiling. The default client implements it, and the local runner models pause and resume. The Lambda client goes to v1.117.0, the first version with PauseDurableExecution.
The pause feature is stable. Remove its experimental markers from the README, the release notes, the godoc and the example. Fix the RetryDecision.Pause doc: only a failed pause call logs a warning. A client without ExecutionPauser retries as usual with no warning.
ParidelPooya
added a commit
to aws/aws-durable-execution-docs
that referenced
this pull request
Nov 19, 2026
Add Go and Rust tabs to the "Pause after a failed attempt" section of the retries page: pause every N attempts, pause from a custom strategy, pause instead of failing, and pauses and other branches. Document Go RetryConfig.PauseEvery and RetryDecision.Pause, and Rust RetryStrategyConfigBuilder::pause_every and RetryDecision::Retry pause with RetryDecision::with_pause, in the signature sections. The Go examples build and pass go vet against aws/aws-durable-execution-sdk-go#190. The Rust examples pass rustfmt and clippy against aws/aws-durable-execution-sdk-rust#214.
TestAbandonedStepDoesNotPause failed about 4% of runs under -race. The fast branch could return before the slow branch started its step, so the batch abandoned the slow branch before any RETRY was recorded. The fast branch now waits until the slow step body has started. The handler also runs on the test goroutine, so t.Fatalf is safe. New tests cover these pause branches: - the Go type name in the warning for an API error without a code; - RunUntilComplete returning ErrExecutionPaused while paused; - Reset clearing the paused state; - a registered durable target that pauses reporting PENDING; - a repeated pause without a client token returning a conflict. The ResumeExecution doc now says it resumes only the runner's own execution. The release notes now name the aws-sdk-go-v2 and smithy-go bumps.
ParidelPooya
marked this pull request as ready for review
November 19, 2026 17:44
This was referenced Nov 19, 2026
ParidelPooya
added a commit
to aws/aws-durable-execution-docs
that referenced
this pull request
Nov 19, 2026
* docs: pause the execution from a retry decision - Add "Pause after a failed attempt" to the retries page, with TypeScript and Python examples for pauseEvery / pause_every and for a custom strategy that sets pause - Add a redrive tip: pause instead of failing, fix the cause, then resume the execution to run the failed step again - Add pause to the RetryDecision signatures and pauseEvery / pause_every to the RetryStrategyConfig signatures - Add "Pause and resume an execution" to the manage executions page - Link the wait-based pause and resume page to the new sections * docs: drop experimental from pause and list Go and Rust support The pause option is not experimental. The SDK support note now lists the TypeScript, Python, Go and Rust SDKs. The Java and C# SDKs do not support it yet. * docs: add Go and Rust tabs for pause Add Go and Rust tabs to the "Pause after a failed attempt" section of the retries page: pause every N attempts, pause from a custom strategy, pause instead of failing, and pauses and other branches. Document Go RetryConfig.PauseEvery and RetryDecision.Pause, and Rust RetryStrategyConfigBuilder::pause_every and RetryDecision::Retry pause with RetryDecision::with_pause, in the signature sections. The Go examples build and pass go vet against aws/aws-durable-execution-sdk-go#190. The Rust examples pass rustfmt and clippy against aws/aws-durable-execution-sdk-rust#214. * docs: match the pause SDK support note to other pages List only the SDKs that do not have the feature, as the plugins page does. * docs: address review on pause from retry Use language-neutral names for the callback submitter and the retry block. Describe how a TypeScript step in another branch behaves when a pause ends the invocation. Add kms:Decrypt to the permissions the resume caller needs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
A retry decision can now pause the durable execution. An operator can then check the cause of the failures and resume the execution with the Lambda
ResumeDurableExecutionAPI.Design note
The design was agreed in the JavaScript SDK (aws-durable-execution-sdk-js#1253) and the Python SDK (aws-durable-execution-sdk-python#961). This PR ports it to Go.
Problem. A step that keeps failing retries until its retry strategy stops. An operator has no way to stop it after a few failures, check the cause and then let it continue.
Cause. The SDK has no link between step retries and the Lambda pause API. Lambda added
PauseDurableExecutionandResumeDurableExecutionon 2026-11-17.Behavior.
RetryDecisiongets aPause boolfield. Its zero value is false, so existing strategies and decision literals work as before.RetryConfiggetsPauseEvery int. The strategy thatNewRetryStrategybuilds setsPauseon a retry after attempt n whenn % PauseEvery == 0. SoPauseEvery: 5pauses after attempts 5, 10, 15 and so on. The delay and the jitter do not change.LinearRetryConfiggets no option.RetryandPause. The step then:PauseDurableExecutionwith the execution ARN, a ClientToken and the ReasonStep "<name or wire id>" failed attempt <n>;After the resume, the next attempt runs normally.
AtMostOncePerRetryreplay path also calls the strategy. It ignoresPause.Pausenever changes the attempt count. WithRetryfalse, the SDK ignoresPauseand the step fails as usual.ResourceConflictExceptionit says "The execution has closed (for example, stopped) or is already paused." Every other error says a missinglambda:PauseDurableExecutionpermission is one possible cause.ExecutionPauseris an optional interface. The step checks for it at run time, so a customExecutionClientkeeps compiling and retries as usual. The default Lambda client implements it.PauseEvery. The doc comment states the expected values.Why the retry is saved before the pause. Once a pause takes effect, Lambda saves one more checkpoint and then accepts no more. So the step saves RETRY first and asks to pause after that. The checkpointer sends calls in queue order, so an earlier START queued without waiting is also saved before RETRY returns. If Lambda answers RETRY without a token, the existing tokenless-response path halts the checkpointer, and the step requests no pause.
Abandoned steps. A step in a
ParallelorMapbranch that the batch abandoned after early completion still records its RETRY in Go. It does not pause: the step checks the abandon handle before the pause call. Its wait for the retry then returns at once, so it does not hang.Other branches. Steps already running in other branches finish their current attempt. The SDK sends no checkpoint after the pause request, so an attempt that finishes after it runs again after the resume.
Decisions not covered by the agreed design
Pauseis a plainboolfield on the struct. The[0]func()guard already blocks unkeyed literals, so adding the field breaks no caller.settleStepFailureasmayPause. Only the body-failure path passes true.ExecutionPauser.PauseExecutiontakesPauseExecutionInput{ExecutionArn, ClientToken, Reason}and returns an emptyPauseExecutionOutput, matching the style of the other client methods.ExecutionTimeoutPausedis not exposed; the API default applies.pause:<execution ARN>:<wire step id>:<attempt>, as 64 lowercase hex characters, as in the JavaScript SDK.APIErrorcode when there is one, and the Go type name otherwise.durable.RetryignoresPause, because it does not run as a step. TheWaitForCallbacksubmitter honors it, because it runs as a step. Both are documented.PauseEvery: 0never pauses. A negative value pauses on the multiples of its absolute value. Not validated, as agreed.service/lambdarequirement goes from v1.99.0 to v1.117.0, the first version withPauseDurableExecution.go mod tidyalso movesaws-sdk-go-v2to v1.48.0 andsmithy-goto v1.28.9 in the root,telemetry,conformanceandexamplesmodules. The release notes name all three bumps.durabletest: the in-memory client implementsExecutionPauser. It marks the execution paused without waiting for the invocation to end, answers the next checkpoint without a token, and returnsResourceConflictExceptionfor a pause with another token, a pause of an ended execution, or a resume of an execution that is not paused.LocalRunnergainsPaused(),ResumeExecution()andErrExecutionPaused, whichRunandRunUntilCompletereturn while the execution is paused.ResumeExecutionresumes only the runner's own execution. A registered durable target that pauses stays PENDING; the cloud follow-up covers resuming it.ExecutionPausedorExecutionResumedhistory events. The history-to-operation fold ignores unknown event types, so the new client version needs no mapping.step-pause-every, local only. It is not inbuild.shortemplate.yaml, so the cloud tests skip it. It is inparity-map.txtagainst the JavaScriptstep/pause-everyexample.Assumptions
PauseDurableExecutioncall succeeds, Lambda holds the next invocation until the execution is resumed. So the pausing attempt needs no special delay or jitter.durabletest.CloudRunnerkeeps polling a PAUSED execution until its own timeout, because PAUSED is not a terminal status. The cloud follow-up will handle it.Tests
go build ./...,go vet ./...,golangci-lint run ./...andgolangci-lint fmt --diff ./...pass in every module. The release tool'scheckandconsumercommands pass.go test -race ./...: 2377 tests and subtests pass, 1 skipped.TestDefaultClientRequestTimeoutswas skipped locally. It fails onmaintoo in my environment, because the shell's AWS profile has no shared config.telemetry,conformance,examplesandanalysis:scripts/ci-local.sh <module>passes.durable/pause_test.go: pause onPause: true(RETRY then pause, no later checkpoint), no pause on false or absentPause(attempt 2 runs in the same invocation), no pause when not retrying, the interrupted path ignoresPause, a tokenless RETRY answer requests no pause, an abandonedParallelbranch does not pause and does not hang (it passes 300 of 300 runs withgo test -race -count=300), pause failures log the warning and retry as usual (four error kinds: a service error, a plain error, an API error without a code that reports its Go type, andResourceConflictExceptionwith its text), a client withoutExecutionPauserretries as usual, ClientToken stable and different per ARN, step and attempt (64 hex), Reason for an unnamed step and truncation,PauseEveryon multiples only (4 no, 5 yes, 6 no, 10 yes, 15 yes), the full jitter kept on the pausing attempt, and no pause fromExponentialBackoffor a config without the option.durable/lambda_client_adapter_test.go: the adapter maps the fields, leavesExecutionTimeoutPausedunset, and returns the service error.durabletest: pause after attempts 2 and 4 withPauseEvery: 2,Runrefused while paused, resume, success on attempt 5;RunUntilCompletereturnsErrExecutionPausedwhile paused;Resetclears the pause; a registered durable target that pauses stays PENDING and is not invoked again; theWaitForCallbacksubmitter pauses; resume of an execution that is not paused conflicts; the in-memory client's idempotent token, a repeated pause without a token conflicts, the other conflicts and the tokenless answer.Pause, ignoremayPause, drop the abandon check, treat a failed pause as success, drop the warning, drop the conflict hint, drop the empty-code check on the error name, pause on every retry, pause before the RETRY checkpoint, drop truncation, drop the ARN from the token, use the raw step id in the token, drop the ClientToken in the adapter, drop the local tokenless answer, turn off the local idempotent token, make an empty token idempotent, let a pause of an ended execution succeed, let the runner invoke while paused, letRunorRunUntilCompleteignore the pause, keep the pause acrossReset, let a resume of a running execution succeed, and drop the PENDING answer for a paused registered target. RemovingPauseEveryfrom the example fails its test.Follow-up: cloud integration test
The example runs only on the local runner. A cloud run needs three more changes in a follow-up PR:
lambda:PauseDurableExecution.durabletest.CloudRunnergets a way to wait forPAUSEDand callResumeDurableExecution.lambda:ResumeDurableExecution.That test also checks the unverified assumptions above against the real service.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.