Repository navigation
feat(sdk): pause the execution from a retry decision - #961
Merged
Merged
Conversation
Add the experimental StepConfig.pause_every option. When a step function raises and the retry strategy retries, the SDK checks the attempt. If the attempt is a multiple of pause_every, the SDK calls the Lambda PauseDurableExecution API. It then sends the normal RETRY checkpoint and ends the invocation with PENDING. The next attempt runs after an operator resumes the execution. - pause_every does not change how many attempts run. When the retry strategy stops, the step fails as usual. - An interrupted AT_MOST_ONCE_PER_RETRY attempt does not pause. - A failed pause call logs a warning, and the step retries as usual. - A service client without pause_durable_execution retries as usual. The method lives on a separate PausableDurableServiceClient protocol. - The client token is a hash of the operation id and the attempt. The in-memory test client pauses the execution, so local tests can resume it with DurableFunctionTestRunner.resume_execution(). Raise the boto3 minimum to 1.44.5, the first release with the pause API. Add the step_with_pause_every example.
The step now saves the RETRY checkpoint first. Only after Lambda accepts it does the SDK call PauseDurableExecution and suspend the invocation. Before, the pause came first. So Lambda could answer this step's RETRY, its earlier START, or a sibling's checkpoint without a token, and that update was lost. - If the RETRY answer has no token, the execution is already pausing. The step then suspends and does not request a pause. - The client token now hashes the durable execution ARN, the operation id and the attempt. So two executions never share a token. - The pause warning names the error. It says a missing permission is one possible cause. A ResourceConflictException says the execution is already pausing or paused. - Docs: use pause_every with long retry delays (30 seconds or more) and no full jitter. A short delay can let one more attempt run before the pause applies. A step in a map or parallel branch can still pause the execution after the map has finished. - The example uses a 60 second delay with no jitter and no backoff. - The testing checkpoint processor drops a redundant is_paused check.
A retry of a step that runs alone suspends the invocation while it waits. Only map and parallel branches wait inside the invocation. So the docs no longer say the SDK waits inside the invocation. Also fix an early line break in the pause_every docstring.
Lambda returns ResourceConflictException for a pause only when the execution has closed or is already paused. A second pause call while the execution is pausing has no effect. So the warning no longer says "already pausing".
yaythomas
reviewed
Nov 18, 2026
Move the pause from StepConfig.pause_every to the retry decision. RetryDecision gets pause: bool = False. A step pauses the execution when the decision retries with pause=True and the step function raised in this invocation. The interrupted AT_MOST_ONCE_PER_RETRY replay path ignores pause. A decision with should_retry=False never pauses. RetryStrategyConfig gets pause_every. create_retry_strategy returns pause=True on attempts that are multiples of pause_every. The pausing attempt uses the delay without jitter, so its delay is never near zero. Other attempts keep their jitter. StepConfig.pause_every is removed. It was never released.
Contributor
|
It looks like this PR description was copied from the JavaScript PR: it refers to |
yaythomas
reviewed
Nov 18, 2026
yaythomas
reviewed
Nov 18, 2026
A live test against Lambda showed that Lambda holds the next invocation after the PauseDurableExecution call succeeds. So no extra attempt can run before the pause takes effect. The pause_every helper no longer skips jitter on the pausing attempt. That attempt uses jitter_strategy like every other attempt. The docs no longer ask for long delays or jitter limits. The docs now name kms:Decrypt for functions that use a customer managed AWS KMS key. They say that with_retry ignores pause and that the wait_for_callback submitter uses it. The examples of a pausing decision use RetryDecision.retry(delay, pause=True). The test test_step_config_has_no_pause_every is removed. It passed on main too.
ParidelPooya
marked this pull request as ready for review
November 18, 2026 21:31
1 task done
ParidelPooya
added a commit
that referenced
this pull request
Nov 18, 2026
* chore: prepare Python SDK 3.1.0 and Testing 1.5.0 release Core 3.1.0 adds the experimental pause_every retry option (#961). Testing 1.5.0 adds local pause support for it. Update the three conformance packages to pin Core 3.1.0. * chore: prepare Extras and Compute Worker alphas The publish workflow builds and publishes every package in its matrix. PyPI has Extras 0.1.0a0 and Compute Worker 0.0.1a0. The repository versions 0.1.0 and 0.0.1 would publish as stable releases. Both packages are experimental, so this release publishes the next prereleases.
This was referenced Nov 18, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available
None.
Description of changes
A retry decision can now pause the durable execution. An operator can then check the cause of the failures and resume the execution with the Lambda
ResumeDurableExecutionAPI. The feature is experimental.Design
Design note
Problem. A step that keeps failing retries until its retry strategy stops. An operator has no way to stop it after a few failures, check the cause and then let it continue.
Cause. The SDK has no link between step retries and the Lambda pause API. Lambda added
PauseDurableExecutionandResumeDurableExecutionon 2026-11-17.Options considered.
StepConfig.pause_everyoption on the step. Rejected by the maintainer: the retry strategy already decides each retry, so a second per-step setting splits that decision in two places.pausefield on the retry decision, plus a helper option on the built-in strategy. Chosen. A custom strategy can pause on any condition, and the common case needs one option.Chosen behavior (maintainer decision).
RetryDecisiongetspause: bool = False, marked experimental.RetryDecision.retry(delay, *, pause=False)sets it. Strategies that buildRetryDecision(should_retry, delay)work as before.RetryStrategyConfiggetspause_every: int | None = None(keyword-only), marked experimental.create_retry_strategyreturnspause=Truewhen it retries andattempt % pause_every == 0. Sopause_every=5pauses after attempts 5, 10, 15 and so on. Nothing else changes: the delay and the jitter are the same as without the option.should_retry=Trueandpause=True. The raise path passesmay_pause=Trueto the retry handler. The step then:PauseDurableExecutionwith the execution ARN, a ClientToken and the ReasonStep "<name or id>" failed attempt <n>;After the resume, the next attempt runs normally.
AT_MOST_ONCE_PER_RETRYreplay path also calls the retry strategy. It does not passmay_pause, so it ignorespause.pausenever changes how many attempts run. Withshould_retry=False, the SDK ignorespauseand the step fails as usual.ClientError, or the exception type. It gives a missinglambda:PauseDurableExecutionpermission as one possible cause. For aResourceConflictException, it says the execution has closed (for example, stopped) or is already paused.PausableDurableServiceClientis a runtime-checkable protocol withpause_durable_execution(arn, request).ExecutionState.pause_executionreturns False for a client without it, and the step retries as usual.pause_every. The docstring states the expected values.lambda:PauseDurableExecutionon its own durable executions. A function that uses a customer managed AWS KMS key also needskms:Decrypton that key, as the Lambda pause and resume prerequisites require.Why the retry is saved before the pause. Once a pause takes effect, Lambda saves one more checkpoint and then accepts no more. So the step saves RETRY with a blocking checkpoint first and asks to pause after that:
What else changes.
lambda_service.py:PauseExecutionRequest(frozen dataclass withclient_tokenandreason),PausableDurableServiceClient, andLambdaClient.pause_durable_execution, which calls boto3pause_durable_executionwithDurableExecutionArn,ClientTokenandReason.pause:<execution ARN>:<operation id>:<attempt>. It is the same when an attempt runs again and differs across attempts and executions.state.py:ExecutionState.pause_execution(request).boto3goes from>=1.43.114to>=1.44.5, the first version withpause_durable_execution.InMemoryServiceClient.pause_durable_executionandCheckpointProcessor.pause_execution. They mark the local execution paused without waiting for the calling invocation to end. The next checkpoint of that invocation is answered without a token, as Lambda does.DurableFunctionTestRunner.resume_execution()continues it.step/step_with_pause_every.py("integration": falsein the catalog). It usescreate_retry_strategywithpause_every=3. The SDK README has a section that shows the helper and a custom strategy.What the change removes.
StepConfig.pause_everyfrom the earlier revision of this PR. It was never released.Simpler options considered
pausefield. A custom strategy would then copy the backoff, jitter and error filters ofcreate_retry_strategyto pause every Nth attempt. The helper option is one condition.Net change
RetryDecision.pause, thepauseargument ofRetryDecision.retry,RetryStrategyConfig.pause_every,PauseExecutionRequest,PausableDurableServiceClient,LambdaClient.pause_durable_execution,ExecutionState.pause_execution. Testing package:InMemoryServiceClient.pause_durable_execution,CheckpointProcessor.pause_execution.Assumptions
PauseDurableExecutioncall succeeds, Lambda holds the next invocation until the execution is resumed. Before the call succeeds, the current invocation is still running. So no extra attempt runs between the RETRY checkpoint and the pause.PauseDurableExecutioncall. See Pausing and resuming durable executions.PauseDurableExecution. The docs say the action applies to the durable execution ARN. The cloud follow-up will check it.Decisions not covered by the issue or the design note
pauseis a dataclass field with default False, soRetryDecision(should_retry, delay)still works.RetryDecision.retrytakespauseas keyword-only.pause_everyis keyword-only, so it does not change the positional order ofRetryStrategyConfig.with_retryignorespause. TheWithRetryConfigdocstring, theRetryDecision.pausedocstring and the README say so.wait_for_callbacksubmitter honorspause, because it runs as acontext.stepwith the configured retry strategy.create_linear_retry_strategyhas nopause_everyoption. A developer can wrap it to setpause.ResourceConflictException. Every other error gets the permission hint.LambdaClient.pause_durable_executionuses the API default for the execution timeout while paused. It passes client errors on unchanged, and the step logs them.pause_every=0raisesZeroDivisionErrorin the strategy, and a negative value pauses on its multiples. Not validated, as agreed."integration": false.DurableFunctionCloudTestRunner.resume_execution()raisesNotImplementedError, and the generated function role has no pause permission.InMemoryServiceClientimportsPauseExecutionRequestfor typing only, so the testing package still loads with a core SDK that has no pause support.Tests
hatch run dev-core:cov: 2281 passed, 98.56% coverage.hatch run dev-testing:test: 1886 passed.hatch run dev-examples:test: 204 passed, 9 skipped.hatch run dev-core:typecheckandhatch run types:check: no issues.hatch fmt --checkin the core, testing and examples packages: clean.tests/operation/step_pause_test.py(29 tests): pause when the decision pauses, with RETRY saved before the pause call;pause_everypauses on later multiples and not on other attempts; no pause for a decision withoutpauseor withpause=False; a strategy withoutpausestill retries; no pause when the strategy stops or whenshould_retry=False; the warnings; a client without pause support; no pause on the interrupted at-most-once replay path; a revoked token on RETRY suspends without a pause; the pause waits until RETRY is accepted; a realExecutionStatesends START and RETRY before the pause; the ClientToken is stable per attempt and differs per execution; the Reason;ExecutionState.pause_execution;LambdaClient.pause_durable_execution.tests/retries_test.py(17 pause cases): thepausedefault andRetryDecision.retry(pause=True);pause_everydefaults to None; pause on multiples only (1, 4, 6, 9 no; 5, 10, 15 yes); no pause when attempts are exhausted or the error is not retryable; the pausing attempt keeps the configured jitter (FULL, HALF and NONE); jitter on other attempts; backoff and themax_delaycap on the pausing attempt.tests/e2e/step_pause_every_test.py(pause on attempts 2 and 4 with resume, no pause when the step succeeds first, a custom strategy that pauses on attempt 3), 3CheckpointProcessor.pause_executiontests and 2InMemoryServiceClienttests.test/step/test_step_with_pause_every.pypauses after attempt 3 and finishes on attempt 5 after resume.Edited test expectations (rule 5):
test_no_retry_factoryintests/retries_test.pygets one more assertion,decision.pause is False. No existing expectation was changed.Checklist
Follow-up: cloud integration test
The example runs only on the local runner. A cloud run needs three more changes. We will add them in a follow-up PR after this one merges:
lambda:PauseDurableExecution.DurableFunctionCloudTestRunnergets a way to wait forPAUSEDand callResumeDurableExecution.lambda:ResumeDurableExecution.That test also checks the unverified assumptions above against the real service.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.