Skip to content

feat(sdk): pause the execution from a retry decision - #961

Merged
ParidelPooya merged 7 commits into
mainfrom
feat/step-pause-every
Nov 18, 2026
Merged

ParidelPooya merged 7 commits into
mainfrom
feat/step-pause-every

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available

None.

Description of changes

A retry decision can now pause the durable execution. An operator can then check the cause of the failures and resume the execution with the Lambda ResumeDurableExecution API. The feature is experimental.

Design

Design note

Problem. A step that keeps failing retries until its retry strategy stops. An operator has no way to stop it after a few failures, check the cause and then let it continue.

Cause. The SDK has no link between step retries and the Lambda pause API. Lambda added PauseDurableExecution and ResumeDurableExecution on 2026-11-17.

Options considered.

  1. A StepConfig.pause_every option on the step. Rejected by the maintainer: the retry strategy already decides each retry, so a second per-step setting splits that decision in two places.
  2. A pause field on the retry decision, plus a helper option on the built-in strategy. Chosen. A custom strategy can pause on any condition, and the common case needs one option.

Chosen behavior (maintainer decision).

  1. RetryDecision gets pause: bool = False, marked experimental. RetryDecision.retry(delay, *, pause=False) sets it. Strategies that build RetryDecision(should_retry, delay) work as before.
  2. RetryStrategyConfig gets pause_every: int | None = None (keyword-only), marked experimental. create_retry_strategy returns pause=True when it retries and attempt % pause_every == 0. So pause_every=5 pauses after attempts 5, 10, 15 and so on. Nothing else changes: the delay and the jitter are the same as without the option.
  3. A step pauses only when the step function raised in this invocation and the decision has should_retry=True and pause=True. The raise path passes may_pause=True to the retry handler. The step then:
    1. sends the normal RETRY checkpoint, with the error and the attempt count, and waits until Lambda accepts it;
    2. calls PauseDurableExecution with the execution ARN, a ClientToken and the Reason Step "<name or id>" failed attempt <n>;
    3. ends the invocation with PENDING, with no resume time.
      After the resume, the next attempt runs normally.
  4. The interrupted AT_MOST_ONCE_PER_RETRY replay path also calls the retry strategy. It does not pass may_pause, so it ignores pause.
  5. pause never changes how many attempts run. With should_retry=False, the SDK ignores pause and the step fails as usual.
  6. When the pause call raises, the SDK logs a warning through the step logger and retries as usual. The warning names the error code of a botocore ClientError, or the exception type. It gives a missing lambda:PauseDurableExecution permission as one possible cause. For a ResourceConflictException, it says the execution has closed (for example, stopped) or is already paused.
  7. Pausing is optional for a service client. PausableDurableServiceClient is a runtime-checkable protocol with pause_durable_execution(arn, request). ExecutionState.pause_execution returns False for a client without it, and the step retries as usual.
  8. The SDK does not validate pause_every. The docstring states the expected values.
  9. The docs say that the developer gives the execution role lambda:PauseDurableExecution on its own durable executions. A function that uses a customer managed AWS KMS key also needs kms:Decrypt on that key, as the Lambda pause and resume prerequisites require.

Why the retry is saved before the pause. Once a pause takes effect, Lambda saves one more checkpoint and then accepts no more. So the step saves RETRY with a blocking checkpoint first and asks to pause after that:

  • The batcher sends updates in queue order. So the blocking RETRY also sends this step's earlier asynchronous START before the pause call.
  • A checkpoint from another operation cannot use up the last save before this step's RETRY, because RETRY is saved before the pause call.
  • If Lambda answers RETRY without a checkpoint token, the existing tokenless-response path suspends the invocation, and no pause is requested.

What else changes.

  • lambda_service.py: PauseExecutionRequest (frozen dataclass with client_token and reason), PausableDurableServiceClient, and LambdaClient.pause_durable_execution, which calls boto3 pause_durable_execution with DurableExecutionArn, ClientToken and Reason.
  • ClientToken: the blake2b hash (digest size 32, so 64 hex characters) of pause:<execution ARN>:<operation id>:<attempt>. It is the same when an attempt runs again and differs across attempts and executions.
  • state.py: ExecutionState.pause_execution(request).
  • boto3 goes from >=1.43.114 to >=1.44.5, the first version with pause_durable_execution.
  • Testing package: InMemoryServiceClient.pause_durable_execution and CheckpointProcessor.pause_execution. They mark the local execution paused without waiting for the calling invocation to end. The next checkpoint of that invocation is answered without a token, as Lambda does. DurableFunctionTestRunner.resume_execution() continues it.
  • New example step/step_with_pause_every.py ("integration": false in the catalog). It uses create_retry_strategy with pause_every=3. The SDK README has a section that shows the helper and a custom strategy.

What the change removes. StepConfig.pause_every from the earlier revision of this PR. It was never released.

Simpler options considered

  • No helper option, only the pause field. A custom strategy would then copy the backoff, jitter and error filters of create_retry_strategy to pause every Nth attempt. The helper option is one condition.
  • Pause without saving RETRY first. Lambda could then accept another operation's checkpoint as the last save, and this step's retry state would be lost.

Net change

  • Source: 355 lines added and 8 removed, in 7 source files (core 256 added, testing 47, example 52).
  • Public API: RetryDecision.pause, the pause argument of RetryDecision.retry, RetryStrategyConfig.pause_every, PauseExecutionRequest, PausableDurableServiceClient, LambdaClient.pause_durable_execution, ExecutionState.pause_execution. Testing package: InMemoryServiceClient.pause_durable_execution, CheckpointProcessor.pause_execution.

Assumptions

  • Confirmed by a live test against Lambda: after the PauseDurableExecution call succeeds, Lambda holds the next invocation until the execution is resumed. Before the call succeeds, the current invocation is still running. So no extra attempt runs between the RETRY checkpoint and the pause.
  • Confirmed by the docs: the pause is asynchronous, and the status does not change during the PauseDurableExecution call. See Pausing and resuming durable executions.
  • Unverified: a function's execution role can pause its own running execution with PauseDurableExecution. The docs say the action applies to the durable execution ARN. The cloud follow-up will check it.
  • Unverified: when the pause is requested after RETRY is saved, the invocation returns PENDING, and Lambda then moves the execution to PAUSED. The docs cover the case where the function is not running, but not this exact order. The cloud follow-up will check it.
  • Unverified: after a pause takes effect, Lambda answers the next checkpoint of the running invocation without a token. The local testing client models it this way.

Decisions not covered by the issue or the design note

  • pause is a dataclass field with default False, so RetryDecision(should_retry, delay) still works. RetryDecision.retry takes pause as keyword-only.
  • pause_every is keyword-only, so it does not change the positional order of RetryStrategyConfig.
  • with_retry ignores pause. The WithRetryConfig docstring, the RetryDecision.pause docstring and the README say so.
  • The wait_for_callback submitter honors pause, because it runs as a context.step with the configured retry strategy.
  • create_linear_retry_strategy has no pause_every option. A developer can wrap it to set pause.
  • The Reason uses the step name, or the operation id when the step has no name. It is cut to 1024 characters, the API limit.
  • The warning checks the error code only for ResourceConflictException. Every other error gets the permission hint.
  • LambdaClient.pause_durable_execution uses the API default for the execution timeout while paused. It passes client errors on unchanged, and the step logs them.
  • pause_every=0 raises ZeroDivisionError in the strategy, and a negative value pauses on its multiples. Not validated, as agreed.
  • The example has "integration": false. DurableFunctionCloudTestRunner.resume_execution() raises NotImplementedError, and the generated function role has no pause permission.
  • InMemoryServiceClient imports PauseExecutionRequest for typing only, so the testing package still loads with a core SDK that has no pause support.

Tests

  • hatch run dev-core:cov: 2281 passed, 98.56% coverage.
  • hatch run dev-testing:test: 1886 passed.
  • hatch run dev-examples:test: 204 passed, 9 skipped.
  • hatch run dev-core:typecheck and hatch run types:check: no issues.
  • hatch fmt --check in the core, testing and examples packages: clean.
  • tests/operation/step_pause_test.py (29 tests): pause when the decision pauses, with RETRY saved before the pause call; pause_every pauses on later multiples and not on other attempts; no pause for a decision without pause or with pause=False; a strategy without pause still retries; no pause when the strategy stops or when should_retry=False; the warnings; a client without pause support; no pause on the interrupted at-most-once replay path; a revoked token on RETRY suspends without a pause; the pause waits until RETRY is accepted; a real ExecutionState sends START and RETRY before the pause; the ClientToken is stable per attempt and differs per execution; the Reason; ExecutionState.pause_execution; LambdaClient.pause_durable_execution.
  • tests/retries_test.py (17 pause cases): the pause default and RetryDecision.retry(pause=True); pause_every defaults to None; pause on multiples only (1, 4, 6, 9 no; 5, 10, 15 yes); no pause when attempts are exhausted or the error is not retryable; the pausing attempt keeps the configured jitter (FULL, HALF and NONE); jitter on other attempts; backoff and the max_delay cap on the pausing attempt.
  • Testing package: 3 local-runner e2e tests in tests/e2e/step_pause_every_test.py (pause on attempts 2 and 4 with resume, no pause when the step succeeds first, a custom strategy that pauses on attempt 3), 3 CheckpointProcessor.pause_execution tests and 2 InMemoryServiceClient tests.
  • Example: test/step/test_step_with_pause_every.py pauses after attempt 3 and finishes on attempt 5 after resume.
  • The pausing-attempt jitter test fails for FULL and HALF jitter when the pausing attempt skips jitter.
  • Integration tests were not run.

Edited test expectations (rule 5): test_no_retry_factory in tests/retries_test.py gets one more assertion, decision.pause is False. No existing expectation was changed.

Checklist

  • Every new branch has a test that fails without it (rule 4)
  • For a bug fix: the first commit adds a test that fails on the current code (rule 5). Not a bug fix.
  • Every edited test expectation is listed in the description, with the reason (rule 5)

Follow-up: cloud integration test

The example runs only on the local runner. A cloud run needs three more changes. We will add them in a follow-up PR after this one merges:

  • The example function role gets lambda:PauseDurableExecution.
  • DurableFunctionCloudTestRunner gets a way to wait for PAUSED and call ResumeDurableExecution.
  • The CI credentials get lambda:ResumeDurableExecution.

That test also checks the unverified assumptions above against the real service.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Add the experimental StepConfig.pause_every option. When a step function
raises and the retry strategy retries, the SDK checks the attempt. If
the attempt is a multiple of pause_every, the SDK calls the Lambda
PauseDurableExecution API. It then sends the normal RETRY checkpoint and
ends the invocation with PENDING. The next attempt runs after an
operator resumes the execution.

- pause_every does not change how many attempts run. When the retry
  strategy stops, the step fails as usual.
- An interrupted AT_MOST_ONCE_PER_RETRY attempt does not pause.
- A failed pause call logs a warning, and the step retries as usual.
- A service client without pause_durable_execution retries as usual.
  The method lives on a separate PausableDurableServiceClient protocol.
- The client token is a hash of the operation id and the attempt.

The in-memory test client pauses the execution, so local tests can
resume it with DurableFunctionTestRunner.resume_execution().

Raise the boto3 minimum to 1.44.5, the first release with the pause
API. Add the step_with_pause_every example.
The step now saves the RETRY checkpoint first. Only after Lambda accepts
it does the SDK call PauseDurableExecution and suspend the invocation.
Before, the pause came first. So Lambda could answer this step's RETRY,
its earlier START, or a sibling's checkpoint without a token, and that
update was lost.

- If the RETRY answer has no token, the execution is already pausing.
  The step then suspends and does not request a pause.
- The client token now hashes the durable execution ARN, the operation
  id and the attempt. So two executions never share a token.
- The pause warning names the error. It says a missing permission is
  one possible cause. A ResourceConflictException says the execution is
  already pausing or paused.
- Docs: use pause_every with long retry delays (30 seconds or more) and
  no full jitter. A short delay can let one more attempt run before the
  pause applies. A step in a map or parallel branch can still pause the
  execution after the map has finished.
- The example uses a 60 second delay with no jitter and no backoff.
- The testing checkpoint processor drops a redundant is_paused check.
A retry of a step that runs alone suspends the invocation while it
waits. Only map and parallel branches wait inside the invocation. So
the docs no longer say the SDK waits inside the invocation.

Also fix an early line break in the pause_every docstring.
Lambda returns ResourceConflictException for a pause only when the
execution has closed or is already paused. A second pause call while
the execution is pausing has no effect. So the warning no longer says
"already pausing".
Move the pause from StepConfig.pause_every to the retry decision.
RetryDecision gets pause: bool = False. A step pauses the execution
when the decision retries with pause=True and the step function
raised in this invocation. The interrupted AT_MOST_ONCE_PER_RETRY
replay path ignores pause. A decision with should_retry=False never
pauses.

RetryStrategyConfig gets pause_every. create_retry_strategy returns
pause=True on attempts that are multiples of pause_every. The pausing
attempt uses the delay without jitter, so its delay is never near
zero. Other attempts keep their jitter.

StepConfig.pause_every is removed. It was never released.
@ParidelPooya ParidelPooya changed the title feat(sdk): add pause_every step option feat(sdk): pause the execution from a retry decision Nov 18, 2026
@yaythomas

Copy link
Copy Markdown
Contributor

It looks like this PR description was copied from the JavaScript PR: it refers to pauseEvery, createRetryStrategy, @aws-sdk/client-lambda, SHA-256, JavaScript files, and npm results. Can it be replaced with the Python-specific design, decisions, and test results?

A live test against Lambda showed that Lambda holds the next
invocation after the PauseDurableExecution call succeeds. So no
extra attempt can run before the pause takes effect.

The pause_every helper no longer skips jitter on the pausing
attempt. That attempt uses jitter_strategy like every other attempt.
The docs no longer ask for long delays or jitter limits.

The docs now name kms:Decrypt for functions that use a customer
managed AWS KMS key. They say that with_retry ignores pause and that
the wait_for_callback submitter uses it. The examples of a pausing
decision use RetryDecision.retry(delay, pause=True).

The test test_step_config_has_no_pause_every is removed. It passed
on main too.

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@ParidelPooya
ParidelPooya marked this pull request as ready for review November 18, 2026 21:31
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 18, 2026 21:31 — with GitHub Actions Active
@ParidelPooya
ParidelPooya merged commit fc7c752 into main Nov 18, 2026
45 of 46 checks passed
@ParidelPooya
ParidelPooya deleted the feat/step-pause-every branch November 18, 2026 21:31
ParidelPooya added a commit that referenced this pull request Nov 18, 2026
* chore: prepare Python SDK 3.1.0 and Testing 1.5.0 release

Core 3.1.0 adds the experimental pause_every retry option (#961).
Testing 1.5.0 adds local pause support for it. Update the three
conformance packages to pin Core 3.1.0.

* chore: prepare Extras and Compute Worker alphas

The publish workflow builds and publishes every package in its matrix.
PyPI has Extras 0.1.0a0 and Compute Worker 0.0.1a0. The repository
versions 0.1.0 and 0.0.1 would publish as stable releases. Both packages
are experimental, so this release publishes the next prereleases.

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 6ca306bd Deployed Nov 18, 2026 by ParidelPooya via notify / Generate notification summary #453
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants