Skip to content

Use exponential backoff for throttled API requests - #8956

Open
isaacroldan wants to merge 3 commits into
mainfrom
codex/api-throttle-backoff
Open

isaacroldan wants to merge 3 commits into
mainfrom
codex/api-throttle-backoff

Conversation

@isaacroldan

@isaacroldan isaacroldan commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

Concurrent CLI operations share API quotas. Fixed retries can fail before the quota recovers, as seen in #8947.

Bug fixed Previous behavior
Retry exhaustion One-second retries could give up before quota recovery.
Synchronized retries Concurrent requests retried together and hit the quota again.
Numeric Retry-After Seconds were treated as milliseconds; fractional values were truncated.
HTTP-date Retry-After Date headers were parsed incorrectly; asctime used local time instead of UTC.
Partial mutation data The throttle handler retried non-null mutation data that could include a completed write.
Unrelated errors Mixed throttle and permission/validation errors could trigger retries.
Nullable read queries Throttled reads returning data: {appsConnection: null} lost their retries.

WHAT is this pull request doing?

Retry THROTTLED/429 requests with exponential backoff and jitter.

  • Delay: up to eight seconds, unless Retry-After specifies a delay.
  • Budget: ten retries or 30 seconds of outer throttle retries. In-flight requests can finish.
  • Request handling: preserve variables, keep read queries eligible, and apply the partial-data guard to mutations.
  • Server delays: support numeric and HTTP-date Retry-After values.

How to manually test your changes?

From a disposable app:

shopify app deploy --no-release --verbose

If throttled, retry logs should show increasing, varied delays, and deployment should resume when quota recovers. Invalid configuration should still return its original error.

Validation

96 focused API tests, changed-file lint, and CLI Kit type checks pass.

Nullable-query regressions fail before the fix for both string and typed-document requests. Retry-After cases pass in UTC, Europe/Madrid, and America/Los_Angeles.

Stress-test results

Tested commit: c9285ac. Expected successes and expected errors matched the checks below.

Test Scale Result
Timing and deadlines 46,500 scenario executions across three time zones Retry caps, deadlines, server delays, and final errors/IDs matched expectations.
Response handling 1,472 cases Nullable reads, mutation guards, and error classification passed.
Real local HTTPS 200 operations; concurrency 24; 580 requests 130 successes, 70 expected errors, 80 token refreshes; no token/variable/ID mix-ups or open sockets after cleanup.

Quota unavailable for 20 seconds — eight batches of 128 concurrent operations:

Metric Base 66cab949 This PR
Completed operations 0 / 1,024 1,024 / 1,024
Total requests 11,264 7,160
Peak retry burst per simulated 10 ms 128 5–8
Stress-test setup and checks

Timing: the same 15,000 seeded cases and 500 fake-timer cases ran in each of UTC, Europe/Madrid, and America/Los_Angeles, plus 13 boundary checks per zone. Checks covered expected successes/failures, retry caps, deadlines, changing Retry-After headers, wall-clock jumps, late timers, and final errors/request IDs. The fake-timer cases left no pending timers.

Response handling: 1,456 document/error/data/status combinations and 16 cases using real GraphQL execution and the GraphQL client. Coverage included nullable and partial reads, mutation data guards, mixed errors, and invalid/ambiguous/batch documents.

Refilling quota: a separate model with capacity 32 and refill rate 32/second completed all 1,024 operations across four batches.

HTTPS: the public GraphQL client used a loopback HTTPS server, real timers, and default request behavior. Variables, tokens, and final request IDs stayed correct. There were no unhandled errors, open sockets after cleanup, or further requests during the 250 ms observation window after settlement.

Scope: quota and timing probes used simulated time; HTTPS used real local transport. No Shopify API quota, stores, or credentials were used. Deadline checks cover the outer throttle retry budget, not a total request timeout. These temporary probes are separate from committed unit tests.

Separate custom-option edge case

Forcing defaultDelayMs: 0 and limitRetriesTo: 1100 produced NaN delays starting at retry index 1024. This was recorded as a failure and excluded from the passing counts.

The sole production caller uses the default limit of ten retries and cannot reach this case.

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact
  • The change is user-facing — I've identified the correct bump type (patch) and added a changeset

@github-actions github-actions Bot added the Area: @shopify/cli @shopify/cli package issues label Oct 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

  • Some seemingly private modules might be re-exported through public modules.
  • If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/private/node/api.d.ts
@@ -60,6 +60,8 @@ export declare function simpleRequestWithDebugLog<T extends {
  * Makes a HTTP request to some API, retrying if response headers indicate a retryable error.
  *
  * If a request fails with a 429, the retry-after header determines a delay before an automatic retry is performed.
+ * Without a valid header, rate-limit retries use exponential backoff with jitter, capped at 8 seconds per delay.
+ * No new rate-limit retry starts after 30 seconds. A server-requested delay beyond that budget is not shortened.
  *
  * If unauthorizedHandler is provided, then it will be called in the case of a 401 and a retry performed. This allows
  * for a token refresh for instance.

@isaacroldan
isaacroldan marked this pull request as ready for review October 20, 2026 16:35
@isaacroldan
isaacroldan requested a review from a team as a code owner October 20, 2026 16:35
@isaacroldan
isaacroldan added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: @shopify/cli @shopify/cli package issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants