Skip to content

Stop the EKS executor concurrency test from starving its own builders - #75197

Merged
vincbeck merged 1 commit into
apache:mainfrom
pankajastro:fix-eks-concurrent-builds-test-timeout
Oct 20, 2026
Merged

vincbeck merged 1 commit into
apache:mainfrom
pankajastro:fix-eks-concurrent-builds-test-timeout

Conversation

@pankajastro

@pankajastro pankajastro commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

test_concurrent_builds_never_change_sts_regional_endpoints_env_var (added in #74399) runs a watcher thread that polls os.environ while 8 workers build EKS clients. The watcher spun without ever yielding, so it held the GIL against the builders. The test took about 95 seconds locally and hit its 60 second timeout in the lowest-dependencies CI job (Low dep tests: providers, amazon), failing it on unrelated PRs. Failing runs:

The watcher now sleeps 1 ms between samples. The test takes about 1 second.

Checked locally:

  • Before: the test alone took 94-98 seconds. After: about 1.1 seconds, stable over 12 runs, about 1.6 seconds with 6 busy CPU processes running, and about 1.1 seconds with the lowest supported boto3/botocore (1.42.79).
  • The test still catches the bug it guards: a temporary write-and-restore of AWS_STS_REGIONAL_ENDPOINTS during a build and a write that is never restored both fail it through the env_writes assertion, and a write that bypasses os.environ.__setitem__ fails it through the sampling assertion (5 of 5 runs).
  • All 16 tests in test__client_factory.py pass.

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Sonnet 5.5)

Generated-by: Claude Code (Sonnet 5.5) following the guidelines

🤖 Generated with Claude Code

@boring-cyborg boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Oct 20, 2026
@pankajastro
pankajastro force-pushed the fix-eks-concurrent-builds-test-timeout branch from e5871aa to a3090ba Compare October 20, 2026 10:09
The watcher thread spun on os.environ.get without yielding, holding the GIL
against the client-building threads. The test took about 95 seconds locally
and exceeded its 60 second timeout in the lowest-dependencies CI job.

Co-Authored-By: Claude <noreply@anthropic.com>
@pankajastro
pankajastro force-pushed the fix-eks-concurrent-builds-test-timeout branch from a3090ba to 41c5414 Compare October 20, 2026 10:26
@vincbeck

Copy link
Copy Markdown
Contributor

@seanghaeli

@vincbeck
vincbeck merged commit b0cfa1e into apache:main Oct 20, 2026
71 checks passed
@pankajastro
pankajastro deleted the fix-eks-concurrent-builds-test-timeout branch October 20, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants