Add AwsEksExecutor to the Amazon provider by seanghaeli · Pull Request #74399 · apache/airflow

seanghaeli · 2026-10-07T08:24:47Z

Adds AwsEksExecutor, which runs each task as a pod on an Amazon EKS cluster. You select it with executor = airflow.providers.amazon.aws.executors.eks.AwsEksExecutor. It subclasses the Kubernetes executor and only replaces authentication. The Kubernetes client is built from an Airflow AWS connection with no kubeconfig, and the short-lived EKS token is re-minted in-process shortly before it expires. That avoids the subprocess-per-refresh path that has produced repeated credential-refresh bugs (#55195, #58743, #65212, #65335, #74306). On startup the executor checks that the cluster is usable and (unless check_health_on_startup is off) that it can list pods, so a missing access entry fails right away instead of showing up later as a watcher error loop. Settings live in a new [aws_eks_executor] section, and there's a new docs page.

The executor plugs its client builders into the client_factory and async_client_factory settings from #73014 itself, so users don't set those. If the installed cncf.kubernetes doesn't have that option, the executor raises an AirflowOptionalProviderFeatureException on import, naming the version it needs.

Because those settings are shared, a plain KubernetesExecutor in the same scope (both global, or the same team) would pick up the EKS client and send its pods to the EKS cluster. So the executor refuses to start in that setup and says how to fix the config. A global KubernetesExecutor next to a team AwsEksExecutor works fine. Letting a subclass bring its own client without going through the shared setting is tracked in #74844.

To mint the token with the connection's credentials, fetch_access_token_for_cluster now takes an optional session argument. The executor passes in the session from its EksHook. The argument defaults to None, which keeps today's behavior, so existing callers are unaffected. It no longer sets AWS_STS_REGIONAL_ENDPOINTS in-process, since botocore has defaulted to the regional endpoint since 1.40; only the eks_get_token command-line entry point still sets it, in its own short-lived process.

MIN_CNCF_KUBERNETES_VERSION is set to 10.24.0, the expected next cncf.kubernetes minor carrying #73014.

Testing

Unit tests in Breeze: executors/eks/, utils/test_eks_get_token.py and EksHook.generate_config_file: 58 passed.

System tests: On the system test dashboard, using this PR's branch, 60/60 system tests succeed using worker pods on a real EKS cluster.

Before, with the stock KubernetesExecutor:

kubernetes.config.config_exception.ConfigException: Service host/port is not set.
RUN e2e_first state=scheduler_exited

After, with AwsEksExecutor and async_pod_creation = True. The second run comes from the same scheduler process, after the first token has expired:

EKS Executor health check has succeeded.
RUN e2e_first state=success elapsed_since_scheduler_start=40s
scheduler pid 89 still alive at 1090s, triggering e2e_after_token_expiry
RUN e2e_after_token_expiry state=success elapsed_since_scheduler_start=1111s

With executor = AwsEksExecutor,KubernetesExecutor, where the Kubernetes executor has its own kubeconfig. Before the guard, the Kubernetes executor never used its kubeconfig and silently ran on the EKS client:

KE exec plugin calls: 0

After, the scheduler refuses to start:

ValueError: AwsEksExecutor can't run alongside KubernetesExecutor in the global scope. AwsEksExecutor sets the [kubernetes_executor] client factory for the global scope, so KubernetesExecutor would silently send its pods to the EKS cluster. Configure the two executors for different teams, or don't use them together. See https://github.com/apache/airflow/issues/74844

And executor = KubernetesExecutor;team_a=AwsEksExecutor still starts, with the Kubernetes executor on its own kubeconfig:

EKS Executor health check has succeeded.
RUN e2e_first state=success elapsed_since_scheduler_start=40s
KE exec plugin calls: 2

Docs

EKS executor docs page

Multi-team deployments, including the limit on running it next to a Kubernetes executor in the same scope

EKS executor config options

Amazon executors page (the "Choosing an executor" table is gone)

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Code (claude-fable-5) following the guidelines

Callers that already hold a boto3 session (with its credentials and region) can pass it in instead of having the helper build a new one from the environment.

A KubernetesExecutor subclass that runs tasks as pods on an Amazon EKS cluster. It supplies its own Kubernetes client through the cncf.kubernetes client_factory and async_client_factory settings, built from an Airflow AWS connection with the EKS token re-minted in-process, and checks cluster access at startup. The factories live in a private _client_factory module.

… clients kubernetes 35 has no alias kwarg on get_api_key_with_prefix and kubernetes_asyncio 32 has a synchronous auth_settings(), so assert through each client's own auth_settings() instead.

Add an eks/utils.py with CONFIG_GROUP_NAME, CONFIG_DEFAULTS and AllEksConfigKeys like ecs/batch/lambda, add check_health_on_startup and a public check_health() with the same startup flow and messages, order the config options like the other sections, and use the shared logging include and option-list style in the docs.

Checks that AllEksConfigKeys and CONFIG_DEFAULTS stay in step with the [aws_eks_executor] section in provider.yaml.

Raise AirflowOptionalProviderFeatureException naming the minimum cncf.kubernetes version at import time, for both a missing and a too-old provider, instead of checking it in the executor constructor, and drop the docs paragraph that described the check.

The executor accepts ACTIVE and UPDATING clusters, but the error for any other status only named ACTIVE.

kubernetes 36 reads the bearer token from the BearerToken api_key, older releases from authorization, and kubernetes_asyncio has always used BearerToken. Set the one key each client reads instead of both, so the branch can go once the minimum kubernetes version is 36.

Use Sphinx references for the config option, the set-config page and the multi-team page, link the boto3 credentials guide and the EKS update docs instead of describing them, and say what the startup check catches.

Use one message, naming only the minimum cncf.kubernetes version, for both the missing and the too-old provider, and inline the version since nothing else reads MIN_CNCF_KUBERNETES_VERSION.

Name the verbs on pods and pods/log, matching the Helm chart's pod-launcher role, in both the health check error and the 'Grant access to the cluster' docs step, and drop the docs sentence on AWS credential refresh, which works the same as for every Amazon hook.

… in page

Set the team-scoped [kubernetes_executor] client_factory and async_client_factory for a team executor, since cncf.kubernetes looks a team's factory up in the team's own config only. Teams share the cluster from [aws_eks_executor], as KubernetesExecutor teams share its default client, and get their own team-scoped pod settings.

The installed kubernetes version cannot change at runtime, so compute the api_key name for the bearer token at module level instead of on every client build, and test the version check directly.

Tell the user where to set cluster_name when it is missing, and what to do when the cluster is not in a usable state. Also correct the comment on refused cluster statuses, which left out PENDING.

The Batch executor refuses EKS jobs, so drop EKS from its compute options. An unset EKS region_name falls back to the AWS connection's region before boto3's. Worker pods are only deleted when their task succeeds, since failed pods are kept by default.

Link the [kubernetes_executor] and [aws_eks_executor] options, pod_override and pod templates to their reference docs, link the EKS access entry and aws-auth ConfigMap guides, and point the shared logging section at the logging docs with :doc: rather than hard-coded stable URLs.

The refresh hook runs before every Kubernetes API request. Like the kubernetes client's in-cluster config, only get a new token when the cached one is about to expire, a minute before the 15 minute lifetime (TOKEN_EXPIRATION_MINUTES), instead of signing a new one each time.

seanghaeli · 2026-10-14T07:23:29Z

As of today 60/60 system tests are successfully running on the AWS system test dashboard using the EksExecutor

seanghaeli · 2026-10-14T07:36:00Z

Using the Eks Executor depends on cncf.kubernetes >= 10.24.0 but I haven't bumped the version floor in this PR, and instead just throw an error if the user is on an older version of cncf. @vincbeck do you agree with this, or prefer that I bump the version floor?

^ Nevermind, question answered. I'm already following the approach Niko recommend (and you agreed with) here

#72542 (review)

seanghaeli · 2026-10-14T20:39:25Z

@eladkal Any thoughts? Now that it's working e2e, I'm happy to merge this

o-nikolas

This is a big one! Submitting my comments for the docs now. Will have to circle back later for the code and tests.

With no pod template, no pod_override image and no worker_container_repository/tag, the worker pod has no image and the API server rejects it with a bare 422. Check the built pod before it is sent and fail the task with an error naming the three ways to set one.

seanghaeli · 2026-10-15T22:03:41Z

Thanks @o-nikolas , addressed your docs comments!

The client factory deleted the variable after resolving the STS endpoint, dropping any value the user had set, and every token fetch then set it to "regional" for good. Both now set it only while they need it and put the previous value back.

…same scope AwsEksExecutor installs its client factory through the [kubernetes_executor] client_factory env vars, so a plain KubernetesExecutor in the same scope (both global, or the same team) picks it up and silently sends its pods to the EKS cluster. Fail at construction instead, with a message that says how to fix the config. A global KubernetesExecutor next to a team AwsEksExecutor is unaffected. The proper fix is tracked in apache#74844.

The executor reads [aws_eks_executor] globally, so a team's own cluster_name, region_name or conn_id would be silently ignored and its pods sent to the global cluster. Fail at construction until per-team clusters land (apache#74844).

Every API server log read builds a new client, and each one left another CA temp file behind. Reuse the file written for a CA, and remove it when the process exits, like the kubernetes kubeconfig loader does.

botocore has defaulted sts_regional_endpoints to "regional" since 1.40.0, and the amazon provider requires botocore 1.42.79 or newer, so setting AWS_STS_REGIONAL_ENDPOINTS while building the client is not needed. It was also unsafe: the env var is shared by the whole process, so threads building clients at the same time, as in the API server, could race on it. The eks_get_token CLI still sets it, since it runs in its own short-lived process.

…n the docs

Rendered from 35686a9; the Requirements section now includes the botocore note.

o-nikolas

Nice, work! Just a few more nits

Reverts bca7914. The check isn't specific to EKS, and overriding cncf's private _build_pod_request is brittle, so it moves to cncf instead. A pod with no image now fails with the API server's own error.

boring-cyborg Bot added area:providers provider:amazon AWS/Amazon - related issues labels Oct 7, 2026

seanghaeli marked this pull request as ready for review October 7, 2026 17:18

seanghaeli requested a review from o-nikolas as a code owner October 7, 2026 17:18

vincbeck reviewed Oct 7, 2026

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/utils/eks_get_token.py

seanghaeli added 4 commits October 7, 2026 21:24

Let fetch_access_token_for_cluster reuse a caller's boto3 session

93f6071

Callers that already hold a boto3 session (with its credentials and region) can pass it in instead of having the helper build a new one from the environment.

Add the [aws_eks_executor] config section and register AwsEksExecutor

19c6f4f

Add the AwsEksExecutor docs page

86701fb

seanghaeli force-pushed the amazon/eks-get-token-session branch from bb6fed2 to 86701fb Compare October 7, 2026 21:57

seanghaeli changed the title ~~Let fetch_access_token_for_cluster reuse a caller's boto3 session~~ Add AwsEksExecutor to the Amazon provider Oct 7, 2026

seanghaeli added 4 commits October 7, 2026 23:06

Make EKS client factory tests pass on the lowest supported kubernetes…

a03107d

… clients kubernetes 35 has no alias kwarg on get_api_key_with_prefix and kubernetes_asyncio 32 has a synchronous auth_settings(), so assert through each client's own auth_settings() instead.

Drop the TODO notes on the cncf.kubernetes version

bc18261

Add tests for the EKS executor config keys

a947f00

Checks that AllEksConfigKeys and CONFIG_DEFAULTS stay in step with the [aws_eks_executor] section in provider.yaml.

eladkal reviewed Oct 8, 2026

View reviewed changes

Comment thread providers/amazon/src/airflow/providers/amazon/aws/executors/eks/eks_executor.py

eladkal reviewed Oct 8, 2026

View reviewed changes

seanghaeli added 13 commits October 8, 2026 18:01

Say UPDATING is accepted in the EKS cluster status error

6eda87f

The executor accepts ACTIVE and UPDATING clusters, but the error for any other status only named ACTIVE.

Tighten the EKS executor docs

6f056d4

Use Sphinx references for the config option, the set-config page and the multi-team page, link the boto3 credentials guide and the EKS update docs instead of describing them, and say what the startup check catches.

Add a section on choosing between the Amazon executors

1a12b2b

Shorten the cncf.kubernetes requirement error in the EKS executor

ba16e65

Use one message, naming only the minimum cncf.kubernetes version, for both the missing and the too-old provider, and inline the version since nothing else reads MIN_CNCF_KUBERNETES_VERSION.

Simplify the EKS cluster status note and link check_health_on_startup…

b5361cc

… in page

Resolve the EKS executor's kubernetes token key once at import

698e1fc

The installed kubernetes version cannot change at runtime, so compute the api_key name for the bearer token at module level instead of on every client build, and test the version check directly.

Say how to fix EKS executor cluster errors

e0cd673

Tell the user where to set cluster_name when it is missing, and what to do when the cluster is not in a usable state. Also correct the comment on refused cluster statuses, which left out PENDING.

seanghaeli and others added 5 commits October 9, 2026 00:27

Shorten the EKS token refresh comment

68eee9e

Trim the EKS executor tests and comments

d632704

Tighten the EKS executor docs page

7c88348

Merge branch 'main' into amazon/eks-get-token-session

22508f4

Merge branch 'main' into amazon/eks-get-token-session

be164c5

o-nikolas reviewed Oct 15, 2026

View reviewed changes

seanghaeli added 2 commits October 15, 2026 21:43

Address docs review on the EKS executor pages

313255f

ferruzzi reviewed Oct 15, 2026

View reviewed changes

seanghaeli mentioned this pull request Oct 16, 2026

KubernetesExecutor: let each team connect to its own cluster #74844

Open

seanghaeli added a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Oct 16, 2026

Refresh the AwsEksExecutor docs screenshots for apache#74399

5502618

o-nikolas reviewed Oct 16, 2026

View reviewed changes

seanghaeli added 4 commits October 16, 2026 16:58

Write each EKS cluster CA file once per process

54e0e1d

Every API server log read builds a new client, and each one left another CA temp file behind. Reuse the file written for a CA, and remove it when the process exits, like the kubernetes kubeconfig loader does.

Explain the first EKS token fetch and note the botocore requirement i…

35686a9

…n the docs

seanghaeli added a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Oct 19, 2026

Refresh the AwsEksExecutor docs screenshots for apache#74399

dfafc8b

Rendered from 35686a9; the Requirements section now includes the botocore note.

o-nikolas approved these changes Oct 19, 2026

View reviewed changes

seanghaeli added 2 commits October 19, 2026 23:37

Mark the EKS executor config options as added in 9.39.0

5b146a2

Leave the missing-image check to the KubernetesExecutor

73190ef

Reverts bca7914. The check isn't specific to EKS, and overriding cncf's private _build_pod_request is brittle, so it moves to cncf instead. A pod with no image now fails with the API server's own error.

o-nikolas merged commit 3134ac2 into apache:main Oct 20, 2026
72 checks passed

o-nikolas deleted the amazon/eks-get-token-session branch October 20, 2026 01:02

pankajastro mentioned this pull request Oct 20, 2026

Stop the EKS executor concurrency test from starving its own builders #75197

Merged

1 task done

Conversation

seanghaeli commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Docs

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seanghaeli commented Oct 14, 2026

Uh oh!

seanghaeli commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seanghaeli commented Oct 14, 2026

Uh oh!

o-nikolas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seanghaeli commented Oct 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

o-nikolas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

seanghaeli commented Oct 7, 2026 •

edited

Loading

seanghaeli commented Oct 14, 2026 •

edited

Loading