KubernetesPodOperator task is marked SUCCESS when the task process receives SIGTERM mid-execute (pod deleted by on_kill, cleanup() short-circuits on _killed)

Under which category would you file this issue?

Providers

Apache Airflow version

3.3.1

What happened and how to reproduce it?

Issue Description

When the task process running a KubernetesPodOperator receives SIGTERM while execute() is in flight, the task instance is committed as success even though the operator's pod was destroyed mid-run and its work never completed.

The two interacting pieces:

(a) SIGTERM invokes on_kill() but does not fail the task.
airflow/sdk/execution_time/task_runner.py:1544:

def _on_term(signum, frame):
    pid = os.getpid()
    if pid != parent_pid:
        return

    ti.task.on_kill()

signal.signal(signal.SIGTERM, _on_term)

The handler calls on_kill() and returns. It does not raise, does not set a terminating flag, and does not exit — so execution resumes exactly where it was interrupted, inside KubernetesPodOperator.execute(). The except AirflowTaskTerminated branch at task_runner.py:1655 carries the comment "these exceptions should ideally never be thrown", and on this path nothing throws it.

(b) KubernetesPodOperator.on_kill() destroys the work, then cleanup() silently skips the failure raise.

providers/cncf/kubernetes/operators/pod.py:1556:

def on_kill(self) -> None:
    self._killed = True
    ...
    self.client.delete_namespaced_pod(**kwargs)   # deletes the pod doing the actual work

providers/cncf/kubernetes/operators/pod.py:1339:

def cleanup(self, pod, remote_pod, xcom_result=None, context=None) -> None:
    # Skip cleaning the pod in the following scenarios.
    # 1. If a task got marked as failed, "on_kill" method would be called ...
    # 2. remote pod is null (ex: pod creation failed)
    if self._killed or not remote_pod:
        return

cleanup() is the only thing that raises AirflowException for a pod that did not reach Succeeded. The _killed guard (introduced in d3b4a91, "fix: Avoid retrying after KubernetesPodOperator has been marked as failed (#36749)") skips it.

That guard's premise — _killed means the TI is already being marked failed elsewhere — holds when a user marks a task failed in the UI. It does not hold when the SIGTERM comes from a pod eviction or any other external termination, because in that case nothing else marks the TI failed.

Net result: execute_sync() returns None normally, no exception ever reaches _run_task_and_map_outcome, and the task is committed success with a normal end_date.

Steps to reproduce

Minimal form — does not require an eviction, any executor that runs the task in its own process will do:

Define a DAG with a single KubernetesPodOperator running a long container:

KubernetesPodOperator(
    task_id="sleeper",
    image="alpine:3",
    cmds=["sh", "-c", "sleep 600 && echo done"],
    on_finish_action="delete_pod",   # the default
    get_logs=True,
)

Trigger the DAG. Wait until the pod reaches Running and the operator is streaming its logs.
Send SIGTERM to the task process (not the pod):
```
kill -TERM <pid of the airflow task-runner process>
```
Equivalently, under KubernetesExecutor: kubectl delete pod <the KE worker pod>, or cordon/drain the node it is on — anything that delivers a graceful SIGTERM to the worker.
Observe:
- the alpine pod is deleted immediately by on_kill(), having never printed done;
- the task instance is committed success.

How it shows up in production

In our deployment this fires naturally roughly 0.73×/day (33 occurrences over 45 days of task_instance retention). The trigger is an EKS managed-node-group rolling drain: eks:node-manager issues create pods/eviction against the KubernetesExecutor worker pod. From the EKS control-plane audit log:

23:49:43.038  create  pods/eviction  <ke-worker-pod>       user=eks:node-manager
23:49:43.048  delete  pods           <kpo-child-pod>       user=<airflow-worker SA>   # on_kill(), 10 ms later
23:49:51.052  delete  pods           <ke-worker-pod>       user=system:node:<node>
23:49:52.071  patch   pods           <ke-worker-pod>       user=<scheduler SA>  rc=404  (x3)

The task log fingerprint is distinctive and is the easiest way for others to recognize this. Every non-container line from one affected attempt, in full:

23:50:35.863  Pod has reached Running phase before launch timeout
23:54:52.093  ::group::Post Execute
23:54:52.112  ::endgroup::

Four minutes of container log streaming, then straight to Post Execute — approximately 20–30 ms after the final container log line. Note what is missing:

no Pod %s has phase %s — PodManager.await_pod_completion logs this on every non-terminal poll, so its absence shows the loop exited on the first read;
no Deleting pod: %s and no Skipping deleting pod: %s — proving cleanup() returned at the _killed guard before reaching process_pod_deletion;
no exception, no traceback, no warning.

A healthy run of the same task always logs both Pod ... has phase Running and Deleting pod: ... before Post Execute.

The work loss is real and varies with where the kill lands: in one case the container had written 6 of its 8 output files and was mid-upload of the 7th; in others it had written none. Duration of the false-success attempt ranged from 11% to 228% of the same task's eventual successful runtime, consistent with a kill at a uniformly random point.

What you think should happen instead?

The task should be marked failed (and therefore be eligible for its configured retries), not success. Silently reporting success for a task whose pod was destroyed mid-run means downstream tasks consume incomplete output with no signal at all — the failure is undetectable unless the DAG happens to independently verify the operator's side effects.

Two candidate fixes; the second is broader and probably the more correct one:

(a) Provider — narrow the _killed guard in cleanup(). Only skip the failure raise when the pod actually succeeded, e.g.:

if not remote_pod:
    return
if self._killed and remote_pod.status.phase == PodPhase.SUCCEEDED:
    return

This preserves the intent of #36749 (don't re-cleanup / don't trigger a retry for a task the user deliberately marked failed) while restoring the failure signal when the pod did not succeed. It would need care so that a UI-initiated "mark failed" does not start producing unwanted retries — that is exactly the regression #36749 fixed.

(b) Task SDK — make _on_term actually terminate the task. After ti.task.on_kill(), _on_term could raise AirflowTaskTerminated (or set a flag checked immediately after execute() returns) so the existing except AirflowTaskTerminated handler at task_runner.py:1655 fires and the TI is marked FAILED. That handler's own comment says such exceptions "should ideally never be thrown" — but on the SIGTERM path there is currently no mechanism that throws one, so a signal-terminated task can complete as success for any operator whose on_kill() tears down out-of-process work, not just KubernetesPodOperator.

Either fix alone resolves the KPO case. (b) additionally covers other operators with the same shape.

Operating System

Ubuntu 24.04.5 LTS (container image, Python 3.12.14)

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

amazon, celery, cncf-kubernetes, google, standard, pagerduty, http

Versions of Apache Airflow Providers

apache-airflow==3.3.1
apache-airflow-core==3.3.1
apache-airflow-task-sdk==1.3.1
apache-airflow-providers-cncf-kubernetes==10.21.1
apache-airflow-providers-celery==3.23.1
apache-airflow-providers-standard==1.18.0

Official Helm Chart version

1.22.0 (latest released)

Kubernetes Version

1.34

Helm Chart configuration

executor: "KubernetesExecutor,CeleryExecutor"

config:
  kubernetes_executor:
    delete_worker_pods: "False"
    delete_worker_pods_on_failure: "False"

workers:
  labels:
    executor: kubernetes
  podAnnotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

Relevant only as context for how the SIGTERM arrives — the bug reproduces regardless of these settings. Note that safe-to-evict: "false" is honored by cluster-autoscaler only; an Eviction API call from the cloud provider's node manager is not affected by it, and there is no PodDisruptionBudget on KubernetesExecutor worker pods.

Docker Image customizations

No response

Anything else?

Possibly related, but distinct:

The retry mechanism in KubernetesPodOperator doesn't handle pod evictions. #53015 — retry/reattach behavior after a pod eviction (evicted pod reused via label selector). Different failure; that one at least fails loudly.
KubernetesPodOperator Mark Task Failed after SUCCESS Randomly #21420 / discussion KubernetesPodOperator Mark Task Failed after SUCCESS Randomly #21558 — tasks marked SUCCESS then immediately FAILED under memory pressure.
discussion KubernetesPodOperator finishes despite marked as failed #45830 — a UI-marked-failed KPO task turning green when its pod later completes.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

@adrianrego

Hi @adrianrego, I’d like to work on this issue.

I can reproduce the reasoning behind the failure path where SIGTERM triggers on_kill(), the pod is deleted, and cleanup() returns early because _killed is set, allowing the task to be marked successful.

I’m planning to investigate the existing behavior and tests around _killed and #36749 first, then add regression coverage and implement the narrowest fix that preserves the existing retry behavior.

If that approach sounds good, I’d be happy to submit a PR.

@AbarnaaSree

@AbarnaaSree you mentioned picking this up on 12 Sep — there was no PR after eight days, so I went ahead and opened #73395. Happy to hand it back or have you review it.

I fixed it in the task runner rather than in cleanup(): AirflowTaskTerminated is caught in _run_task_and_map_outcome but grepping the tree shows nothing ever raises it, so the raise that Airflow 2 did after on_kill() is simply missing. Putting it back covers any operator whose on_kill tears down the work in flight, not only KubernetesPodOperator. It only fires while the task body is running, so the supervisor's task_success_overtime kill still cannot turn a finished task into a failed one.

A fix for this is up in #73395 — it's temporarily closed due to Apache Airflow's new open-PR-limit policy, combined with a GitHub-side bug that's currently blocking me from reopening it. Flagging here so it doesn't get duplicated while that gets sorted out.

Drafted-by: Claude Code (Sonnet 5)

Fail a task whose execute is interrupted by SIGTERM

@arkadiuszbach

My merged PR #69034 addresses the worker-shutdown path discussed in this issue and #71202: when SIGTERM reaches the supervisor, it waits for the running task instead of forwarding SIGTERM and triggering the operator's on_kill(). For KPO, this allows the operator to continue monitoring its application pod and report its outcome.

I verified this with Airflow 3.3.1 / Task SDK 1.3.1 / Kubernetes provider 10.21.1, applying only #69034's SDK changes and its corresponding worker configuration (DUMB_INIT_SETSID=0; baseline: 1). All 441 installed package versions remained identical. In the unpatched control, deleting only the worker produced a false SUCCESS before the application completed.

The following tests used KubernetesExecutor and synchronous KPO on local kind. Worker means the pod running the supervisor and KPO; application means the separate pod created by KPO. The workload was alpine:3, running sh -c 'sleep 600 && echo done'. We initiated deletion of the selected pod(s) using two shutdown budgets:

700 seconds: graceful shutdown with enough time to finish. Kubernetes sends SIGTERM and honours the configured grace period. We deliberately allow enough time for the 600-second application workload to complete and for the worker to report its outcome. The selected pod(s) finish before the deadline, so SIGKILL is not needed.
30 seconds: insufficient time, followed by forced termination. Kubernetes sends SIGTERM, then sends SIGKILL to containers still running when the 30-second grace period expires. This simulates the effect of a short shutdown window, such as Spot reclamation or another event that terminates a node before its workloads can finish.

These are simulation parameters. The 30-second deadline is not a universal Spot or EKS managed-node-group limit; actual shutdown windows depend on the provider, configuration, and event, as described below. In both groups, Kubernetes waits within the applicable grace period; the difference is whether the workload can finish before that period expires.

Pod(s) selected for termination	Grace	Worker pod: phase / exit	Application pod: phase / exit	Airflow task state
Worker only	700 s	Succeeded / 0	Succeeded / 0	SUCCESS, after `done`
Application only	700 s	Succeeded / 0	Succeeded / 0	SUCCESS, after `done`
Both	700 s	Succeeded / 0	Succeeded / 0	SUCCESS, after `done`
Worker only	30 s	Failed / 137	Still Running	UP_FOR_RETRY
Application only	30 s	Succeeded / 0	Failed / 137	UP_FOR_RETRY
Both	30 s	Failed / 137	Failed / 137	UP_FOR_RETRY

No false SUCCESS occurred in this patched matrix. The Dag allowed one retry; the table records first-attempt results. When the worker was killed, KubernetesExecutor observed the failed worker pod and the scheduler handled the failure; this did not depend on the dead KPO detecting anything or waiting for a heartbeat timeout. When only the application was killed, KPO detected its failure. A worker pod exiting 0 therefore does not necessarily mean the Airflow task succeeded.

The application continued after SIGTERM in this particular image/command. That is not guaranteed for arbitrary applications: two additional tests used an application that handled SIGTERM and exited 143, and the task entered UP_FOR_RETRY before the 30-second deadline. #69034 does not add signal handling to code inside the application pod. An application that exits 0 after abandoning unfinished work can still appear successful.

I also ran both worker-killed scenarios through an actual retry, with a 10-second retry delay and the default durable=True / reattachment behavior. In the worker-only termination test, the retry reattached to the existing application pod. When both pods were terminated, the retry started a new application pod after the original container had exited with code 137 (SIGKILL). The second task attempt succeeded in both tests. We monitored the pods throughout both attempts and observed no simultaneous execution of two application containers within either test.

There are also limits to the fix:

The kill -TERM <task-runner PID> reproduction in this issue sends the signal directly to the task subprocess, bypassing the supervisor. To verify worker shutdown with Warm-shutdown supervisor on SIGTERM instead of killing the running task #69034, initiate termination of the worker pod so the signal reaches the supervisor through the configured entrypoint.
SIGKILL or loss of the VM can still interrupt the worker and leave its application running without an active worker. The successful reattachment verified above depends on the tested configuration and on the existing pod being discoverable.
Mixed placement, such as a Spot worker and an on-demand application pod, can produce the asymmetric cases above. Each pod has its own shutdown budget; these six cases are not an exhaustive test of cloud events or signal ordering.

cluster-autoscaler.kubernetes.io/safe-to-evict: "false" can prevent Cluster Autoscaler scale-down of a node hosting the pod; it does not prevent Spot reclamation or every managed-node maintenance event. Setting it to "true" allows Cluster Autoscaler to consider eviction but does not select a shutdown timeout. See the Cluster Autoscaler FAQ.

These tests ran on a local Kubernetes cluster without a Cluster Autoscaler or Spot instances. We explicitly deleted pods with a 700-second grace period to allow the workload to finish, or a 30-second grace period to force SIGKILL before completion. The latter simulates the effect of a short shutdown window, such as during a Spot interruption or forced node removal. Actual shutdown limits depend on the provider, configuration, and event.

The available time depends on the provider, node configuration, Kubernetes version, and event:

AWS/EKS: EC2 Spot stop/termination has a best-effort two-minute interruption notice, not two guaranteed minutes for each pod. EKS warns that pods may receive less time or no termination signal. Managed-node-group desired-capacity reductions and AZRebalance have a separate 15-minute eviction limit before node termination; an ASG lifecycle hook can extend eligible terminations, but cannot extend Spot reclamation. See Spot notices, managed node groups, and ASG termination limits. For managed-node-group version/AMI updates, the 15-minute drain timeout instead fails an unforced update with PodEvictionFailure; it is not a universal hard termination deadline. See EKS update behavior.
GKE: Default Spot shutdown is 30 seconds total, split into 15 seconds for ordinary pods and 15 for critical system pods. Current documentation allows up to 120 seconds of total node shutdown time through node configuration on supported versions (Standard control plane >= 1.35.0-gke.1171000); ordinary pods receive the total node budget minus the critical-pod allocation. Separately, autoscaler scale-down allows one hour from 1.32.7-gke.1079000, versus 10 minutes on older versions. See GKE Spot shutdown and GKE autoscaling.
Azure/AKS: Spot eviction notifications are best effort, up to 30 seconds; AKS also attempts automatic cordon/drain on Preempt. Autoscaler scale-down separately defaults to 600 seconds through the configurable max-graceful-termination-sec profile setting. See Azure Spot eviction, AKS auto-drain, and AKS autoscaling. Rolling upgrades have a different drain timeout: 30 minutes by default, configurable from 5 minutes to 24 hours; expiry with pods remaining stops the upgrade. See AKS upgrade draining.

During normal graceful pod termination, Kubernetes honours the configured terminationGracePeriodSeconds. Configure it separately for the Airflow worker and KPO application pod, allowing enough time for the remaining work and result reporting. The default is 30 seconds, and preStop consumes part of this budget. See Kubernetes pod termination. With #69034, the worker can continue monitoring the running task during this period. If the underlying node is terminated earlier, the available time is constrained by that event, such as the Spot and managed-node-group termination limits described above.

Could you test by applying #69034's Task SDK changes to your existing test deployment and setting DUMB_INIT_SETSID=0 in the actual KubernetesExecutor worker pod template, then restarting the affected components? You do not need to build from main or upgrade all packages to test the backport. Please include the node-removal event, executor/provider versions, and both pod grace periods when reporting the result.

I am also considering adding integration tests to Airflow's test suite to cover these six termination scenarios.

Drafted-by: Codex (GPT-6); reviewed by @arkadiuszbach before posting

Navigation Menu

Repository navigation

KubernetesPodOperator task is marked SUCCESS when the task process receives SIGTERM mid-execute (pod deleted by on_kill, cleanup() short-circuits on _killed) #73006

Under which category would you file this issue?

Apache Airflow version

What happened and how to reproduce it?

Issue Description

Steps to reproduce

How it shows up in production

What you think should happen instead?

Operating System

Deployment

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Official Helm Chart version

Kubernetes Version

Helm Chart configuration

Docker Image customizations

Anything else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

KubernetesPodOperator task is marked SUCCESS when the task process receives SIGTERM mid-execute (pod deleted by on_kill, cleanup() short-circuits on _killed) #73006

Description

Under which category would you file this issue?

Apache Airflow version

What happened and how to reproduce it?

Issue Description

Steps to reproduce

How it shows up in production

What you think should happen instead?

Operating System

Deployment

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Official Helm Chart version

Kubernetes Version

Helm Chart configuration

Docker Image customizations

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

AbarnaaSree commented on Sep 12, 2026

namanjain24-sudo commented on Sep 20, 2026

namanjain24-sudo commented on Sep 25, 2026

arkadiuszbach commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions