KubernetesPodOperator task is incorrectly marked as SUCCESS instead of FAILED when both the task pod and its child pod are interrupted in close succession

Under which category would you file this issue?

Providers

Apache Airflow version

3.3.0

What happened and how to reproduce it?

Architecture context

This happens when using KubernetesExecutor together with KubernetesPodOperator, which involves two separate pods:

The per-task pod created by KubernetesExecutor to execute the task instance.
The child pod created by KubernetesPodOperator.execute() to run the user's workload.

Issue Description

When both pods are interrupted within roughly one second of each other, the task instance is sometimes finalized as success, even though the child pod never completed its work.

Steps to reproduce

Trigger a DAG task using KubernetesPodOperator whose child pod runs long enough to still be running when interrupted. Example:

from datetime import datetime
from airflow import DAG
from airflow.models import Variable
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

default_args = {
    "owner": "data",
    "depends_on_past": False,
    "start_date": datetime.strptime("2026-08-05", "%Y-%m-%d"),
    "params": {
        "severity": 0,
    },
    "retries": 1,
}

dag = DAG(
    dag_id="air-test-sigterm-repro",
    default_args=default_args,
    catchup=False,
    schedule=None,
    max_active_runs=1
)

sleep_worker = KubernetesPodOperator(
    dag=dag,
    task_id="sleep-worker",
    name="sleep-worker",
    namespace=Variable.get("var-k8s-namespace"),
    image="python:3.9",
    cmds=["python", "-c"],
    do_xcom_push=False,
    get_logs=True,
    is_delete_operator_pod=True,
    arguments=[
        "import time\n"
        "for i in range(120):\n"
        "    print('tick', i, 'still running, still streaming logs', flush=True)\n"
        "    time.sleep(5)\n"
        "print('finished without interruption', flush=True)\n"
    ],
)

While the task is running, delete both the child pod and the per-task pod within the same short time window (rather than waiting for one deletion to complete). The most reliable sequence we found is to delete the child pod first, then the per-task pod about one second later.

Terminal 1 — child pod:
```
kubectl delete pod sleep-worker-xxxxxxx -n airflow --grace-period=30
```
Terminal 2 (~1 second later) — per-task pod:
```
kubectl delete pod air-test-sigterm-repro-sleep-worker-xxxxxxxxx -n airflow --grace-period=30
```
Observe the task's final state in the Airflow UI or API.

logs sleep-worker-xxxxxxx

tick 0 still running, still streaming logs
tick 1 still running, still streaming logs
tick 2 still running, still streaming logs
tick 3 still running, still streaming logs
tick 4 still running, still streaming logs
tick 5 still running, still streaming logs
tick 6 still running, still streaming logs
tick 7 still running, still streaming logs
tick 8 still running, still streaming logs
tick 9 still running, still streaming logs
tick 10 still running, still streaming logs
tick 11 still running, still streaming logs
tick 12 still running, still streaming logs
tick 13 still running, still streaming logs
tick 14 still running, still streaming logs
tick 15 still running, still streaming logs
tick 16 still running, still streaming logs
tick 17 still running, still streaming logs

logs air-test-sigterm-repro-sleep-worker-xxxxxxxxx

{"timestamp":"2026-08-05T18:49:28.622020Z","level":"info","event":"::endgroup::","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":186}
{"timestamp":"2026-08-05T18:49:29.219279Z","level":"info","event":"[base] tick 0 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:34.224101Z","level":"info","event":"[base] tick 1 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:39.229335Z","level":"info","event":"[base] tick 2 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:44.230741Z","level":"info","event":"[base] tick 3 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:49.235802Z","level":"info","event":"[base] tick 4 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:54.240729Z","level":"info","event":"[base] tick 5 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:59.245828Z","level":"info","event":"[base] tick 6 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:04.247038Z","level":"info","event":"[base] tick 7 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:09.247528Z","level":"info","event":"[base] tick 8 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:14.250719Z","level":"info","event":"[base] tick 9 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:19.255868Z","level":"info","event":"[base] tick 10 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:24.256810Z","level":"info","event":"[base] tick 11 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:29.261832Z","level":"info","event":"[base] tick 12 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:34.266976Z","level":"info","event":"[base] tick 13 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:39.272003Z","level":"info","event":"[base] tick 14 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:44.277099Z","level":"info","event":"[base] tick 15 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:49.282109Z","level":"info","event":"[base] tick 16 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:54.286787Z","level":"info","event":"[base] tick 17 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:59.288915Z","level":"info","event":"[base] tick 18 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:04.293390Z","level":"info","event":"[base] tick 19 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:09.296800Z","level":"info","event":"[base] tick 20 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:14.301846Z","level":"info","event":"[base] tick 21 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:19.306809Z","level":"info","event":"[base] tick 22 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:24.311349Z","level":"info","event":"[base] tick 23 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:29.315332Z","level":"info","event":"[base] tick 24 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:34.316805Z","level":"info","event":"[base] tick 25 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:36.975521Z","level":"info","event":"Received signal, forwarding to task subprocess","signal":"SIGTERM","pid":13,"logger":"supervisor","filename":"supervisor.py","lineno":1403}
{"timestamp":"2026-08-05T18:51:39.344660Z","level":"info","event":"[base] tick 26 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:44.322040Z","level":"info","event":"[base] tick 27 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}

Deleting only one of the two pods did not reproduce the issue in our testing. In those cases, the task correctly transitioned to failed or up_for_retry.

What we observe when it reproduces

The last log emitted by the per-task pod is:

{"event":"Received signal, forwarding to task subprocess","signal":"SIGTERM","pid":13,"logger":"supervisor","filename":"supervisor.py","lineno":1403}

After that, we never see:

on_kill() being invoked.
The provider's "Deleting pod: ..." log line.
Any exception or traceback.
Any further output from the child pod.

Despite the child pod being interrupted before completing its work, the task is later reported by the API server as success.

Observed in production

We observed this four times across two different DAGs over four days. Every occurrence coincided with our autoscaler reclaiming a Spot/preemptible node hosting one or both pods.

When the issue reproduces:

19:12:28Z  [child pod progress log — task actively running]
19:12:42Z  <node hosting the per-task pod begins draining>
19:12:44Z  {"event":"Received signal, forwarding to task subprocess","signal":"SIGTERM","pid":13,"logger":"supervisor"}
           <nothing else logged>
           --> Task instance later shows state = success

A normal execution of the same task instead includes:

{"event":"Deleting pod: <pod-name>", ...}
{"event":"::group::Post Execute", ...}
{"event":"Workload finished", "final_state":"success", ...}

Contrast: interrupting only the child pod

If only the child pod is interrupted while the per-task pod continues running, the task correctly fails. We've observed both AirflowException (PodFailed) and NotFoundException (404 after pod deletion), but in both cases the task is never reported as success.

What you think should happen instead?

The task should end in failed (or up_for_retry), just as it does when only the child pod is interrupted. Reporting success for a task whose workload never completed is a correctness issue, downstream tasks proceed as if the work succeeded, on_failure_callback is never triggered, and no retry occurs.

We haven't identified the root cause and are only reporting the observed, reproducible behavior. This may be related to #58936 (fixed by #61627 in 3.3.0), which changed how SIGTERM is forwarded from the supervisor to the task subprocess. The last log we consistently see ("Received signal, forwarding to task subprocess") originates from that change, although we can't confirm whether this is a regression from that fix or a separate issue.

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

cncf-kubernetes

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==10.19.0

Official Helm Chart version

1.18.0

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

I've observed this behavior as well on our production clusters.

fix: fail KubernetesPodOperator when on_kill interrupts the child pod

@arkadiuszbach

My merged PR #69034 addresses the worker-shutdown path discussed in this issue and #73006: when SIGTERM reaches the supervisor, it waits for the running task instead of forwarding SIGTERM and triggering the operator's on_kill(). For KPO, this allows the operator to continue monitoring its application pod and report its outcome.

I verified this with Airflow 3.3.1 / Task SDK 1.3.1 / Kubernetes provider 10.21.1, applying only #69034's SDK changes and its corresponding worker configuration (DUMB_INIT_SETSID=0; baseline: 1). All 441 installed package versions remained identical. In the unpatched control, deleting only the worker produced a false SUCCESS before the application completed.

The following tests used KubernetesExecutor and synchronous KPO on local kind. Worker means the pod running the supervisor and KPO; application means the separate pod created by KPO. The workload was alpine:3, running sh -c 'sleep 600 && echo done'. We initiated deletion of the selected pod(s) using two shutdown budgets:

700 seconds: graceful shutdown with enough time to finish. Kubernetes sends SIGTERM and honours the configured grace period. We deliberately allow enough time for the 600-second application workload to complete and for the worker to report its outcome. The selected pod(s) finish before the deadline, so SIGKILL is not needed.
30 seconds: insufficient time, followed by forced termination. Kubernetes sends SIGTERM, then sends SIGKILL to containers still running when the 30-second grace period expires. This simulates the effect of a short shutdown window, such as Spot reclamation or another event that terminates a node before its workloads can finish.

These are simulation parameters. The 30-second deadline is not a universal Spot or EKS managed-node-group limit; actual shutdown windows depend on the provider, configuration, and event, as described below. In both groups, Kubernetes waits within the applicable grace period; the difference is whether the workload can finish before that period expires.

Pod(s) selected for termination	Grace	Worker pod: phase / exit	Application pod: phase / exit	Airflow task state
Worker only	700 s	Succeeded / 0	Succeeded / 0	SUCCESS, after `done`
Application only	700 s	Succeeded / 0	Succeeded / 0	SUCCESS, after `done`
Both	700 s	Succeeded / 0	Succeeded / 0	SUCCESS, after `done`
Worker only	30 s	Failed / 137	Still Running	UP_FOR_RETRY
Application only	30 s	Succeeded / 0	Failed / 137	UP_FOR_RETRY
Both	30 s	Failed / 137	Failed / 137	UP_FOR_RETRY

No false SUCCESS occurred in this patched matrix. The Dag allowed one retry; the table records first-attempt results. When the worker was killed, KubernetesExecutor observed the failed worker pod and the scheduler handled the failure; this did not depend on the dead KPO detecting anything or waiting for a heartbeat timeout. When only the application was killed, KPO detected its failure. A worker pod exiting 0 therefore does not necessarily mean the Airflow task succeeded.

The application continued after SIGTERM in this particular image/command. That is not guaranteed for arbitrary applications: two additional tests used an application that handled SIGTERM and exited 143, and the task entered UP_FOR_RETRY before the 30-second deadline. #69034 does not add signal handling to code inside the application pod. An application that exits 0 after abandoning unfinished work can still appear successful.

I also ran both worker-killed scenarios through an actual retry, with a 10-second retry delay and the default durable=True / reattachment behavior. In the worker-only termination test, the retry reattached to the existing application pod. When both pods were terminated, the retry started a new application pod after the original container had exited with code 137 (SIGKILL). The second task attempt succeeded in both tests. We monitored the pods throughout both attempts and observed no simultaneous execution of two application containers within either test.

There are also limits to the fix:

The kill -TERM <task-runner PID> reproduction in KubernetesPodOperator task is marked SUCCESS when the task process receives SIGTERM mid-execute (pod deleted by on_kill, cleanup() short-circuits on _killed) #73006 sends the signal directly to the task subprocess, bypassing the supervisor. To verify worker shutdown with Warm-shutdown supervisor on SIGTERM instead of killing the running task #69034, initiate termination of the worker pod so the signal reaches the supervisor through the configured entrypoint.
SIGKILL or loss of the VM can still interrupt the worker and leave its application running without an active worker. The successful reattachment verified above depends on the tested configuration and on the existing pod being discoverable.
Mixed placement, such as a Spot worker and an on-demand application pod, can produce the asymmetric cases above. Each pod has its own shutdown budget; these six cases are not an exhaustive test of cloud events or signal ordering.

cluster-autoscaler.kubernetes.io/safe-to-evict: "false" can prevent Cluster Autoscaler scale-down of a node hosting the pod; it does not prevent Spot reclamation or every managed-node maintenance event. Setting it to "true" allows Cluster Autoscaler to consider eviction but does not select a shutdown timeout. See the Cluster Autoscaler FAQ.

These tests ran on a local Kubernetes cluster without a Cluster Autoscaler or Spot instances. We explicitly deleted pods with a 700-second grace period to allow the workload to finish, or a 30-second grace period to force SIGKILL before completion. The latter simulates the effect of a short shutdown window, such as during a Spot interruption or forced node removal. Actual shutdown limits depend on the provider, configuration, and event.

The available time depends on the provider, node configuration, Kubernetes version, and event:

AWS/EKS: EC2 Spot stop/termination has a best-effort two-minute interruption notice, not two guaranteed minutes for each pod. EKS warns that pods may receive less time or no termination signal. Managed-node-group desired-capacity reductions and AZRebalance have a separate 15-minute eviction limit before node termination; an ASG lifecycle hook can extend eligible terminations, but cannot extend Spot reclamation. See Spot notices, managed node groups, and ASG termination limits. For managed-node-group version/AMI updates, the 15-minute drain timeout instead fails an unforced update with PodEvictionFailure; it is not a universal hard termination deadline. See EKS update behavior.
GKE: Default Spot shutdown is 30 seconds total, split into 15 seconds for ordinary pods and 15 for critical system pods. Current documentation allows up to 120 seconds of total node shutdown time through node configuration on supported versions (Standard control plane >= 1.35.0-gke.1171000); ordinary pods receive the total node budget minus the critical-pod allocation. Separately, autoscaler scale-down allows one hour from 1.32.7-gke.1079000, versus 10 minutes on older versions. See GKE Spot shutdown and GKE autoscaling.
Azure/AKS: Spot eviction notifications are best effort, up to 30 seconds; AKS also attempts automatic cordon/drain on Preempt. Autoscaler scale-down separately defaults to 600 seconds through the configurable max-graceful-termination-sec profile setting. See Azure Spot eviction, AKS auto-drain, and AKS autoscaling. Rolling upgrades have a different drain timeout: 30 minutes by default, configurable from 5 minutes to 24 hours; expiry with pods remaining stops the upgrade. See AKS upgrade draining.

During normal graceful pod termination, Kubernetes honours the configured terminationGracePeriodSeconds. Configure it separately for the Airflow worker and KPO application pod, allowing enough time for the remaining work and result reporting. The default is 30 seconds, and preStop consumes part of this budget. See Kubernetes pod termination. With #69034, the worker can continue monitoring the running task during this period. If the underlying node is terminated earlier, the available time is constrained by that event, such as the Spot and managed-node-group termination limits described above.

Could you test by applying #69034's Task SDK changes to your existing test deployment and setting DUMB_INIT_SETSID=0 in the actual KubernetesExecutor worker pod template, then restarting the affected components? You do not need to build from main or upgrade all packages to test the backport. Please include the node-removal event, executor/provider versions, and both pod grace periods when reporting the result.

I am also considering adding integration tests to Airflow's test suite to cover these six termination scenarios.

Drafted-by: Codex (GPT-6); reviewed by @arkadiuszbach before posting

Navigation Menu

Repository navigation

KubernetesPodOperator task is incorrectly marked as SUCCESS instead of FAILED when both the task pod and its child pod are interrupted in close succession #71202

Under which category would you file this issue?

Apache Airflow version

What happened and how to reproduce it?

What you think should happen instead?

Operating System

Deployment

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Official Helm Chart version

Kubernetes Version

Helm Chart configuration

Docker Image customizations

Anything else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

KubernetesPodOperator task is incorrectly marked as SUCCESS instead of FAILED when both the task pod and its child pod are interrupted in close succession #71202

Description

Under which category would you file this issue?

Apache Airflow version

What happened and how to reproduce it?

What you think should happen instead?

Operating System

Deployment

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Official Helm Chart version

Kubernetes Version

Helm Chart configuration

Docker Image customizations

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

boring-cyborg commented on Aug 5, 2026

adrianrego commented on Aug 6, 2026

arkadiuszbach commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions