[Andrushika]

Exact same work in #72493. Created this because that one was closed by the PR number limit policy.

Why

execution_timeout is only enforced by a SIGALRM handler inside the task runner process. When that process is stuck in native code or hit by SIGSEGV, the handler never runs and the task stays running forever (#57174, #57712, #45930).

To tackle the problem at its root, the better solution is to handle the runner's timeout from the supervisor side.

What

The task process sends a new SetExecutionTimeout message to the supervisor right before it arms its own alarm, so both clocks start at the same point. If the task has not reported a terminal state within [core] killed_task_cleanup_time seconds (default 60) after the timeout, the supervisor reuses kill(SIGTERM, force=True) to send SIGTERM to the process group, and SIGKILL after the same period again. kill() blocks the monitor loop while it waits, so no heartbeat is sent during that time. By default that is at most 120 seconds, under the 300 second task_instance_heartbeat_timeout.

This keeps the task runner's SIGALRM handler instead of "moving it", which is a bit different from what #53337 asks for. The reason is that on_kill, the retry policy and the failure callbacks can only run inside the task process, and dropping the handler would skip them on every timeout.

closes: #53337, closes #57174, closes #57712, closes #45930

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Fable 5.1)

Generated-by: Claude Code (Fable 5.1) following the guidelines

[SameerMesiah97]

Just 2 questions.

[SameerMesiah97]

It looks good to me but since this is touching a fairly sensitive part of airflow core, I think this needs opinions from the tagged maintainers for review before merge.

[ashb]

I'm not sure this will fix #57174 though -- as dag parsing is needed before the child can know the desired timeout to report to the scheduler.

• I wonder if we need two timeouts like this:
• child gets sent the TaskWorkload
• Sends a SetTimeout to the supervisor of the dabgag import timeout (existing config)
• Then the child parses the dag
• Once done, either we send a 0 timeout to clear it in the parent;
• Or if the gap between this and the next is small enough (bear in mind things like listeners or other things could be involved here) just send a new SetTimeout with the desired timeout.

Given not all tasks have timesouts, I think the SetTimeout(0) approach is "it's always correct, and easier to reason about even if fractionaly slower"

[Andrushika]

I'm not sure this will fix #57174 though -- as dag parsing is needed before the child can know the desired timeout to report to the scheduler.

I think this PR will fix that issue, since the log there shows the SIGTERM comes after parsing.
(I changed those #57174 and #57712 into related since I have no way to validate)

• I wonder if we need two timeouts like this:
• child gets sent the TaskWorkload
• Sends a SetTimeout to the supervisor of the dabgag import timeout (existing config)
• Then the child parses the dag
• Once done, either we send a 0 timeout to clear it in the parent;
• Or if the gap between this and the next is small enough (bear in mind things like listeners or other things could be involved here) just send a new SetTimeout with the desired timeout.

Given not all tasks have timesouts, I think the SetTimeout(0) approach is "it's always correct, and easier to reason about even if fractionaly slower"

But yeah I agree we need this. Could we do it in a follow-up? (I would like to make this as small as possible)

[ashb]

Much, much easier to understand now.

[Andrushika]

Thanks!
Waiting for CI green :)

[ashb]

This should also close and fix #57174, #57712, #45930, right?

[Andrushika]

Yes. I moved it to related bc I couldn't really reproduce it.
But maintainers in all three threads already pointed to #53337 (the previous closed PR, similar solution) as the fix, so I'll move them back to closes:.