Repository navigation
Handle task timeouts (execution_timeout) at supervisor #53337
Description
Body
Right now we handle these at the task runner (fork: https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/task_runner.py#L1149-L1165) instead of handling it at the supervisor. We should move the check to the supervisor level.
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Activity
ashb commentedon Jul 14, 2025
Since the timeout is defined in the dag we'll need to communicate the timeout to the supervisor- possibly by the Run request (but there's no point sending it to the API server that I can immediately think of)
potiuk commentedon Jul 14, 2025
Another option is to make a second fork:
And have the timeout fork only handle the timeout via ALRM signal. And the timeout fork could do dag parsing and deciding what to run before forking the task fork. This way it can retrieve timeout from DAG definition, and then (because it will fork) - the resulting task fork will already have the task parsed in memory.
While forks are not as "good" for Python interpreter for memory as for regular c-code (because of reference counting almost all memory is very quickly copied-on-write. this is not the case when the fork will merely join the forked process and react to ALRM signal.
Would be worth seeing if this approach might be good for memory usage - but it has the "isolation" property that is rather nice. We can have single ALRM handler in the "timeout" fork rather than handling more complex cases of supervisor waiting for the timeout additionally. But also it might be an overkill - I thought I will just mention it as we discussed that option as potential implementation in 2.0
Handle task timeouts (execution_timeout) at supervisor
Handle task timeouts (execution_timeout) at supervisor
amoghrajesh commentedon Jun 22, 2026
No b/w for 3.3.0, moving up
Andrushika commentedon Sep 3, 2026
Hi! I picked this one up since it has been open for a while. #72493 fixes the case where the task runner hangs permanently and
execution_timeoutnever fires. The implementation is a bit different from what this issue describes -- it does not completely move timeout handling to the supervisor (please see #72493 PR body). Would appreciate a look when someone has a chance, thanks!seanmuth commentedon Sep 17, 2026
Worth naming specifically what's still open even once #72493 lands, following the note in the PR body that its implementation differs from what this issue describes.
#72493's supervisor-side deadline only exists from the moment
SetExecutionTimeoutis sent, which happens right beforeexecute()starts — detail on the exact code path here. Everything before that — Dag bundle loading, parsing — runs with no external enforcement of any kind, regardless ofexecution_timeout. A fork-orphaned-lock wedge during that window (#71707) would still run forever under #72493 as currently scoped.Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting
Andrushika commentedon Sep 27, 2026
Reopened #72493 as #73806 since the new PR number limit policy.