Repository navigation
Enforce execution_timeout from the task supervisor - #72493
Andrushika wants to merge 4 commits into
Conversation
The timeout is only enforced by a SIGALRM handler inside the task process. When that process is stuck in native code, deadlocked on a lock inherited across fork, or corrupted by SIGSEGV, the handler never runs and the task stays running forever. The task process now tells the supervisor its timeout before starting execute(), and the supervisor sends SIGTERM and then SIGKILL if the task has not reported a terminal state within a grace period. The in-process handler stays as the first responder because on_kill, the retry policy and callbacks can only run there.
|
if timeout_seconds <= 0:
raise AirflowTaskTimeout()
SUPERVISOR_COMMS.send(SetExecutionTimeout(timeout_seconds=timeout_seconds))
with timeout(timeout_seconds):
result = ctx.run(execute, context=context)So Given the mechanism already exists and doesn't depend on the task process's cooperation, would it make sense for the supervisor to arm a (longer, generic) deadline of its own covering the pre- Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting |
|
Good catch, you are right! I fixed the docstring and updated the description. Thanks for the idea of a deadline for the pre- I also updated #57174 and #57712 in the PR description as |
…7-takeover Resolve supervisor.py: port the SetExecutionTimeout handling onto main's request-handler dispatch, replacing the isinstance branch with _handle_set_execution_timeout plus a register_request_method entry.
|
Hello @Andrushika - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 7 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
Why
execution_timeoutis only enforced by a SIGALRM handler inside the task runner process. When that process is stuck in native code, deadlocked on a lock inherited across fork, or hit by SIGSEGV, the handler never runs and the task stays running forever (#57174, #57712, #45930).What
The task process sends a new
SetExecutionTimeoutmessage to the supervisor right before it arms its own alarm, so both clocks start at the same point. If the task has not reported a terminal state within 5 seconds after the timeout, the supervisor sends SIGTERM to the process group, and SIGKILL 5 seconds later.This keeps the task runner's SIGALRM handler instead of "moving it", which is a bit different from what #53337 asks for. The reason is that
on_kill, the retry policy and the failure callbacks can only run inside the task process, and dropping the handler would skip them on every timeout.closes: #53337, #57174, #57712
related: #45930
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Fable 5.1) following the guidelines