Skip to content

Enforce execution_timeout from the task supervisor - #72493

Closed
Andrushika wants to merge 4 commits into
apache:mainfrom
Andrushika:evaluate-issue-53337-takeover
Closed

Andrushika wants to merge 4 commits into
apache:mainfrom
Andrushika:evaluate-issue-53337-takeover

Conversation

@Andrushika

@Andrushika Andrushika commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Why

execution_timeout is only enforced by a SIGALRM handler inside the task runner process. When that process is stuck in native code, deadlocked on a lock inherited across fork, or hit by SIGSEGV, the handler never runs and the task stays running forever (#57174, #57712, #45930).

What

The task process sends a new SetExecutionTimeout message to the supervisor right before it arms its own alarm, so both clocks start at the same point. If the task has not reported a terminal state within 5 seconds after the timeout, the supervisor sends SIGTERM to the process group, and SIGKILL 5 seconds later.

This keeps the task runner's SIGALRM handler instead of "moving it", which is a bit different from what #53337 asks for. The reason is that on_kill, the retry policy and the failure callbacks can only run inside the task process, and dropping the handler would skip them on every timeout.

closes: #53337, #57174, #57712
related: #45930


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Fable 5.1)

Generated-by: Claude Code (Fable 5.1) following the guidelines

The timeout is only enforced by a SIGALRM handler inside the task
process. When that process is stuck in native code, deadlocked on a
lock inherited across fork, or corrupted by SIGSEGV, the handler never
runs and the task stays running forever. The task process now tells the
supervisor its timeout before starting execute(), and the supervisor
sends SIGTERM and then SIGKILL if the task has not reported a terminal
state within a grace period. The in-process handler stays as the first
responder because on_kill, the retry policy and callbacks can only run
there.
@Andrushika

Andrushika commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor Author

It may also fix related issues like #57174 and #57712, but I have not reproduced those yet, so that still needs confirmation. If so, I'll edit the close: block later.

@seanmuth

Copy link
Copy Markdown
Contributor

SetExecutionTimeout is only sent right before entering execute():

if timeout_seconds <= 0:
    raise AirflowTaskTimeout()
SUPERVISOR_COMMS.send(SetExecutionTimeout(timeout_seconds=timeout_seconds))
with timeout(timeout_seconds):
    result = ctx.run(execute, context=context)

So _execution_timeout_enforce_at doesn't exist until that message arrives — the supervisor-side backstop is inert for anything before execute() starts: bundle loading, parsing, bundle refresh. A wedge there (e.g. the fork-orphaned-lock case named directly in _handle_execution_timeout_if_needed's own docstring, #71707) runs with no deadline at all regardless of execution_timeout, since the message establishing one hasn't been sent yet.

Given the mechanism already exists and doesn't depend on the task process's cooperation, would it make sense for the supervisor to arm a (longer, generic) deadline of its own covering the pre-execute() window too — bundle load/parse start to SetExecutionTimeout arrival — rather than leaving that window fully unenforced? Not asking for scope creep on this PR necessarily, just flagging it doesn't close #71707 as fully as the related: line implies.


Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting

@Andrushika

Andrushika commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Good catch, you are right! I fixed the docstring and updated the description.

Thanks for the idea of a deadline for the pre-execute() window -- it does sound useful. However, the supervisor does not know execution_timeout before the Dag is parsed, so it would need its own config value.
I would like to leave it in a separate PR.

I also updated #57174 and #57712 in the PR description as close:. Please correct me if it's wrong.

@Andrushika
Andrushika marked this pull request as draft September 18, 2026 15:45
@Andrushika
Andrushika marked this pull request as ready for review September 18, 2026 15:45
…7-takeover

Resolve supervisor.py: port the SetExecutionTimeout handling onto main's
request-handler dispatch, replacing the isinstance branch with
_handle_set_execution_timeout plus a register_request_method entry.
@potiuk

potiuk commented Sep 25, 2026

Copy link
Copy Markdown
Member

Hello @Andrushika - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 7 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Handle task timeouts (execution_timeout) at supervisor

3 participants