Repository navigation
DagBag Processing SIGSEGV causes runaway tasks with LocalExecutor #57174
Description
Apache Airflow version
3.1.0
If "Other Airflow 2/3 version" selected, which one?
No response
What happened?
Using Airflow 3.1.0 - airflow standalone with the LocalExecutor on MacOS 14 - M3 ARM
I've noticed that occasionally tasks that are dynamically created (with .expand(), I don't think this is the issue fyi, just including), a subset of the them can runway, and will not fail even with a execution_timeout set. It completely disregards the timeout after this error is found in the logs:
ERROR - Received SIGSEGV signal while processing <my dag path>. source=airflow.models.dagbag.DagBag loc=dagbag.py:375
Here's some screenshots from a manually failed run, note the duration of 30 minutes even though a max timeout of 3 minutes:
Log from the Task:
Dag definition (linting errors are unrelated to this issue and are in a different task.):
I have a hunch this happens when the DAG Processor starts running at the same time as all of the tasks are dynamically created, but I truthfully have no clue.
I don't really have an issue with tasks like this failing on standalone due to a perfect storm of conditions (and non-intended OSes) as I only use it for local testing, but I would like the tasks to not run away.
Possibly Related Ticket:
#47011
#55838
#50303
#35967
What you think should happen instead?
No response
How to reproduce
No clue, besides try casting a wide net with many dynamically created tasks in the chance that this error bubbles up.
I've included my DAG code besides my custom BoxHook in case this helps.
Operating System
MacOS 14 - M3 ARM
Versions of Apache Airflow Providers
apache-airflow-providers-google==18.0.0
apache-airflow-providers-standard==1.8.0
Deployment
Other
Deployment details
Local Standalone
Anything else?
I think it bubbles up in the DAG run about 20% of the time.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
potiuk commentedon Oct 25, 2025
Corrrect - currently timeout is handled in the task itself via SIGALRM - and if another signal (SIGSEGV) comes it might get task into a hanging state. This happens usually when you have native code that runs in a long tight loop and does not handle signals in Python. We've discussed it in the past and solution to that is to handle the timeout in supervisor task (which is in another, parent process - tasks are in forked processes).
In order to communicate that we have to make a new task -> supervisor API (to send the timeout information after parsing the DAG, because the supervisor does not have that information as it d does not parse the task). This would generally handle all the possible cases where the task hangs - including possible escalation of signals to kill such forked tasks (SIGTERM followed by SIGKILL after a short additional timeout if the task does not exit).
That's all possible and if someone would like to take on that task - it's not even that difficult. I marked it as a good-first issue.
cc: @ashb @amoghrajesh if you have something to add.
LeonardoIshida commentedon Oct 25, 2025
Hi, can I be assigned to this?
potiuk commentedon Oct 25, 2025
Feel free.
amoghrajesh commentedon Oct 27, 2025
@LeonardoIshida this is the actual ticket: #53337, feel free to pick it up and if you have any qns, feel free to ask on that issue
qwe-kev commentedon Dec 18, 2025
Hi, I would like to work on this.
potiuk commentedon Dec 18, 2025
Added you - assuming @LeonardoIshida is not about to work on it :) ?
Handle task timeouts (execution_timeout) at supervisor
potiuk commentedon Mar 2, 2026
@qwe-kev We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
potiuk commentedon Mar 2, 2026
@LeonardoIshida We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
Handle task timeouts (execution_timeout) at supervisor
Andrushika commentedon Sep 4, 2026
Hi, in #72493 I implemented the approach suggested above -- handle the task timeout in the supervisor, so a stuck task runner process gets stopped from outside. I found it hard to reproduce the SIGSEGV scenario you described, but I think that PR can close this issue.