Repository navigation
wait_for_past_depends_before_skipping does not seem to have any effect #55146
Description
Apache Airflow version
Airflow 3.1.0
What happened?
I am part of a data engineering team. We use airflow to orchestrate our data ingestion pipelines. At a high-level, they are made of:
- An ingestion task group, that moves data from a source system to our S3 storage
- Validation tests
- Processing jobs that clean up and prepares the data for use by downstream consumers
- Validations tests on the processed data
The processing jobs run in Spark. To avoid overhead, I have been looking at using Airflow's skipping mechanisms to skip the rest of the pipeline if there is no data to be ingested.
However, we need the processing jobs to run sequentially. For this purpose, we use depends_on_past. I then stumbled upon the PR introducing wait_for_past_depends_before_skipping, which seems like it would allow me to skip tasks while still preserving the dependencies on past runs. However, the flag does not seem to have an effect.
What you think should happen instead?
The flag should behave as advertised in the base PR and in the docs.
How to reproduce
from airflow import DAG
from airflow.models import Param
from airflow.providers.standard.operators.empty import EmptyOperator
from airflow.providers.standard.operators.python import ShortCircuitOperator
from airflow.sdk import get_current_context
def check_continue():
ctx = get_current_context()
skip_downstream = ctx["params"]["skip"]
return not skip_downstream
with DAG(
dag_id="test_airflow_skip",
schedule=None,
params={
"skip": Param(default=False, type="boolean"),
},
) as dag:
depends_on_past = EmptyOperator(
task_id="depends_on_past",
depends_on_past=True,
wait_for_past_depends_before_skipping=True,
)
skip = ShortCircuitOperator(
task_id="skip",
python_callable=check_continue,
)
skip >> depends_on_past- Launch a run
- Then manually set the state of the
depends_on_pasttask tofailed - Then launch a run with the
skipparameter enabled:depends_on_pastwill be skipped when it shouldn't be.
Operating System
Ubuntu 22.04.5 LTS
Versions of Apache Airflow Providers
No response
Deployment
Docker-Compose
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Brunda10 commentedon Sep 3, 2025
I would like to work on this issue -Please assign it to me if nobody else is already working on it?
eladkal commentedon Sep 3, 2025
cc @hussein-awala
matthieucx commentedon Oct 3, 2025
@eladkal @hussein-awala Hello! I just tested this in an airflow 3.1.0 standalone setup, this issue does affect the latest version. I updated the issue description to reflect this and further streamlined the reproduction example.
matthieucx commentedon Nov 19, 2025
@eladkal @hussein-awala Hello! Sorry for the repeated pings. Would it be possible to triage the issue, as I managed to reproduce it after the latest minor airflow update?
Tomi-1997 commentedon Mar 11, 2026
Hey, I think I achieved the effect you wanted with these changes
Also, If the dependent task is a direct child, it will be skipped- so I added an empty task to bridge between them.
This is my result:
Here's the changes, let me know what you think
luc-pimentel commentedon Oct 19, 2026
Reproduced on 3.3.2 with
airflow standaloneon SQLite:ShortCircuitOperator>>PythonOperator(depends_on_past=True, wait_for_past_depends_before_skipping=True), run 1 fails the downstream task, run 2 short-circuits.depends_on_past=faileddepends_on_past=skipped(expectedNone, waiting)depends_on_past=faileddepends_on_past=skippedThe cause is in two places:
SkipMixin.skipin the task SDK skips every direct downstream task through the execution API before any scheduler dependency runs, and the deps only read the flag from theDepContext, which nothing sets in Airflow 3.Fix in #75028.
Drafted-by: Claude Code (Fable 5.1); reviewed by @luc-pimentel before posting