Skip to content

Fix asset dependency graph dropping outlets for tasks that both consume and produce - #72272

Open
ColtenOuO wants to merge 1 commit into
apache:mainfrom
ColtenOuO:fix-asset-dependency-graph-shared-processed-tasks
Open

ColtenOuO wants to merge 1 commit into
apache:mainfrom
ColtenOuO:fix-asset-dependency-graph-shared-processed-tasks

Conversation

@ColtenOuO

Copy link
Copy Markdown
Contributor

Summary

get_data_dependencies() (the Assets dependency-graph API) shares a single processed_tasks set between its producer-side and consumer-side BFS branches. A task that both consumes one asset and produces another gets marked "processed" by whichever branch the BFS reaches it from first, which silently skips the batched lookup the other branch would have run — dropping any asset only reachable through that skipped direction, with no error or warning that the graph is incomplete.

Example

Task transform reads asset X and writes assets Y and Z:

flowchart LR
    X([asset X]) --> transform[[transform]]
    transform --> Y([asset Y])
    transform --> Z([asset Z])
Loading

Querying the dependency graph rooted at Y:

sequenceDiagram
    participant Y as asset Y (query root)
    participant BFS
    participant X as asset X

    Note over BFS: Round 1 — visiting Y
    Y->>BFS: transform produces Y
    BFS->>BFS: mark transform processed<br/>query its INLETS → finds X
    Note over BFS: Round 2 — visiting X
    X->>BFS: transform consumes X
    BFS->>BFS: transform already processed<br/>→ OUTLET query skipped
    Note over BFS: Z is never queried — silently missing from the graph
Loading

transform is discovered twice — once as a producer of Y (which queries its inlets, finding X), and once as a consumer of X (which should query its outlets to keep tracing downstream). Because both branches gate their lookup on the same processed_tasks set, the second visit sees transform already marked and skips the outlet query entirely — so Z, an outlet unrelated to X or Y, never appears in the graph.

Fix

Split the shared processed_tasks set into two independently-tracked sets, processed_inlet_tasks and processed_outlet_tasks, so a task reached from one direction still has the other direction's lookup run exactly once. The final team-annotation step, which previously derived all_dag_ids from processed_tasks, now derives it directly from the task nodes already collected in the graph — the more direct source of truth, and unaffected by the split.

Testing

Added test_data_dependencies_finds_all_outlets_for_task_that_also_consumes_an_asset, reproducing the exact scenario above (task reads X, writes Y and Z; query rooted at Y). Verified the test fails on the pre-fix code (Z missing from the response) and passes after the fix.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Sonnet 5) for writing test

@fat-catTW

Copy link
Copy Markdown
Contributor

Look nice!
Thanks for contribution :D

@ColtenOuO
ColtenOuO force-pushed the fix-asset-dependency-graph-shared-processed-tasks branch from d23bfb5 to 2d2461e Compare September 24, 2026 03:32
@ColtenOuO

Copy link
Copy Markdown
Contributor Author

Resolved the conflict :D

@potiuk

potiuk commented Sep 25, 2026

Copy link
Copy Markdown
Member

Hello @ColtenOuO - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 19 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

…me and produce

A task that both consumes one asset and produces another shared a
single processed_tasks set between the graph's inlet-lookup and
outlet-lookup branches. Whichever branch the BFS reached the task
from first marked it processed for both directions, so the other
direction's batched lookup was silently skipped -- any asset only
reachable through that skipped direction never appeared in the
graph, with no error to indicate the graph was incomplete.

Track each direction in its own set (processed_inlet_tasks /
processed_outlet_tasks) so a task reached from one side still has
its other side's assets resolved.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API closed because of open PR limit Closed as a one-time step of introducing the open pull request limit

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants