Repository navigation
Conversation
|
Look nice! |
d23bfb5 to
2d2461e
Compare
|
Resolved the conflict :D |
|
Hello @ColtenOuO - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 19 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
…me and produce A task that both consumes one asset and produces another shared a single processed_tasks set between the graph's inlet-lookup and outlet-lookup branches. Whichever branch the BFS reached the task from first marked it processed for both directions, so the other direction's batched lookup was silently skipped -- any asset only reachable through that skipped direction never appeared in the graph, with no error to indicate the graph was incomplete. Track each direction in its own set (processed_inlet_tasks / processed_outlet_tasks) so a task reached from one side still has its other side's assets resolved.
2d2461e to
00a90c3
Compare
Summary
get_data_dependencies()(the Assets dependency-graph API) shares a singleprocessed_tasksset between its producer-side and consumer-side BFS branches. A task that both consumes one asset and produces another gets marked "processed" by whichever branch the BFS reaches it from first, which silently skips the batched lookup the other branch would have run — dropping any asset only reachable through that skipped direction, with no error or warning that the graph is incomplete.Example
Task
transformreads assetXand writes assetsYandZ:flowchart LR X([asset X]) --> transform[[transform]] transform --> Y([asset Y]) transform --> Z([asset Z])Querying the dependency graph rooted at
Y:sequenceDiagram participant Y as asset Y (query root) participant BFS participant X as asset X Note over BFS: Round 1 — visiting Y Y->>BFS: transform produces Y BFS->>BFS: mark transform processed<br/>query its INLETS → finds X Note over BFS: Round 2 — visiting X X->>BFS: transform consumes X BFS->>BFS: transform already processed<br/>→ OUTLET query skipped Note over BFS: Z is never queried — silently missing from the graphtransformis discovered twice — once as a producer ofY(which queries its inlets, findingX), and once as a consumer ofX(which should query its outlets to keep tracing downstream). Because both branches gate their lookup on the sameprocessed_tasksset, the second visit seestransformalready marked and skips the outlet query entirely — soZ, an outlet unrelated toXorY, never appears in the graph.Fix
Split the shared
processed_tasksset into two independently-tracked sets,processed_inlet_tasksandprocessed_outlet_tasks, so a task reached from one direction still has the other direction's lookup run exactly once. The final team-annotation step, which previously derivedall_dag_idsfromprocessed_tasks, now derives it directly from the task nodes already collected in the graph — the more direct source of truth, and unaffected by the split.Testing
Added
test_data_dependencies_finds_all_outlets_for_task_that_also_consumes_an_asset, reproducing the exact scenario above (task reads X, writes Y and Z; query rooted at Y). Verified the test fails on the pre-fix code (Zmissing from the response) and passes after the fix.Was generative AI tooling used to co-author this PR?