Repository navigation
Reduce per-task DB round trips in the asset dependency graph API - #71915
Merged
vatsrahul1001 merged 2 commits intoSep 7, 2026
Merged
Conversation
get_data_dependencies() already batches Dag entry-point resolution into one query per BFS round, but still issued one inlet/outlet TaskInletAssetReference/TaskOutletAssetReference query per individual task. An asset shared by many Dags/tasks (common in multi-team deployments) turned opening its dependency graph into hundreds of sequential round trips within a single request. Collect the task keys needing an inlet or outlet lookup across the whole round and resolve each with one batched query, the same tuple_(...).in_(...) pattern already used for entry-point resolution and elsewhere in the API layer.
ColtenOuO
requested review from
bugraoz93,
choo121600,
ephraimbuddy,
henry3260,
jason810496,
pierrejeambrun,
rawwar and
shubhamraj-git
as code owners
August 20, 2026 19:01
pierrejeambrun
approved these changes
Aug 27, 2026
The new regression test only exercised the inlet-lookup batching path, leaving the outlet-lookup path (traced from consuming tasks) uncovered.
ColtenOuO
force-pushed
the
batch-asset-graph-inlet-outlet-queries
branch
from
August 29, 2026 19:51
3659a8f to
34a8c84
Compare
Contributor
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker a3b5b0b v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
pierrejeambrun
pushed a commit
to astronomer/airflow
that referenced
this pull request
Sep 8, 2026
…che#71915) * Reduce per-task DB round trips in the asset dependency graph API get_data_dependencies() already batches Dag entry-point resolution into one query per BFS round, but still issued one inlet/outlet TaskInletAssetReference/TaskOutletAssetReference query per individual task. An asset shared by many Dags/tasks (common in multi-team deployments) turned opening its dependency graph into hundreds of sequential round trips within a single request. Collect the task keys needing an inlet or outlet lookup across the whole round and resolve each with one batched query, the same tuple_(...).in_(...) pattern already used for entry-point resolution and elsewhere in the API layer. * Add outlet-resolution batching test for the asset dependency graph The new regression test only exercised the inlet-lookup batching path, leaving the outlet-lookup path (traced from consuming tasks) uncovered.
1 task done
imrichardwu
pushed a commit
to imrichardwu/airflow
that referenced
this pull request
Sep 11, 2026
…che#71915) * Reduce per-task DB round trips in the asset dependency graph API get_data_dependencies() already batches Dag entry-point resolution into one query per BFS round, but still issued one inlet/outlet TaskInletAssetReference/TaskOutletAssetReference query per individual task. An asset shared by many Dags/tasks (common in multi-team deployments) turned opening its dependency graph into hundreds of sequential round trips within a single request. Collect the task keys needing an inlet or outlet lookup across the whole round and resolve each with one batched query, the same tuple_(...).in_(...) pattern already used for entry-point resolution and elsewhere in the API layer. * Add outlet-resolution batching test for the asset dependency graph The new regression test only exercised the inlet-lookup batching path, leaving the outlet-lookup path (traced from consuming tasks) uncovered.
regarmukesh3g
pushed a commit
to regarmukesh3g/airflow
that referenced
this pull request
Sep 27, 2026
…che#71915) * Reduce per-task DB round trips in the asset dependency graph API get_data_dependencies() already batches Dag entry-point resolution into one query per BFS round, but still issued one inlet/outlet TaskInletAssetReference/TaskOutletAssetReference query per individual task. An asset shared by many Dags/tasks (common in multi-team deployments) turned opening its dependency graph into hundreds of sequential round trips within a single request. Collect the task keys needing an inlet or outlet lookup across the whole round and resolve each with one batched query, the same tuple_(...).in_(...) pattern already used for entry-point resolution and elsewhere in the API layer. * Add outlet-resolution batching test for the asset dependency graph The new regression test only exercised the inlet-lookup batching path, leaving the outlet-lookup path (traced from consuming tasks) uncovered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
get_data_dependencies()(the Assets dependency-graph API) resolved each producing/consuming task's inlet/outlet asset references with one DB query per task, inside a BFS loop. An asset shared by many Dags/tasks — common in multi-team deployments — turned opening its graph into hundreds of sequential round trips in a single request.Change
_get_dag_entry_points); this extends that pattern to the two remaining per-task query sites.tuple_(dag_id, task_id).in_(...)query each, instead of one query per task.Round-trip reduction
Measured with the new regression test (3 Dags each with one task producing the same asset, in a single BFS round): total DB queries per request dropped from 11 to 9 — 2 fewer round trips (3 individual inlet queries collapsed into 1 batched query).
This scales with the number of tasks touched per round: for N tasks needing an inlet lookup, it goes from N round trips to 1; same for M tasks needing an outlet lookup. For an asset shared by hundreds of tasks, this is a reduction from hundreds of round trips down to at most 2 per BFS round (one inlet batch, one outlet batch).
Was generative AI tooling used to co-author this PR?