Repository navigation
Expose task instance loop pass and expansion size to listeners - #75132
Conversation
ashb
left a comment
There was a problem hiding this comment.
I'm not sure integer trajectory is right way to expose this though
|
I'll need to take another look from a computer |
The trajectory itself is identified by the loop region UUID (loop.region_id). The integer is only that region’s position in the chain of forks, the same number the task log path already shows as pass=3.2. I’d like to keep it so a single event can tell which rerun it comes from, and whether it’s newer, without having seen the other region’s UUID (UUIDv7 ordering needs both). I reworded the docs and docstrings to make that split explicit. Happy to rename it if “trajectory” clashes with something different you have in mind. |
dd5a010 to
7db75fd
Compare
ashb
left a comment
There was a problem hiding this comment.
Broadly approved, could of nits/questions about the generated schema and models
…ners A single event about a task instance should be enough to tell which execution it describes: which loop pass and rerun of the loop it ran in, how many instances its mapped expansion has, and why a loop gate stopped. Workers only had the loop iteration, a mapped task inside a loop could not be tied to the pass it ran in, reruns could only be ordered by comparing region ids from two events, and server side consumers such as listeners had to read region rows to place a stored task instance. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ed attempts Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
2a1e812 to
f3d974d
Compare
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker e6a2c9f v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
Uses what apache#75132 exposes, so a single event tells which rerun of a loop it comes from, whether that rerun superseded an earlier one, and how many mapped instances to expect, and events for manual state changes get the same details as worker events. Loop details are grouped in one object so it can describe enclosing loops if Airflow ever nests them, and iterations are counted from one as in Airflow. Co-Authored-By: Claude <noreply@anthropic.com>
Uses what apache#75132 exposes, so a single event tells which rerun of a loop it comes from, whether that rerun superseded an earlier one, and how many mapped instances to expect, and events for manual state changes get the same details as worker events. Loop details are grouped in one object so it can describe enclosing loops if Airflow ever nests them, and iterations are counted from one as in Airflow. Co-Authored-By: Claude <noreply@anthropic.com>
…eners (apache#75132) A single event about a task instance should be enough to tell which execution it describes: which loop pass and rerun of the loop it ran in, how many instances its mapped expansion has, and why a loop gate stopped. Workers only had the loop iteration, a mapped task inside a loop could not be tied to the pass it ran in, reruns could only be ordered by comparing region ids from two events, and server side consumers such as listeners had to read region rows to place a stored task instance. (cherry picked from commit e6a2c9f) Co-authored-by: Claude <noreply@anthropic.com>
…eners (apache#75132) A single event about a task instance should be enough to tell which execution it describes: which loop pass and rerun of the loop it ran in, how many instances its mapped expansion has, and why a loop gate stopped. Workers only had the loop iteration, a mapped task inside a loop could not be tied to the pass it ran in, reruns could only be ordered by comparing region ids from two events, and server side consumers such as listeners had to read region rows to place a stored task instance. (cherry picked from commit e6a2c9f) Co-authored-by: Claude <noreply@anthropic.com>
…eners (#75132) (#75309) A single event about a task instance should be enough to tell which execution it describes: which loop pass and rerun of the loop it ran in, how many instances its mapped expansion has, and why a loop gate stopped. Workers only had the loop iteration, a mapped task inside a loop could not be tied to the pass it ran in, reruns could only be ordered by comparing region ids from two events, and server side consumers such as listeners had to read region rows to place a stored task instance. (cherry picked from commit e6a2c9f) Co-authored-by: Claude <noreply@anthropic.com>
Uses what apache#75132 exposes, so a single event tells which rerun of a loop it comes from, whether that rerun superseded an earlier one, and how many mapped instances to expect, and events for manual state changes get the same details as worker events. Loop details are grouped in one object so it can describe enclosing loops if Airflow ever nests them, and iterations are counted from one as in Airflow. Co-Authored-By: Claude <noreply@anthropic.com>
…code The API server holds task instances without their task when a user sets a state, so those events never carried the airflow run facet the guide promised. The fallbacks for loop details only served Airflow builds from before apache#75132, which no release contains. Timing the stored position lookup makes its per task instance queries in the scheduler visible. Co-Authored-By: Claude <noreply@anthropic.com>
Listener consumers such as OpenLineage should be able to trace an event back to the task execution that produced it from what the listener gets, not by looking the attempt up by
ti.idin the metadata database. For mapped tasks and task loops part of that wasn't exposed:LoopContext.region_id: the loop region holding the iteration. A mapped task's ownregion_idis its expansion's, so this is what ties it to the other tasks of the same pass.LoopContext.trajectory,forked_from_region_idandresumes_from_index: which rerun of the loop an iteration ran in, and which execution it replaced from which iteration on. Before, ordering reruns meant comparing the UUIDv7 region ids of two events. With this new information, we know everything from a single event.RuntimeTaskInstance.loop,loop_decisionandexpanded_ti_count. A gate's decision was only in the private_loop_decision, so its listener couldn't telluntilbeing met frommax_iterationsbeing reached.get_task_instance_execution_position(ti, session=...), so listeners called on the API server (manual state changes) get the same data as the worker, instead of readingdynamic_regionrows.TIRunContext.expanded_ti_count: the number of live instances in the mapped task's expansion, counted within the iteration inside a loop. Without it a listener doesn't know how many events to expect, so it can't tell "3 of 20 instances failed" or that all of them finished. This one was not required for backtracing, but it's a really useful piece of information.The fields go into the unreleased
2026-10-30Execution API version, so they can ship with 3.4.0.related: #74752
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.🤖 Generated with Claude Code