Repository navigation
Describe loop reruns and expansion size in OpenLineage events - #75133
Closed
kacpermuda wants to merge 8 commits into
Closed
kacpermuda wants to merge 8 commits into
kacpermuda wants to merge 8 commits into
Conversation
…ners A single event about a task instance should be enough to tell which execution it describes: which loop pass and rerun of the loop it ran in, how many instances its mapped expansion has, and why a loop gate stopped. Workers only had the loop iteration, a mapped task inside a loop could not be tied to the pass it ran in, reruns could only be ordered by comparing region ids from two events, and server side consumers such as listeners had to read region rows to place a stored task instance. Co-Authored-By: Claude <noreply@anthropic.com>
Provider mypy and the docs spellcheck run in CI through breeze rather than the default prek hooks, so they were not caught locally before the previous push. Co-Authored-By: Claude <noreply@anthropic.com>
Alerting on why a loop stopped (for example, its until condition still being false at max_iterations) needs the gate's decision in the event itself, without reading the Airflow metadata database. Co-Authored-By: Claude <noreply@anthropic.com>
The run id must not depend on the optional loop iteration, and the lineage macro should place a task instance the same way the listener does. Co-Authored-By: Claude <noreply@anthropic.com>
A region of an unexpected shape, or a failed lookup of a mapped task's iteration, must not merge the runs of loop iterations or cost an event. Tests now pin the region shapes Airflow creates, so a change to them fails the provider's tests instead of silently changing events. Co-Authored-By: Claude <noreply@anthropic.com>
The dbt Cloud and Dataproc tests patched how OpenLineage builds a task's run id, so they broke when that changed. Stubbing the macros they actually use keeps them independent of it. Co-Authored-By: Claude <noreply@anthropic.com>
Alerting on task loops and mapped tasks has to work from one event at a time: telling which pass and which rerun of a loop an event comes from, whether that rerun superseded an earlier one, and how many mapped instances to expect. Loop details are grouped in one object so that it can describe enclosing loops if Airflow ever nests them, and manual state changes now get the same details from Airflow as worker events instead of reading its region tables. Co-Authored-By: Claude <noreply@anthropic.com>
boring-cyborg
Bot
added
area:API
Collaborator
Author
|
Closed in favor of #74752 as the core PR was merged before the OL, so we can just do it all in single OL PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #74752 and #75132, and stacked on both: only the last commit (
Describe loop reruns and expansion size in OpenLineage task events) is new here.With #75132, OpenLineage events can carry everything needed to trace a task execution back from a single event, without reading the Airflow database:
taskInstance.loopgroups the loop details:group_id,index,max_iterations, plus the loopregion_id,trajectory,forked_from_region_idandresumes_from_index(which rerun of the loop an event comes from and what it replaced), and the gate'sdecision. This replaces the flatloop_*fields added in Fix task loop information and tests for OpenLineage provider #74752, which haven't been released, so the loop details can later describe enclosing loops too.taskInstance.expanded_ti_count: how many instances a mapped task's expansion has, so consumers know how many events to expect.RuntimeTaskInstanceattributes. Events for manual state changes get the same details fromget_task_instance_execution_positioninstead of queryingdynamic_region. Airflow 3.4 pre-releases without these fall back to what Fix task loop information and tests for OpenLineage provider #74752 does.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.🤖 Generated with Claude Code