Skip to content

Expose task instance loop pass and expansion size to listeners - #75132

Merged
ashb merged 5 commits into
apache:mainfrom
kacpermuda:expose-loop-and-expansion-position
Oct 21, 2026
Merged

ashb merged 5 commits into
apache:mainfrom
kacpermuda:expose-loop-and-expansion-position

Conversation

@kacpermuda

@kacpermuda kacpermuda commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Listener consumers such as OpenLineage should be able to trace an event back to the task execution that produced it from what the listener gets, not by looking the attempt up by ti.id in the metadata database. For mapped tasks and task loops part of that wasn't exposed:

  • LoopContext.region_id: the loop region holding the iteration. A mapped task's own region_id is its expansion's, so this is what ties it to the other tasks of the same pass.
  • LoopContext.trajectory, forked_from_region_id and resumes_from_index: which rerun of the loop an iteration ran in, and which execution it replaced from which iteration on. Before, ordering reruns meant comparing the UUIDv7 region ids of two events. With this new information, we know everything from a single event.
  • Public RuntimeTaskInstance.loop, loop_decision and expanded_ti_count. A gate's decision was only in the private _loop_decision, so its listener couldn't tell until being met from max_iterations being reached.
  • get_task_instance_execution_position(ti, session=...), so listeners called on the API server (manual state changes) get the same data as the worker, instead of reading dynamic_region rows.
  • TIRunContext.expanded_ti_count: the number of live instances in the mapped task's expansion, counted within the iteration inside a loop. Without it a listener doesn't know how many events to expect, so it can't tell "3 of 20 instances failed" or that all of them finished. This one was not required for backtracing, but it's a really useful piece of information.

The fields go into the unreleased 2026-10-30 Execution API version, so they can ship with 3.4.0.

related: #74752


Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code)

Generated-by: Claude Code (Opus 5.5) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

🤖 Generated with Claude Code

@ashb ashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure integer trajectory is right way to expose this though

Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/src/airflow/api_fastapi/execution_api/datamodels/taskinstance.py Outdated
Comment thread airflow-core/src/airflow/models/task_coordinates.py Outdated
Comment thread airflow-core/src/airflow/models/task_coordinates.py Outdated
Comment thread airflow-core/tests/unit/models/test_dynamic_region.py Outdated
@ashb

ashb commented Oct 19, 2026

Copy link
Copy Markdown
Member

I'll need to take another look from a computer

@kacpermuda

Copy link
Copy Markdown
Collaborator Author

I'm not sure integer trajectory is right way to expose this though

The trajectory itself is identified by the loop region UUID (loop.region_id). The integer is only that region’s position in the chain of forks, the same number the task log path already shows as pass=3.2. I’d like to keep it so a single event can tell which rerun it comes from, and whether it’s newer, without having seen the other region’s UUID (UUIDv7 ordering needs both). I reworded the docs and docstrings to make that split explicit. Happy to rename it if “trajectory” clashes with something different you have in mind.

Comment thread task-sdk/src/airflow/sdk/execution_time/task_runner.py
Comment thread task-sdk/src/airflow/sdk/execution_time/task_runner.py
Comment thread airflow-core/src/airflow/models/task_coordinates.py
@kacpermuda
kacpermuda force-pushed the expose-loop-and-expansion-position branch from dd5a010 to 7db75fd Compare October 20, 2026 13:19
@kacpermuda kacpermuda added the full tests needed We need to run full set of tests for this PR to merge label Oct 20, 2026
@kacpermuda
kacpermuda marked this pull request as ready for review October 20, 2026 13:25

@ashb ashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broadly approved, could of nits/questions about the generated schema and models

Comment thread airflow-core/docs/authoring-and-scheduling/loops.rst Outdated
Comment thread airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py Outdated
Comment thread go-sdk/pkg/execution/genmodels/models.gen.go
Comment thread go-sdk/pkg/execution/genmodels/models.gen.go Outdated
Comment thread task-sdk/src/airflow/sdk/execution_time/schema/schema.json Outdated
Comment thread task-sdk/src/airflow/sdk/execution_time/schema/schema.json
kacpermuda and others added 5 commits October 21, 2026 11:28
…ners

A single event about a task instance should be enough to tell which execution it
describes: which loop pass and rerun of the loop it ran in, how many instances its
mapped expansion has, and why a loop gate stopped. Workers only had the loop
iteration, a mapped task inside a loop could not be tied to the pass it ran in,
reruns could only be ordered by comparing region ids from two events, and server
side consumers such as listeners had to read region rows to place a stored task
instance.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ed attempts

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@kacpermuda
kacpermuda force-pushed the expose-loop-and-expansion-position branch from 2a1e812 to f3d974d Compare October 21, 2026 09:42
@ashb ashb added the backport-to-v3-4-test Backport to v3-4-test label Oct 21, 2026
@ashb
ashb merged commit e6a2c9f into apache:main Oct 21, 2026
148 checks passed
@kacpermuda
kacpermuda deleted the expose-loop-and-expansion-position branch October 21, 2026 11:32
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-4-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ v3-4-test Commit Link

You can attempt to backport this manually by running:

cherry_picker e6a2c9f v3-4-test

This should apply the commit to the v3-4-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 21, 2026
Uses what apache#75132 exposes, so a single event tells which rerun of a
loop it comes from, whether that rerun superseded an earlier one, and how many
mapped instances to expect, and events for manual state changes get the same
details as worker events. Loop details are grouped in one object so it can
describe enclosing loops if Airflow ever nests them, and iterations are counted
from one as in Airflow.

Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 21, 2026
Uses what apache#75132 exposes, so a single event tells which rerun of a
loop it comes from, whether that rerun superseded an earlier one, and how many
mapped instances to expect, and events for manual state changes get the same
details as worker events. Loop details are grouped in one object so it can
describe enclosing loops if Airflow ever nests them, and iterations are counted
from one as in Airflow.

Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 21, 2026
…eners (apache#75132)

A single event about a task instance should be enough to tell which execution it
describes: which loop pass and rerun of the loop it ran in, how many instances its
mapped expansion has, and why a loop gate stopped. Workers only had the loop
iteration, a mapped task inside a loop could not be tied to the pass it ran in,
reruns could only be ordered by comparing region ids from two events, and server
side consumers such as listeners had to read region rows to place a stored task
instance.
(cherry picked from commit e6a2c9f)

Co-authored-by: Claude <noreply@anthropic.com>
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 21, 2026
…eners (apache#75132)

A single event about a task instance should be enough to tell which execution it
describes: which loop pass and rerun of the loop it ran in, how many instances its
mapped expansion has, and why a loop gate stopped. Workers only had the loop
iteration, a mapped task inside a loop could not be tied to the pass it ran in,
reruns could only be ordered by comparing region ids from two events, and server
side consumers such as listeners had to read region rows to place a stored task
instance.
(cherry picked from commit e6a2c9f)

Co-authored-by: Claude <noreply@anthropic.com>
ashb pushed a commit that referenced this pull request Oct 21, 2026
…eners (#75132) (#75309)

A single event about a task instance should be enough to tell which execution it
describes: which loop pass and rerun of the loop it ran in, how many instances its
mapped expansion has, and why a loop gate stopped. Workers only had the loop
iteration, a mapped task inside a loop could not be tied to the pass it ran in,
reruns could only be ordered by comparing region ids from two events, and server
side consumers such as listeners had to read region rows to place a stored task
instance.
(cherry picked from commit e6a2c9f)

Co-authored-by: Claude <noreply@anthropic.com>
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 21, 2026
Uses what apache#75132 exposes, so a single event tells which rerun of a
loop it comes from, whether that rerun superseded an earlier one, and how many
mapped instances to expect, and events for manual state changes get the same
details as worker events. Loop details are grouped in one object so it can
describe enclosing loops if Airflow ever nests them, and iterations are counted
from one as in Airflow.

Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 21, 2026
…code

The API server holds task instances without their task when a user sets a
state, so those events never carried the airflow run facet the guide promised.
The fallbacks for loop details only served Airflow builds from before
apache#75132, which no release contains. Timing the stored position
lookup makes its per task instance queries in the scheduler visible.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:go-sdk area:logging area:task-sdk area:ts-sdk backport-to-v3-4-test Backport to v3-4-test full tests needed We need to run full set of tests for this PR to merge kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants