Repository navigation
[v3-4-test] Load the serialized Dag in ti_run only for stub tasks (#75795) - #75802
Merged
jason810496 merged 4 commits intoOct 28, 2026
Merged
Conversation
…ache#75795) Starting any task deserialized its whole Dag so the route could find out whether the task was a stub and read its arg_bindings. Denormalize the serialized is_stub flag onto task_instance, which ti_run already fetches and locks, and load the Dag only when it is set. Co-Authored-By: Claude <noreply@anthropic.com>
jason810496
requested review from
XD-DENG,
amoghrajesh,
ashb,
ephraimbuddy and
kaxil
as code owners
October 27, 2026 15:35
Co-Authored-By: Claude <noreply@anthropic.com>
test_find_executable_task_instances_skips_dag_that_cannot_be_deserialized fails on v3-4-test independently of this backport. apache#75745 brought the guard over from apache#75263 without the starved set that apache#75263 relies on, because that set came with apache#70307, which is not on this branch. The guard then returns False without narrowing the next query, so a cycle whose batch is filled by the bad Dag ends with nothing queued. Co-Authored-By: Claude <noreply@anthropic.com>
FrankYang0529
approved these changes
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #75795 to
v3-4-test, needed before 3.4.0 RC.Why
Starting any task deserialized its whole Dag, so that
ti_runcould find out whether the task was a stub and read itsarg_bindings. On a Dag cache miss that is a fullserialized_dagread plus deserialization per task start, paid while the route holdsSELECT ... FOR UPDATEon thetask_instancerow, and for a Python task the result is thrown away.How
The serialized
is_stubflag is denormalized ontotask_instance, whichti_runalready fetches and locks, and is checked first so the normal Python task path never loads the serialized Dag.Null means "not a stub", with no fallback. Every null row predates the migration, so it was written by 3.3, which has no
arg_bindingsat all, and such a task behaves exactly as it did there.Go and Java now write
is_stubfor aTriggerDagRuntask too, as TypeScript already does. The worker picks a runtime by queue and both SDKs put such a task on the Dag's queue, so their own runtime runs it andti_runhas to treat it as a stub.What differs from #75795
ee86eed19e24is this branch's head, sob4f1a7d2c903simply follows it and becomes the new head. Onmainthe same revision had to be inserted ahead of two 3.5.0 revisions.execution_timeoutchange. [v3-4-test] Revert "Enforce execution_timeout for tasks run by a language SDK" (#75376) #75469 took Enforce execution_timeout for tasks run by a language SDK #74869 back out ofv3-4-test, so there is noTIRunContext.execution_timeouthere and the route keeps its singleget_arg_bindings(dag_bag, ti, session=session)call, which the new gate short-circuits.test_ti_run_skips_queued_duration_metric_when_the_response_failssetsti.is_stub = True. Here the gate short-circuits the mockedget_arg_bindingswithout it; onmainthe gate only decides whether the task is loaded, so the line is not needed there.Everything else, including the SDK change, the tests and both fixture fixes, is the same.
Unrelated: one red test on this branch
test_find_executable_task_instances_skips_dag_that_cannot_be_deserializedfails onv3-4-testwithout this backport, at the branch tip (run 40591272593 on905a81976cc). #75745 brought the guard over from #75263 without the starved set #75263 relies on, since that set came with #70307, which is not on this branch. The guard then returnsFalsewithout narrowing the next query, so a cycle whose batch is filled by the bad Dag ends with nothing queued.The last commit here starves the Dag instead, which is what the test asserts and what the existing
starved_dagsset is already wired for. Happy to split it into its own PR if that reads better.Was generative AI tooling used to co-author this PR?