[jason810496]

• closes: Performance regression: The execution API load serialized Dag for every task #75791
• related: Enforce execution_timeout for tasks run by a language SDK #74869, Send the Python task's execution_timeout to the supervisor #75387
• needs backport-to-v3-4-test before 3.4.0 RC.

Why

Starting any task deserialized its whole Dag in Execution API, so that ti_run could find out whether the task was a stub and read its arg_bindings. On a Dag cache miss that is a full serialized_dag read plus deserialization per task start, paid while the route holds SELECT ... FOR UPDATE on the task_instance row, and for a Python task the result is thrown away.

How

• The serialized is_stub flag is denormalized onto task_instance table for the fast lookup.

• The route loads the task's definition only when the flag is set:

  serialized_task = get_task(dag_bag, ti, session=session) if ti.is_stub else None

• Null means "not a stub", with no fallback. Every null row predates the migration, so it was written by 3.3, which has no arg_bindings at all, and such a task behaves exactly as it did there.

For the release manager

main has moved past the 3.4.0 chain head, and a 3.4.0 revision cannot sit after 3.5.0 ones or db migrate --to-version 3.4.0 would stop short of it. The new revision is therefore inserted, and the two 3.5.0 revisions after it are renumbered:

ee86eed19e24  0148_3_4_0_add_pending_partition_key_to_apdr
b4f1a7d2c903  0149_3_4_0_add_is_stub_to_task_instance        <- new
c71065a1b14f  0150_3_5_0_add_session_id_to_job               <- was 0149, down_revision rewritten
38c8df3fcb9c  0151_3_5_0_add_dag_run_dag_id_run_after_index  <- was 0150, unchanged

Note

• We should remove the execution_timeout resolution that rely on the serialized Dag and refactor that e2e path anyway, that is keep as out of scope on main. Send the Python task's execution_timeout to the supervisor #75387 makes permanent by having the Python task runner send its own timeout to the supervisor. Until it merges, a stuck Python task has no supervisor backstop on main; the runner's own AirflowTaskTimeout still fires. 3.4.0 is unaffected, having no TIRunContext.execution_timeout at all ([v3-4-test] Revert "Enforce execution_timeout for tasks run by a language SDK" (#75376) #75469).
• No SDK changes: TypeScript, Java and Go already write is_stub for Dags defined in them.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code Opus 5 following the guidelines

[ephraimbuddy]

Nits: I think there's no need to have the TABLE_NAME variable. non-blocking

[ashb]

More generally, I'm not sure gating this on "is_stub" is right. This is needed for an go task isn't it? I think we defined a stub task as one written in a language other than the one the dag is defined in. But that isn't the only place we need the arg_binding, is it?

[jason810496]

This is needed for an go task isn't it?

Yes.

I think we defined a stub task as one written in a language other than the one the dag is defined in. But that isn't the only place we need the arg_binding, is it?

Actually, both Python Dag with Stub task and the task define in non-Python Dag will set the "is_stub" as true and the arg_binding field.

[jason810496]

Thanks all for the review.

[jason810496]

  File "/usr/python/lib/python3.11/site-packages/requests/api.py", line 71, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.11/site-packages/requests/sessions.py", line 651, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.11/site-packages/requests/sessions.py", line 784, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.11/site-packages/requests/adapters.py", line 727, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/apache-airflow-providers-common-compat/json (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1016)')))

Flaky CI, merging the PR now.