Skip to content

Parse native Lang-SDK Dags in a Dag bag - #74043

Merged
jason810496 merged 7 commits into
jason/lang-sdk-e2e/03-native-dag-parsefrom
jason/lang-sdk-e2e/03e-lang-sdk-dag-bag
Oct 5, 2026
Merged

Parse native Lang-SDK Dags in a Dag bag#74043
jason810496 merged 7 commits into
jason/lang-sdk-e2e/03-native-dag-parsefrom
jason/lang-sdk-e2e/03e-lang-sdk-dag-bag

Conversation

@jason810496

@jason810496 jason810496 commented Oct 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Stack (bottom to top): #74042, #74035, #74043, #74036, #74037, #73845, #73846, #73847

Why

CLI commands and airflow dags reserialize load Dags through a Dag bag, which cannot parse a native Lang-SDK Dag file yet.

What changes

  • DagBag(parse_lang_sdk_files=True) runs a coordinator-claimed file with the same LangSDKDagFileProcessorProcess as the Dag processor, with the same fill and validation, and bags each Dag as the SerializedDAG the scheduler loads. The Dag importer does not parse it. A Dag bag has no API client, so a runtime request other than the parse result or MaskSecret gets an error.
  • Without it, which is the default, such a file is an import error and no runtime starts.
  • A Dag bag holds BaggedDAG = DAG | SerializedDAG. It keeps its duplicate id check for a native Dag and skips the SDK-only checks and cluster policies.
  • sync_bag_to_db does not store a native Dag: the Dag processor does.
  • airflow tasks list lists a native Dag's tasks. dags test, tasks test and tasks render do not parse a native Dag, and the Python processor refuses its callbacks.
  • The recorded TypeScript and Java runtime payloads must pass validate_serialized_dag.
airflow tasks list typescript_native_example

Deviations from #71929

  • A Dag bag holds a native Dag as a SerializedDAG, not an airflow.sdk.DAG.
Was generative AI tooling used to co-author this PR?

@boring-cyborg boring-cyborg Bot added area:CLI area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:DAG-processing area:task-sdk labels Oct 1, 2026
This was referenced Oct 1, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Closed
Closed
Closed
Merged
@jason810496
jason810496 added this pull request to stack #74044 October 1, 2026 15:18
kaxil
kaxil reviewed Oct 1, 2026
View reviewed changes
Comment thread task-sdk/src/airflow/sdk/importers/base.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/_dag_importer.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/_dag_importer.py
Comment thread airflow-core/src/airflow/dag_processing/dagbag.py Outdated
Comment thread airflow-core/src/airflow/utils/cli.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/_dag_importer.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/_dag_importer.py Outdated
Comment thread airflow-core/tests/unit/dag_processing/test_dagbag.py Outdated
Comment thread task-sdk/tests/task_sdk/coordinators/test_dag_importer.py Outdated
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch 2 times, most recently from 64b7eff to bd89a18 Compare October 2, 2026 04:07
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch from bd89a18 to 96f2076 Compare October 2, 2026 05:44
@pierrejeambrun
pierrejeambrun force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch from 96f2076 to 6fea2b4 Compare October 2, 2026 09:00
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch from 6fea2b4 to 6c3ae5d Compare October 2, 2026 11:41
jason810496
jason810496 commented Oct 2, 2026
View reviewed changes

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Comment thread task-sdk/src/airflow/sdk/coordinators/_dag_importer.py Outdated
Comment thread task-sdk/src/airflow/sdk/importers/base.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/_dag_importer.py
Comment thread airflow-core/src/airflow/dag_processing/dagbag.py Outdated
Comment thread airflow-core/src/airflow/utils/cli.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/_dag_importer.py Outdated
Comment thread task-sdk/src/airflow/sdk/coordinators/_dag_importer.py Outdated
Comment thread airflow-core/tests/unit/dag_processing/test_dagbag.py Outdated
Comment thread task-sdk/tests/task_sdk/coordinators/test_dag_importer.py Outdated
@jason810496
jason810496 marked this pull request as ready for review October 2, 2026 13:59
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch from 964f50b to fe5a99c Compare October 3, 2026 17:59
@jason810496
jason810496 added this pull request to stack #74170 October 3, 2026 17:59
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch from fe5a99c to 29de12f Compare October 4, 2026 01:06
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch from 29de12f to 1724f56 Compare October 4, 2026 08:52
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch from 1724f56 to abfadb4 Compare October 5, 2026 04:53
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch 2 times, most recently from 471491d to 77a24d4 Compare October 5, 2026 09:12
pierrejeambrun
pierrejeambrun approved these changes Oct 5, 2026
View reviewed changes

@pierrejeambrun pierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beside the is_ready issues that cascade from previous PR.

Looks good.

Comment thread airflow-core/src/airflow/dag_processing/lang_sdk_processor.py
@jason810496
Parse coordinator-claimed Dag files with their runtime in a Dag bag
adc8a11 
CoordinatorDagImporter.import_definition now runs the same
LangSDKDagFileProcessorProcess as the Dag processor, for a Dag bag such
as a CLI command's, and returns each Dag as a SerializedLangSDKDAG. A
Dag bag has no API client, so the runtime's requests get an error there.
The Dag bag skips the SDK-only checks and cluster policies for such a
Dag, sync_bag_to_db does not store its Dags, and CLI commands that run
a Dag refuse it.

The recorded TypeScript and Java runtime payloads must pass
validate_serialized_dag.
@jason810496
Let a Dag file processor run without an API client
6c7606e 
A Dag bag parses a Lang-SDK file outside the Dag processor, where
there is no API client. Make the client optional on the base Dag file
processor, which answers each request that needs it with an error,
and add LangSDKDagFileProcessorProcess.run() to parse one file and
wait for the result.
@jason810496
Bag native Lang-SDK Dags in core, only when asked
d6d9f16 
DagImportResult.dags stays list[DAG], and the coordinator importer
goes back to reporting that only the Dag processor parses its files.
A Dag bag built with parse_lang_sdk_files=True runs the file's runtime
itself and bags each Dag as a SerializedDAG, checked only for a
duplicate id. Without it, as for dags reserialize and DAG.test(), such
a file stays a cheap import error. get_bagged_dag takes
allow_lang_sdk_dag to pass it on.

All runtime import errors of a file are joined into one message, since
the bag keeps one error per file.
@jason810496
List the tasks of a native Lang-SDK Dag
124cd3b 
tasks list only prints the ids in dag.tasks, which a SerializedDAG
has, so it asks get_bagged_dag to parse native Lang-SDK files too.
@jason810496
Ignore the bagged Dag union in the external task sensor test
c9f3b5c 
A Dag bag can now hold a SerializedDAG, so the Airflow 2 branch that
calls create_dagrun on a bagged Dag needs union-attr ignored too.
@jason810496
Let a Dag file processor run without a parse log file
72c6990 
A Dag bag parses a Lang-SDK file with no parse log, so the base allows
none. DagFileProcessorProcess still requires the handle and the client.
@jason810496
Default a Dag file processor's parse log to os.devnull
e88645e 
Subclasses no longer check logger_filehandle for None before closing it.
DagFileProcessorProcess drops its own declaration of the attribute and
inherits the default.
@jason810496
jason810496 force-pushed the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch from 77a24d4 to e88645e Compare October 5, 2026 10:41
jason810496
jason810496 commented Oct 5, 2026
View reviewed changes

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review

Comment thread airflow-core/src/airflow/dag_processing/lang_sdk_processor.py
@jason810496
jason810496 merged commit cffc14d into main Oct 5, 2026
89 checks passed
@jason810496
jason810496 deleted the jason/lang-sdk-e2e/03e-lang-sdk-dag-bag branch October 5, 2026 11:39
This was referenced Oct 5, 2026
Draft
Draft
Draft
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pierrejeambrun pierrejeambrun pierrejeambrun approved these changes
@bugraoz93 bugraoz93 Awaiting requested review from bugraoz93 bugraoz93 is a code owner
@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner
@dheerajturaga dheerajturaga Awaiting requested review from dheerajturaga dheerajturaga is a code owner
@henry3260 henry3260 Awaiting requested review from henry3260 henry3260 is a code owner
@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner
@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner
@ashb ashb Awaiting requested review from ashb ashb is a code owner
@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin bolkedebruin is a code owner
@kaxil kaxil Awaiting requested review from kaxil
@uranusjr uranusjr Awaiting requested review from uranusjr
@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh
@gopidesupavan gopidesupavan Awaiting requested review from gopidesupavan
@jscheffl jscheffl Awaiting requested review from jscheffl

Assignees

@jason810496 jason810496

Labels

area:CLI area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:DAG-processing area:task-sdk

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jason810496 @kaxil @pierrejeambrun