From 7f8c702c24d8883a19cb76a8f43fc7c58f5209ad Mon Sep 17 00:00:00 2001 From: LIU ZHE YOU Date: Fri, 21 Aug 2026 09:38:15 +0000 Subject: [PATCH 1/9] Add ADR-0010 for mixed-language Dag processing flow --- .../0010-mixed-language-dag-processing.md | 233 ++++++++++++++++++ airflow-core/adr/lang-sdk/README.md | 1 + 2 files changed, 234 insertions(+) create mode 100644 airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md diff --git a/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md new file mode 100644 index 0000000000000..e9c7409643350 --- /dev/null +++ b/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md @@ -0,0 +1,233 @@ + + +# ADR-0010: Mixed-Language Dag Processing — DagImporter Routing and Persistence + +## Status + +Proposed + +## Context + +A Lang-SDK artifact (JAR, packed Go executable) can serve two roles: + +| Role | Dag defined in | Artifact contributes | +|-------------------------|-------------------------------------|---------------------------| +| **Mixed-language Dag** | Python file with `@task.stub` tasks | Only task implementations | +| **Native Lang-SDK Dag** | The Go/Java source itself | The entire Dag | + +Persisting both the Python Dag and the Lang-SDK Dag for the same `dag_id` +creates conflicting definitions. The `is_mixed_language_dag` flag +([PR #71213](https://github.com/apache/airflow/pull/71213)) disambiguates them. + +This ADR uses `JavaDagImporter` as the concrete example. `ExecutableDagImporter` +(Go) and `NodeDagImporter` (TypeScript) follow the same flow. A common base +class is expected once the second Lang-SDK importer lands. + +## Decision + +### The Flag + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Per-task: is_stub │ +│ AbstractOperator.is_stub = False (default) │ +│ _StubOperator.is_stub = True │ +│ │ +│ Per-Dag: is_mixed_language_dag │ +│ Python side — derived at serialization: │ +│ if any(task.is_stub for task in dag.tasks): True │ +│ Lang-SDK side — set explicitly by the producer │ +└──────────────────────────────────────────────────────────────┘ +``` + +Both sides carry `is_mixed_language_dag: true`, so the flag alone cannot decide +persistence. The decision is `(flag × importer)`. + +### Processing Flows + +#### Flow A — Pure Python Dag (no `@task.stub`) + +``` +PythonDagImporter.import_definition(definition, bundle=...) + │ + ├── Parse → DAG objects + ├── serialize_dag(dag) → is_mixed_language_dag absent + ├── Return DagImportResult(dags=[dag]) + ▼ +DagModelOperation → PERSIST +``` + +#### Flow B — Lang-SDK Importer (native and mixed) + +`JavaDagImporter` **never** persists a Dag with `is_mixed_language_dag: true`. + +``` +JavaDagImporter.import_definition(definition, bundle=...) + │ + ├── BaseCoordinator.run_dag_parsing() + │ └── Spawn JVM → DagFileParsingResult + │ ┌─────────────────────────────────────────┐ + │ │ dag_id: "etl" │ + │ │ is_mixed_language_dag: true │ + │ ├─────────────────────────────────────────┤ + │ │ dag_id: "java_report" │ + │ │ is_mixed_language_dag: false │ + │ └─────────────────────────────────────────┘ + │ + ├── "etl" is_mixed=true → DISCARD + log info + ├── "java_report" is_mixed=false → PERSIST + │ + ├── Return DagImportResult(dags=["java_report"]) + ▼ +DagModelOperation → PERSIST "java_report" only +``` + +#### Flow C — Mixed-Language Dag (validation driven by PythonDagImporter) + +The `PythonDagImporter` owns validation. When it finds `is_mixed_language_dag: +true`, it resolves each stub task's `queue` to its Coordinator, then uses the +Coordinator's DagImporter to fetch the Lang-SDK Dag structure for +cross-validation. + +Resolution goes through the coordinator registry, not the filesystem — the Dag +processor is file-at-a-time and the `PythonDagImporter` never sees the `.jar`. +This also means the Python Dag and the Lang-SDK artifact **do not need to be in +the same DagBundle**. + +``` +PythonDagImporter.import_definition(definition, bundle=...) + │ + ├── Parse → DAG objects + ├── serialize_dag(dag) → is_mixed_language_dag = true + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 1: Resolve stub tasks → Coordinator instances via queue │ + │ │ │ + │ │ stub task queue Coordinator instance │ + │ │ ────────────────────────────────────────────────── │ + │ │ extract → "jdk-11" → JavaCoordinator(name="jdk-11") │ + │ │ transform → "jdk-17" → JavaCoordinator(name="jdk-17") │ + │ │ load → "jdk-11" → JavaCoordinator(name="jdk-11") │ + │ │ │ + │ │ Deduplicate → 2 distinct coordinator instances: │ + │ │ JavaCoordinator(name="jdk-11") │ + │ │ JavaCoordinator(name="jdk-17") │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 2: For each Coordinator, get its DagImporter and │ + │ │ list_dag_definitions to find the matching Dag │ + │ │ │ + │ │ JavaCoordinator(name="jdk-11") │ + │ │ │ │ + │ │ ├── Get artifact root: │ + │ │ │ coordinator.dag_bundle │ + │ │ │ → BaseDagBundle for "java-jdk11-bundle" │ + │ │ │ OR coordinator.artifact_roots │ + │ │ │ → [Path("/opt/airflow/jars/jdk11/")] │ + │ │ │ │ + │ │ ├── JavaDagImporter.list_dag_definitions( │ + │ │ │ bundle=coordinator.dag_bundle, │ + │ │ │ safe_mode=True) │ + │ │ │ → Iterator[DagDefinition] │ + │ │ │ DagDefinition("app-jdk11.jar") │ + │ │ │ DagDefinition("utils-jdk11.jar") │ + │ │ │ ... │ + │ │ │ │ + │ │ └── JavaDagImporter.import_definition( │ + │ │ definition, │ + │ │ bundle=coordinator.dag_bundle, │ + │ │ discover_mixed_language_dags=True) │ + │ │ → DagImportResult with Dag structure for dag_id="etl" │ + │ │ │ + │ │ JavaCoordinator(name="jdk-17") │ + │ │ └── (same flow, different artifact root / bundle) │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 3: Validate stub tasks against Lang-SDK tasks │ + │ │ │ + │ │ Python Dag "etl" Lang-SDK Dag "etl" │ + │ │ stub tasks: tasks: │ + │ │ extract ───────────── extract ✓ match │ + │ │ transform ─────────── transform ✓ match │ + │ │ load ──────────────── load ✓ match │ + │ │ │ + │ │ On mismatch → DagImportError │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + ├── Validation passed → Return DagImportResult(dags=[dag]) + ▼ +DagModelOperation → PERSIST (Python Dag is the sole DB record) +``` + +### Decision Matrix + +``` +┌───────────────────┬───────────────────────┬──────────────────────────────────────────────────────────────┐ +│ Importer │ is_mixed_language_dag │ Action │ +├───────────────────┼───────────────────────┼──────────────────────────────────────────────────────────────┤ +│ PythonDagImporter │ false │ PERSIST │ +├───────────────────┼───────────────────────┼──────────────────────────────────────────────────────────────┤ +│ PythonDagImporter │ true │ PERSIST + VALIDATE via queue → Coordinator → │ +│ │ │ DagImporter.list_dag_definitions + import_definition │ +├───────────────────┼───────────────────────┼──────────────────────────────────────────────────────────────┤ +│ JavaDagImporter │ false (or absent) │ PERSIST │ +├───────────────────┼───────────────────────┼──────────────────────────────────────────────────────────────┤ +│ JavaDagImporter │ true │ DISCARD + log info │ +└───────────────────┴───────────────────────┴──────────────────────────────────────────────────────────────┘ +``` + +### Why the Flag Lives on the Dag, Not the Bundle + +A single artifact can contain both native and mixed-language Dags: + +``` +analytics.jar +├── dag_id="etl" is_mixed_language_dag=true (backs etl.py) +└── dag_id="java_report" is_mixed_language_dag=false (native) +``` + +The importer decides per-Dag, not per-file or per-bundle. + +## Consequences + +- **Python leads, Lang-SDK follows.** `PythonDagImporter` persists the Dag and + drives validation via `queue → Coordinator → DagImporter.list_dag_definitions + → import_definition`. Lang-SDK importers silently discard mixed-language Dags. +- Stub/implementation mismatches surface as `DagImportError` at parse time. +- The Python Dag and Lang-SDK artifact can live in different DagBundles — + resolution goes through the coordinator registry, not the filesystem. +- A single Dag can have stubs targeting different queues (some Java, some Go) — + each resolves to its own coordinator instance independently. +- **Mixed-language is Python-primary only.** Lang-SDK runtimes cannot define stub + operators — the reverse direction (a native Lang-SDK Dag delegating tasks to + Python) is not supported. +- No schema migration, no new `DagModel` column, no REST/UI change. + +## References + +- [ADR-0004](0004-dag-parsing.md) — `BaseCoordinator` / `DagFileParsingResult` +- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display + for mixed-language Dags +- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs +- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter +- [PR #71213](https://github.com/apache/airflow/pull/71213) — the + `is_mixed_language_dag` flag diff --git a/airflow-core/adr/lang-sdk/README.md b/airflow-core/adr/lang-sdk/README.md index 359275ac2696b..b96437fe1ac00 100644 --- a/airflow-core/adr/lang-sdk/README.md +++ b/airflow-core/adr/lang-sdk/README.md @@ -35,6 +35,7 @@ bind core interfaces and apply to every language SDK, not just the Java SDK. - [ADR-0007](0007-taskflow-across-language-boundary.md): TaskFlow across the language boundary — argument binding for Lang-SDK tasks. - [ADR-0008](0008-control-flow-constructs.md): control-flow constructs — grouping, conditional skipping, branching, and triggering a Dag run. - [ADR-0009](0009-provider-operators-as-generated-dsl.md): provider operators as generated, serialization-only DSL in every Lang SDK. +- [ADR-0010](0010-mixed-language-dag-processing.md): mixed-language Dag processing — DagImporter routing and persistence. Decisions specific to a single SDK stay next to that SDK — for example, the Go SDK's bundle-format decisions live in [`go-sdk/adr/`](../../../go-sdk/adr). Java-SDK-only interface-design decisions — From 4a82a239a3d46c1c97cf7ec1fd19c259696b97ae Mon Sep 17 00:00:00 2001 From: LIU ZHE YOU Date: Tue, 25 Aug 2026 16:27:01 +0800 Subject: [PATCH 2/9] Remove the usage of JavaDagImporter and the Dag-level is_mixed_language flag --- .../0010-mixed-language-dag-processing.md | 184 ++++++++++-------- 1 file changed, 105 insertions(+), 79 deletions(-) diff --git a/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md index e9c7409643350..6eb6b34ceb295 100644 --- a/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md @@ -27,22 +27,25 @@ Proposed A Lang-SDK artifact (JAR, packed Go executable) can serve two roles: -| Role | Dag defined in | Artifact contributes | -|-------------------------|-------------------------------------|---------------------------| -| **Mixed-language Dag** | Python file with `@task.stub` tasks | Only task implementations | -| **Native Lang-SDK Dag** | The Go/Java source itself | The entire Dag | +| Role | Dag defined in | Author writes | Artifact contributes | +|-------------------------|-------------------------------------|-------------------|----------------------------| +| **Mixed-language Dag** | Python file with `@task.stub` tasks | `@MixedLangDag` | Only task implementations | +| **Native Lang-SDK Dag** | The Go/Java source itself | `@Dag` | The entire Dag | Persisting both the Python Dag and the Lang-SDK Dag for the same `dag_id` -creates conflicting definitions. The `is_mixed_language_dag` flag -([PR #71213](https://github.com/apache/airflow/pull/71213)) disambiguates them. +creates conflicting definitions. [PR #71213](https://github.com/apache/airflow/pull/71213) +disambiguated them with a per-Dag `is_mixed_language_dag` flag. This ADR +replaces that flag with an Dag authoring level distinction — `@Dag` vs. +`@MixedLangDag` in the case of Java SDK — plus a `DagFileParsingRequest` field, for the reasons below. -This ADR uses `JavaDagImporter` as the concrete example. `ExecutableDagImporter` -(Go) and `NodeDagImporter` (TypeScript) follow the same flow. A common base -class is expected once the second Lang-SDK importer lands. +This ADR uses `JavaDagImporter` and `JavaCoordinator` as the concrete examples. +`ExecutableDagImporter` (Go) and `NodeDagImporter` (TypeScript) follow the same +flow, each backed by its own `BaseCoordinator` subclass. A common base class is +expected once the second Lang-SDK importer lands. ## Decision -### The Flag +### The Annotation, Not a Flag ``` ┌──────────────────────────────────────────────────────────────┐ @@ -50,15 +53,21 @@ class is expected once the second Lang-SDK importer lands. │ AbstractOperator.is_stub = False (default) │ │ _StubOperator.is_stub = True │ │ │ -│ Per-Dag: is_mixed_language_dag │ -│ Python side — derived at serialization: │ -│ if any(task.is_stub for task in dag.tasks): True │ -│ Lang-SDK side — set explicitly by the producer │ +│ Per-Dag: which annotation the Lang-SDK author applied │ +│ @Dag — native, the entire Dag (existing) │ +│ @MixedLangDag — tasks only, backs a Python (new) │ +│ file's `@task.stub` operators │ └──────────────────────────────────────────────────────────────┘ ``` -Both sides carry `is_mixed_language_dag: true`, so the flag alone cannot decide -persistence. The decision is `(flag × importer)`. +`@Dag` (`Builder.Dag`) and `@MixedLangDag` (`Builder.MixedLangDag`, new) drive +the same `BuilderProcessor` annotation processor +([ADR-0003](0003-pure-java-dags.md)) — each generates a `*Builder` whose +`build()` still returns a `DagDef`; `BundleBuilder.getDags(): Iterable` +is unchanged. What differs is which annotation produced it, recorded on the +generated builder itself, so the JVM's parsing entrypoint tells native and +mixed-language Dags apart without a field carried on the Dag object. The +decision is `(annotation × importer)`. ### Processing Flows @@ -68,7 +77,7 @@ persistence. The decision is `(flag × importer)`. PythonDagImporter.import_definition(definition, bundle=...) │ ├── Parse → DAG objects - ├── serialize_dag(dag) → is_mixed_language_dag absent + ├── serialize_dag(dag) → no stub tasks, nothing to cross-validate ├── Return DagImportResult(dags=[dag]) ▼ DagModelOperation → PERSIST @@ -76,23 +85,18 @@ DagModelOperation → PERSIST #### Flow B — Lang-SDK Importer (native and mixed) -`JavaDagImporter` **never** persists a Dag with `is_mixed_language_dag: true`. +`JavaDagImporter` **never** persists a Dag produced by `@MixedLangDag`. ``` JavaDagImporter.import_definition(definition, bundle=...) │ - ├── BaseCoordinator.run_dag_parsing() - │ └── Spawn JVM → DagFileParsingResult + ├── BaseCoordinator.run_dag_parsing( + │ request=DagFileParseRequest(file=..., mixed_language_dags_only=False)) + │ └── Spawn JVM → JVM filters locally: only @Dag-produced entries are serialized │ ┌─────────────────────────────────────────┐ - │ │ dag_id: "etl" │ - │ │ is_mixed_language_dag: true │ - │ ├─────────────────────────────────────────┤ - │ │ dag_id: "java_report" │ - │ │ is_mixed_language_dag: false │ + │ │ dag_id: "java_report" │ (@Dag) │ └─────────────────────────────────────────┘ - │ - ├── "etl" is_mixed=true → DISCARD + log info - ├── "java_report" is_mixed=false → PERSIST + │ "etl" (@MixedLangDag) never leaves the JVM — nothing to discard. │ ├── Return DagImportResult(dags=["java_report"]) ▼ @@ -101,10 +105,14 @@ DagModelOperation → PERSIST "java_report" only #### Flow C — Mixed-Language Dag (validation driven by PythonDagImporter) -The `PythonDagImporter` owns validation. When it finds `is_mixed_language_dag: -true`, it resolves each stub task's `queue` to its Coordinator, then uses the -Coordinator's DagImporter to fetch the Lang-SDK Dag structure for -cross-validation. +The `PythonDagImporter` owns validation. When it finds a stub task, it +resolves the stub's `queue` to its Coordinator, then asks that Coordinator +**directly** for the `@MixedLangDag`-produced Dag structure backing the same +`dag_id` — no `DagImporter` involved, since this is a single parse +request/response, not a discovery operation. Either path bottoms out at the +Coordinator sending a `DagFileParseRequest` and getting back a +`DagFileParsingResult` ([ADR-0004](0004-dag-parsing.md)), so validation calls +that primitive directly instead of detouring through the importer contract. Resolution goes through the coordinator registry, not the filesystem — the Dag processor is file-at-a-time and the `PythonDagImporter` never sees the `.jar`. @@ -115,7 +123,7 @@ the same DagBundle**. PythonDagImporter.import_definition(definition, bundle=...) │ ├── Parse → DAG objects - ├── serialize_dag(dag) → is_mixed_language_dag = true + ├── serialize_dag(dag) → is_stub tasks carry arg_bindings (ADR-0007) │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ Step 1: Resolve stub tasks → Coordinator instances via queue │ @@ -132,43 +140,53 @@ PythonDagImporter.import_definition(definition, bundle=...) │ └─────────────────────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 2: For each Coordinator, get its DagImporter and │ - │ │ list_dag_definitions to find the matching Dag │ + │ │ Step 2: Locate the file backing dag_id="etl", no JVM launch — │ + │ │ reuse BundleScanner (ADR-0003), which reads JAR manifests │ + │ │ directly │ │ │ │ │ │ JavaCoordinator(name="jdk-11") │ - │ │ │ │ │ │ ├── Get artifact root: │ │ │ │ coordinator.dag_bundle │ │ │ │ → BaseDagBundle for "java-jdk11-bundle" │ │ │ │ OR coordinator.artifact_roots │ │ │ │ → [Path("/opt/airflow/jars/jdk11/")] │ │ │ │ │ - │ │ ├── JavaDagImporter.list_dag_definitions( │ - │ │ │ bundle=coordinator.dag_bundle, │ - │ │ │ safe_mode=True) │ - │ │ │ → Iterator[DagDefinition] │ - │ │ │ DagDefinition("app-jdk11.jar") │ - │ │ │ DagDefinition("utils-jdk11.jar") │ - │ │ │ ... │ + │ │ └── BundleScanner.scanBundles(artifact_root) │ + │ │ → Map │ + │ │ → "etl" → ResolvedBundle(mainClass=..., classpath=...) │ + │ │ │ + │ │ JavaCoordinator(name="jdk-17") │ + │ │ └── (same lookup, different artifact root) │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 3: Call the Coordinator directly — the same primitive │ + │ │ ADR-0004 already defines for ordinary parsing │ + │ │ │ + │ │ JavaCoordinator(name="jdk-11").run_dag_parsing( │ + │ │ path=resolved.path, bundle_name=..., bundle_path=..., │ + │ │ request=DagFileParseRequest( │ + │ │ file=resolved.path, │ + │ │ mixed_language_dags_only=True)) │ │ │ │ │ - │ │ └── JavaDagImporter.import_definition( │ - │ │ definition, │ - │ │ bundle=coordinator.dag_bundle, │ - │ │ discover_mixed_language_dags=True) │ - │ │ → DagImportResult with Dag structure for dag_id="etl" │ + │ │ ├── Spawn JVM → sends DagFileParseRequest over the bridge │ + │ │ ├── JVM filters locally: only @MixedLangDag entries serialized │ + │ │ ▼ │ + │ │ DagFileParsingResult → deserialize → LazyDeserializedDAG "etl" │ │ │ │ │ │ JavaCoordinator(name="jdk-17") │ - │ │ └── (same flow, different artifact root / bundle) │ + │ │ └── (same call, different resolved path) │ │ └─────────────────────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 3: Validate stub tasks against Lang-SDK tasks │ + │ │ Step 4: Compare directly against the parsed Python Dag — the │ + │ │ same LazyDeserializedDAG model the scheduler reads │ │ │ │ - │ │ Python Dag "etl" Lang-SDK Dag "etl" │ - │ │ stub tasks: tasks: │ - │ │ extract ───────────── extract ✓ match │ - │ │ transform ─────────── transform ✓ match │ - │ │ load ──────────────── load ✓ match │ + │ │ Python Dag "etl" (stub tasks) LazyDeserializedDAG "etl" │ + │ │ ─────────────────────────────── ───────────────────────── │ + │ │ task_id ↔ task_id (sets must match)│ + │ │ arg_bindings[*].name ↔ declared param name │ + │ │ arg_bindings[*].value_schema ↔ declared param schema │ │ │ │ │ │ On mismatch → DagImportError │ │ └─────────────────────────────────────────────────────────────────────┘ @@ -180,39 +198,37 @@ DagModelOperation → PERSIST (Python Dag is the sole DB record) ### Decision Matrix -``` -┌───────────────────┬───────────────────────┬──────────────────────────────────────────────────────────────┐ -│ Importer │ is_mixed_language_dag │ Action │ -├───────────────────┼───────────────────────┼──────────────────────────────────────────────────────────────┤ -│ PythonDagImporter │ false │ PERSIST │ -├───────────────────┼───────────────────────┼──────────────────────────────────────────────────────────────┤ -│ PythonDagImporter │ true │ PERSIST + VALIDATE via queue → Coordinator → │ -│ │ │ DagImporter.list_dag_definitions + import_definition │ -├───────────────────┼───────────────────────┼──────────────────────────────────────────────────────────────┤ -│ JavaDagImporter │ false (or absent) │ PERSIST │ -├───────────────────┼───────────────────────┼──────────────────────────────────────────────────────────────┤ -│ JavaDagImporter │ true │ DISCARD + log info │ -└───────────────────┴───────────────────────┴──────────────────────────────────────────────────────────────┘ -``` +| Caller | Produced by | `mixed_language_dags_only` | Action | +|------------------------------------------------|------------------------|------------------------------|---------------------------------------------------------------------| +| `PythonDagImporter` | its own parsed Dag | n/a | PERSIST | +| `PythonDagImporter` (via stub → Coordinator) | `@MixedLangDag` | `True` | VALIDATE only — compared against the Python Dag, never persisted | +| `JavaDagImporter` | `@Dag` | `False` (default) | PERSIST | +| `JavaDagImporter` | `@MixedLangDag` | `False` (default) | never returned — filtered inside the JVM | -### Why the Flag Lives on the Dag, Not the Bundle +### Why the Distinction Lives on the Annotation, Not the Bundle A single artifact can contain both native and mixed-language Dags: ``` -analytics.jar -├── dag_id="etl" is_mixed_language_dag=true (backs etl.py) -└── dag_id="java_report" is_mixed_language_dag=false (native) +analytics.jar (one BundleBuilder.getDags() call) +├── EtlTasks @MixedLangDag(id = "etl") (backs etl.py's stub tasks) +└── JavaReportPipeline @Dag(id = "java_report") (native) ``` -The importer decides per-Dag, not per-file or per-bundle. +The importer decides per-Dag, not per-file or per-bundle — and it decides by +which annotation authored each class, not by inspecting a field on the +resulting Dag. ## Consequences - **Python leads, Lang-SDK follows.** `PythonDagImporter` persists the Dag and - drives validation via `queue → Coordinator → DagImporter.list_dag_definitions - → import_definition`. Lang-SDK importers silently discard mixed-language Dags. -- Stub/implementation mismatches surface as `DagImportError` at parse time. + drives validation via `queue → Coordinator → run_dag_parsing + (mixed_language_dags_only=True)`, comparing the resulting + `LazyDeserializedDAG` directly against its own parsed Dag. Lang-SDK importers + never receive an `@MixedLangDag`-produced Dag to begin with — the JVM + filters it out of the response before anything crosses the wire. +- Stub/implementation mismatches (missing task, extra task, parameter name or + schema mismatch) surface as `DagImportError` at parse time. - The Python Dag and Lang-SDK artifact can live in different DagBundles — resolution goes through the coordinator registry, not the filesystem. - A single Dag can have stubs targeting different queues (some Java, some Go) — @@ -220,14 +236,24 @@ The importer decides per-Dag, not per-file or per-bundle. - **Mixed-language is Python-primary only.** Lang-SDK runtimes cannot define stub operators — the reverse direction (a native Lang-SDK Dag delegating tasks to Python) is not supported. +- **No `is_mixed_language_dag` flag.** This ADR supersedes that part of + [PR #71213](https://github.com/apache/airflow/pull/71213): the role is + encoded in which annotation authored the class (`@Dag` vs. + `@MixedLangDag`) and in the `DagFileParsingRequest.mixed_language_dags_only` + field, not in a value carried on the serialized Dag itself. - No schema migration, no new `DagModel` column, no REST/UI change. ## References +- [ADR-0003](0003-pure-java-dags.md) — `BundleBuilder` / `BundleScanner` / + `BuilderProcessor` (`@Dag` / `@Task` annotation processing) - [ADR-0004](0004-dag-parsing.md) — `BaseCoordinator` / `DagFileParsingResult` - [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display for mixed-language Dags +- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / + `TaskArgBinding`, compared against `@MixedLangDag` task signatures in Flow C - [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs - [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter -- [PR #71213](https://github.com/apache/airflow/pull/71213) — the - `is_mixed_language_dag` flag +- [PR #71213](https://github.com/apache/airflow/pull/71213) — introduced the + per-Dag `is_mixed_language_dag` flag, superseded by this ADR's type-level + distinction From 0bd815cf1d4bc675bec46440947a6fd8754ea8a1 Mon Sep 17 00:00:00 2001 From: LIU ZHE YOU Date: Sun, 13 Sep 2026 04:01:06 +0000 Subject: [PATCH 3/9] Define mixed-language task handlers as a separate interface from Dags A Lang-SDK artifact backing `@task.stub` tasks used to author a Dag under the same dag_id the Python file already owns, so two conflicting definitions reached persistence and something downstream had to pick between them. Removing the Dag from the authoring interface answers that once, rather than leaving every consumer of a serialized Dag to ask whether the Dag in front of it is real. Terms follow the Language SDK spec so the decision reads the same in Go, Java, and TypeScript, and the parse-side contract is grounded in the shipped coordinator and supervisor-schema code rather than in a proposed shape. --- .../0010-mixed-language-dag-processing.md | 522 ++++++++++++++---- 1 file changed, 405 insertions(+), 117 deletions(-) diff --git a/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md index 6eb6b34ceb295..fcd647a00ef95 100644 --- a/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md @@ -25,49 +25,289 @@ Proposed ## Context -A Lang-SDK artifact (JAR, packed Go executable) can serve two roles: - -| Role | Dag defined in | Author writes | Artifact contributes | -|-------------------------|-------------------------------------|-------------------|----------------------------| -| **Mixed-language Dag** | Python file with `@task.stub` tasks | `@MixedLangDag` | Only task implementations | -| **Native Lang-SDK Dag** | The Go/Java source itself | `@Dag` | The entire Dag | - -Persisting both the Python Dag and the Lang-SDK Dag for the same `dag_id` -creates conflicting definitions. [PR #71213](https://github.com/apache/airflow/pull/71213) -disambiguated them with a per-Dag `is_mixed_language_dag` flag. This ADR -replaces that flag with an Dag authoring level distinction — `@Dag` vs. -`@MixedLangDag` in the case of Java SDK — plus a `DagFileParsingRequest` field, for the reasons below. +A Lang-SDK artifact (JAR, packed Go executable) can serve either of the two +authoring features the Language SDK spec fixes, or both at once: + +| Feature | Who owns the graph | Author writes | Artifact contributes | +|---------------------------------|----------------------------|----------------------------------|----------------------| +| **Mixed Language Task Handler** | Python, via `@task.stub` | `TaskHandler(dagId, taskId, fn)` | Only task bodies | +| **Native Dag** | the Lang-SDK source itself | `Dag(spec)` | The entire Dag | + +In the mixed-language role the artifact used to author a Dag too — under the +same `dag_id` the Python file already owns — so two conflicting definitions +reached persistence and something downstream had to choose between them. This +ADR removes the conflict at the authoring interface instead of resolving it +afterwards: an artifact backing `@task.stub` tasks registers **task handlers**, +which are not Dags, so no second definition for that `dag_id` is ever produced. + +Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec +version `1.0`): `Dag`, `TaskHandler`, `DagRef`, `TaskHandlerRef`, `bundle`, +`register`, `serve`. The examples below use the Java and Go spellings +(`@Builder.TaskHandler` / `airflow.TaskHandler`); TypeScript spells the same +terms in its own idiom. Note that the spec's `bundle` is the SDK-side +registration container, not Airflow's `DagBundle` — both appear in this ADR. This ADR uses `JavaDagImporter` and `JavaCoordinator` as the concrete examples. `ExecutableDagImporter` (Go) and `NodeDagImporter` (TypeScript) follow the same -flow, each backed by its own `BaseCoordinator` subclass. A common base class is -expected once the second Lang-SDK importer lands. +flow, each pairing its own `SubprocessCoordinator` subclass with its own +`AbstractDagImporter` implementation. ## Decision -### The Annotation, Not a Flag +### Differentiate at the Authoring Interface + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ Per-task (Python): is_stub │ +│ AbstractOperator.is_stub = False (default) │ +│ _StubOperator.is_stub = True │ +│ │ +│ Per-registration (Lang-SDK): which interface the author wrote │ +│ Dag(spec) → DagRef │ +│ TaskHandler(dagId, taskId, fn) → TaskHandlerRef │ +│ │ +│ Both kinds go into one bundle, through one verb: │ +│ bundle.register(dag, handler, ...) → bundle.serve() │ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +`TaskHandler` is a separate interface, not a second flavour of `Dag`. It names +the `(dagId, taskId)` pair Python already declared and the `fn` that implements +it, and returns a `TaskHandlerRef`: no schedule, no task graph, no `dag_id` of +its own to persist. `Dag(spec)` returns a `DagRef`, which owns all three. Java +spells the pair `@Builder.TaskHandler(dagId, taskId)` and `@Builder.Dag(id, +...)`, both driving the `BuilderProcessor` annotation processor +([ADR-0003](0003-pure-java-dags.md)); Go spells them +`airflow.TaskHandler(dagId, taskId, fn)` and `airflow.Dag(spec)`. The spelling +differs per SDK, the split does not. + +The spec puts both kinds in one `bundle`, takes them through one `register` verb +in any mixture, and serves the process with one `bundle.serve()`. So the bundle +cannot be the discriminator — and does not need to be. Dag parsing draws on the +`Dag` registrations; handler validation draws on the `TaskHandler` +registrations. Each registration already says which kind it is. + +The alternative was to keep authoring the Dag on both sides and mark the +Lang-SDK copy with a per-Dag flag (`is_mixed_language_dag`) so importers know +which copy to drop. That keeps producing the definition it then has to discard, +and pushes the question "is this Dag real?" onto every consumer of a serialized +Dag. Taking the Dag out of the authoring interface answers that question once, +for everyone. The only marker left in serialization is `is_stub`, and it is +per-task. + +### The Wire Contract: a Channel of Its Own + +Handler declarations travel on their own message pair, in their own +discriminated unions — not as a field on `DagFileParseRequest`: + +``` +ToRuntime = TaskHandlerParseRequest coordinator → runtime +ToCoordinator = TaskHandlerParsingResult runtime → coordinator +``` + +`task-sdk/src/airflow/sdk/execution_time/schema/` introspects exactly the +unions it is given — `ToTask` / `ToSupervisor` (task-execution channel) and +`ToManager` / `ToDagProcessor` (manager ↔ parser channel) — and union +membership is the only registration step: it is what puts a body into the +generated `schema.json` snapshot and therefore into every SDK's generated +models. So the choice of union *is* the wire-contract decision. + +Neither existing pair fits, because the runtime plays two different roles. When +it parses native Dags it **replaces** the parser process and answers the +manager, so `ToDagProcessor` / `ToManager` describe it exactly, and those +unions stay as ADR-0004 defined them. When it answers a handler query it is a +short-lived child of the Python parser, and the manager is not a peer at all. +Reusing `ToDagProcessor` there would make one union mean two recipients. A +fifth pair, named for the peers the coordinator already has, keeps both honest. + +``` +class TaskHandlerParseRequest: + file: str # the artifact the coordinator resolved + dag_id: str # scope: handlers bound to this dag_id only + type: Literal["TaskHandlerParseRequest"] + +class TaskHandlerParsingResult: + fileloc: str + task_handlers: list[TaskHandlerDeclaration] + import_errors: dict[str, str] | None = None + warnings: list | None = None + type: Literal["TaskHandlerParsingResult"] + +class TaskHandlerDeclaration: + dag_id: str + task_id: str + params: list[TaskHandlerParam] # ordered — arg_bindings are positional + +class TaskHandlerParam: + name: str + value_schema: ArgValueSchema | None = None + required: bool # the handler declares no default +``` + +`task_handlers` is this channel's answer to +`DagFileParsingResult.serialized_dags`, and it is deliberately a stronger +contract. `serialized_dags` is `list[LazyDeserializedDAG]`, which reduces in +the snapshot to `{"data": {"type": "object", "additionalProperties": true}}` — +an opaque blob, which is why every SDK reimplements DagSerialization v3 by hand +and validates it against shared fixtures ([ADR-0004](0004-dag-parsing.md)). +A handler declaration carries no Dag, so it can be fully typed, code-generated, +and schema-validated in every SDK. **The mixed-language path never asks a Lang +SDK to implement DagSerialization at all.** + +`value_schema` reuses the existing `ArgValueSchema` definition — the same one +`LiteralArgBinding.value_schema` and `XComArgBinding.value_schema` already +point at ([ADR-0007](0007-taskflow-across-language-boundary.md)). Validation +therefore compares like against like rather than translating between two +vocabularies. Two properties of that field are load-bearing: + +- **It is nullable on both sides.** An unannotated `@task.stub` parameter + produces `value_schema: null` today, so validation compares schemas only + where neither side is null and falls back to name-and-arity otherwise. + Without that, every untyped stub argument becomes a parse error. +- **`params` is ordered.** `LiteralArgBinding` and `XComArgBinding` are each + documented as "one positional stub-task argument", so position is part of the + contract rather than incidental. + +A declaration carries no class, method, or source location. ADR-0006 already +rules out Lang-SDK source display, and putting it on the wire would invite a +consumer to render it. + +### The Coordinator Interface ``` -┌──────────────────────────────────────────────────────────────┐ -│ Per-task: is_stub │ -│ AbstractOperator.is_stub = False (default) │ -│ _StubOperator.is_stub = True │ -│ │ -│ Per-Dag: which annotation the Lang-SDK author applied │ -│ @Dag — native, the entire Dag (existing) │ -│ @MixedLangDag — tasks only, backs a Python (new) │ -│ file's `@task.stub` operators │ -└──────────────────────────────────────────────────────────────┘ +BaseCoordinator execution_time/coordinator.py + ├── execute_task (shipped) + ├── parse_dag (new — Flow B) + └── parse_task_handler (new — Flow C) + │ +SubprocessCoordinator coordinators/_subprocess.py + implements all three; each resolves (command, subprocess_schema_version) + from a hook and owns the socket lifecycle: + ├── _build_execute_task_command (shipped) + ├── _build_parse_dag_command (new) + └── _build_parse_task_handler_command (new) + │ +JavaCoordinator · ExecutableCoordinator · NodeCoordinator + supply the three commands; no socket or protocol code ``` -`@Dag` (`Builder.Dag`) and `@MixedLangDag` (`Builder.MixedLangDag`, new) drive -the same `BuilderProcessor` annotation processor -([ADR-0003](0003-pure-java-dags.md)) — each generates a `*Builder` whose -`build()` still returns a `DagDef`; `BundleBuilder.getDags(): Iterable` -is unchanged. What differs is which annotation produced it, recorded on the -generated builder itself, so the JVM's parsing entrypoint tells native and -mixed-language Dags apart without a field carried on the Dag object. The -decision is `(annotation × importer)`. +The two parse methods are **not symmetric siblings**, which is the strongest +reason not to collapse them into one `parse(request)`: + +- `parse_dag` is **bridge mode**. The coordinator becomes the parser process; + the manager's `DagFileParseRequest` and the runtime's `DagFileParsingResult` + cross unchanged over the raw byte forwarder, on the manager ↔ parser unions + ([ADR-0004](0004-dag-parsing.md)). The coordinator never reads the payload. +- `parse_task_handler` is **query mode**. The coordinator is itself the peer: + it sends one `TaskHandlerParseRequest` and decodes one + `TaskHandlerParsingResult` on the channel above. + +One collapsed method would also erase a real deployment distinction — a +coordinator can serve mixed-language handlers with no interest in native Dag +parsing, and an absent method says so better than a runtime rejection. +`can_handle_dag_file` still gates Flow B only; Flow C resolves through the +coordinator registry instead. + +Naming follows the shipped `execute_task` / `_build_execute_task_command` pair, +and supersedes ADR-0004's `run_dag_parsing` / `dag_parsing_cmd`. Each hook +returns its own `subprocess_schema_version`, so handler parsing negotiates the +supervisor schema through the mechanism task execution already uses, and +`None` disables migration exactly as it does today. + +### DagImporter Registration + +`AbstractDagImporter` (AIP-85, `task-sdk/src/airflow/sdk/importers/`) is about +source formats — `can_handle`, `list_dag_definitions`, `import_definition`, +`get_source_code` — and returns `DagImportResult.dags: list[DAG]`, i.e. +`airflow.sdk.DAG` objects. The coordinator is about processes. A Lang-SDK +importer composes one, and the coordinator is what hands it out: + +``` +BaseCoordinator.get_dag_importer() -> AbstractDagImporter | None + default: None — this coordinator contributes no importer + JavaCoordinator.get_dag_importer() -> JavaDagImporter(coordinator=self) +``` + +Because the coordinator returns the importer already bound to itself, an +operator never configures an importer's coordinator: that wiring is an +implementation detail of the coordinator that produced it. +`DagImporterRegistry.from_config` gains one tier: + +``` +DagImporterRegistry.from_config(bundle_name) + ├── defaults PythonDagImporter, ZipImporter + ├── coordinators serving bundle_name (new) + │ └── register(coordinator.get_dag_importer()) + ├── [dag_processor] dag_importer_configs (global, unchanged) + └── that bundle's own `importers` list (unchanged) +``` + +`dag_importer_configs` stays the door for importers with no runtime behind them +— a YAML importer, say. A Lang SDK never arrives that way. + +#### One coordinator instance per DagBundle + +The registry keys importers by file extension, one importer per extension, +evicting the previous claimant with a log warning. Two `JavaCoordinator` +instances on different JDKs would both claim `.jar`. + +The boundary that resolves this is deployment-shaped rather than a tie-break: +**a coordinator instance owns a DagBundle.** `SubprocessCoordinator` classifies +that ownership at construction into one of three artifact sources, and only one +of them names a bundle: + +``` +EXPLICIT_ROOT jars_root / executables_root / bundles_root + → no DagBundle at all +NAMED_BUNDLE dag_bundle_name + → that bundle, at the version current when work starts +TASK_BUNDLE neither set + → the bundle the delegating Dag itself lives in, + at the run's version +``` + +The first two are mutually exclusive, and both that conflict and a +`dag_bundle_name` naming an unconfigured bundle are rejected at construction. + +Which mode a coordinator is in decides whether it can back a Flow-B importer: + +- **`NAMED_BUNDLE` registers into that bundle's registry.** This is the + unambiguous case, and the mode a deployment running two JDKs has to use: two + `dag_bundle_name` values, two bundle-scoped registries, `.jar` claimed once + in each. +- **`TASK_BUNDLE` has no fixed bundle**, so its importer registers into every + bundle-scoped registry. Sound only while it is the sole claimant of its + extension — the co-located single-runtime deployment. +- **`EXPLICIT_ROOT` has no DagBundle at all**, so the Dag processor never scans + its artifacts and it cannot produce a Dag to persist. It stays a Flow-C + coordinator, reached by queue. + +`get_importer_registry(bundle_name)` is already cached per bundle, so the +scoping mechanism is in place. What `CoordinatorManager` still needs is the +reverse lookup: it resolves coordinators by queue only today — +`for_queue(queue)`, driven by `[sdk] queue_to_coordinator` — with nothing that +answers "which coordinators serve this bundle". + +#### Resolving artifact roots at parse time + +`_init_root_source` already resolves roots for all three modes, but publishes +them through `_get_scan_roots()`, which is scoped to an active task and raises +outside one. Both parse-side commands need the same roots with no +`TaskInstance` in hand: `EXPLICIT_ROOT` and `NAMED_BUNDLE` resolve from the +coordinator's own configuration, and `TASK_BUNDLE` resolves against the bundle +the Dag processor is parsing. The scope that publishes the roots has to open +for a parse as well as for a task. + +#### Extensionless artifacts + +A packed Go bundle has no suffix, so `ExecutableDagImporter` claims the **empty** +extension as a first-class key, rather than depending on a `can_handle` scan +whose winner varies with registration order. Three places assume a non-empty +suffix today: `_normalize_extensions` rewrites `""` to `"."`; `get_importer` and +`can_handle` guard on `if suffix:`, which skips the extension map entirely for an +extensionless file; and `find_file_dag_definitions` filters on +`path.suffix.lower()`. Empty has to pass through all three, with the guards +testing `suffix is not None`. ### Processing Flows @@ -83,36 +323,47 @@ PythonDagImporter.import_definition(definition, bundle=...) DagModelOperation → PERSIST ``` -#### Flow B — Lang-SDK Importer (native and mixed) +#### Flow B — Lang-SDK Importer (native Dags only) -`JavaDagImporter` **never** persists a Dag produced by `@MixedLangDag`. +A `TaskHandlerRef` carries no Dag, so a mixed-language `dag_id` is never a +candidate for persistence — there is nothing for `JavaDagImporter` to filter. ``` JavaDagImporter.import_definition(definition, bundle=...) │ - ├── BaseCoordinator.run_dag_parsing( - │ request=DagFileParseRequest(file=..., mixed_language_dags_only=False)) - │ └── Spawn JVM → JVM filters locally: only @Dag-produced entries are serialized - │ ┌─────────────────────────────────────────┐ - │ │ dag_id: "java_report" │ (@Dag) - │ └─────────────────────────────────────────┘ - │ "etl" (@MixedLangDag) never leaves the JVM — nothing to discard. + ├── JavaCoordinator.parse_dag(...) ← bridge mode + │ ├── _build_parse_dag_command() → (command, schema_version) + │ └── Spawn JVM, forward fd 0 ⇄ comm socket + │ manager ──DagFileParseRequest───▶ JVM + │ manager ◀─DagFileParsingResult─── JVM + │ ┌────────────────────────────────────────────────────┐ + │ │ serialized_dags: ["java_report"] (@Builder.Dag) │ + │ └────────────────────────────────────────────────────┘ + │ The TaskHandler registrations have no Dag to serialize — + │ no "etl" entry exists to be discarded. │ - ├── Return DagImportResult(dags=["java_report"]) + ├── LazyDeserializedDAG(data=...) → airflow.sdk.DAG + ├── Return DagImportResult(dags=[DAG("java_report")]) ▼ DagModelOperation → PERSIST "java_report" only ``` +`DagImportResult.dags` is `list[DAG]`, so the importer wraps each serialized +entry as a `LazyDeserializedDAG` and transforms it into an `airflow.sdk.DAG` +before returning. The transform is mechanical — the serialized form encodes the +SDK Dag's own fields — but `LazyDeserializedDAG` lives in `airflow-core` today, +so where a Task SDK importer reaches it from is settled by AIP-85's own +`list[DAG]` / `list[LazyDeserializedDAG]` discussion rather than here. Flow C +needs none of this: it compares declarations and returns the `airflow.sdk.DAG` +the Python parser already built. + #### Flow C — Mixed-Language Dag (validation driven by PythonDagImporter) -The `PythonDagImporter` owns validation. When it finds a stub task, it -resolves the stub's `queue` to its Coordinator, then asks that Coordinator -**directly** for the `@MixedLangDag`-produced Dag structure backing the same -`dag_id` — no `DagImporter` involved, since this is a single parse -request/response, not a discovery operation. Either path bottoms out at the -Coordinator sending a `DagFileParseRequest` and getting back a -`DagFileParsingResult` ([ADR-0004](0004-dag-parsing.md)), so validation calls -that primitive directly instead of detouring through the importer contract. +The `PythonDagImporter` owns validation. When it finds a stub task, it resolves +the stub's `queue` to its Coordinator, then calls that Coordinator's +`parse_task_handler` **directly** — no `DagImporter` involved, since this is a +single request/response rather than a discovery operation, and what comes back +is not a Dag. Resolution goes through the coordinator registry, not the filesystem — the Dag processor is file-at-a-time and the `PythonDagImporter` never sees the `.jar`. @@ -126,7 +377,9 @@ PythonDagImporter.import_definition(definition, bundle=...) ├── serialize_dag(dag) → is_stub tasks carry arg_bindings (ADR-0007) │ │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 1: Resolve stub tasks → Coordinator instances via queue │ + │ │ Step 1: Resolve stub tasks → Coordinator instances via │ + │ │ CoordinatorManager.for_queue(queue) │ + │ │ ([sdk] queue_to_coordinator) │ │ │ │ │ │ stub task queue Coordinator instance │ │ │ ────────────────────────────────────────────────── │ @@ -140,53 +393,55 @@ PythonDagImporter.import_definition(definition, bundle=...) │ └─────────────────────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 2: Locate the file backing dag_id="etl", no JVM launch — │ - │ │ reuse BundleScanner (ADR-0003), which reads JAR manifests │ - │ │ directly │ + │ │ Step 2: Locate the artifact backing dag_id="etl", no JVM launch — │ + │ │ reuse BundleScanner (ADR-0003), whose build-time inventory │ + │ │ indexes registered (dagId, taskId) handler pairs alongside │ + │ │ the artifact's native Dag ids │ │ │ │ │ │ JavaCoordinator(name="jdk-11") │ - │ │ ├── Get artifact root: │ - │ │ │ coordinator.dag_bundle │ - │ │ │ → BaseDagBundle for "java-jdk11-bundle" │ - │ │ │ OR coordinator.artifact_roots │ - │ │ │ → [Path("/opt/airflow/jars/jdk11/")] │ + │ │ ├── resolve artifact roots for its mode │ + │ │ │ EXPLICIT_ROOT → jars_root │ + │ │ │ NAMED_BUNDLE → dag_bundle_name, pinned │ + │ │ │ TASK_BUNDLE → the bundle being parsed │ │ │ │ │ - │ │ └── BundleScanner.scanBundles(artifact_root) │ + │ │ └── BundleScanner.scanBundles(roots) │ │ │ → Map │ │ │ → "etl" → ResolvedBundle(mainClass=..., classpath=...) │ │ │ │ │ │ JavaCoordinator(name="jdk-17") │ - │ │ └── (same lookup, different artifact root) │ + │ │ └── (same lookup, its own roots) │ │ └─────────────────────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 3: Call the Coordinator directly — the same primitive │ - │ │ ADR-0004 already defines for ordinary parsing │ + │ │ Step 3: Query the Coordinator — one request, one response │ │ │ │ - │ │ JavaCoordinator(name="jdk-11").run_dag_parsing( │ - │ │ path=resolved.path, bundle_name=..., bundle_path=..., │ - │ │ request=DagFileParseRequest( │ - │ │ file=resolved.path, │ - │ │ mixed_language_dags_only=True)) │ + │ │ JavaCoordinator(name="jdk-11").parse_task_handler( │ + │ │ file=resolved.path, dag_id="etl") │ │ │ │ │ - │ │ ├── Spawn JVM → sends DagFileParseRequest over the bridge │ - │ │ ├── JVM filters locally: only @MixedLangDag entries serialized │ + │ │ ├── _build_parse_task_handler_command() │ + │ │ │ → (command, subprocess_schema_version) │ + │ │ ├── Spawn JVM, send TaskHandlerParseRequest (ToRuntime) │ + │ │ ├── JVM answers from its own TaskHandler registrations │ + │ │ │ whose dagId is "etl" │ │ │ ▼ │ - │ │ DagFileParsingResult → deserialize → LazyDeserializedDAG "etl" │ + │ │ TaskHandlerParsingResult (ToCoordinator) │ + │ │ → [TaskHandlerDeclaration(task_id, params), ...] │ │ │ │ │ │ JavaCoordinator(name="jdk-17") │ - │ │ └── (same call, different resolved path) │ + │ │ └── (same call, its own resolved path and handler set) │ │ └─────────────────────────────────────────────────────────────────────┘ │ │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 4: Compare directly against the parsed Python Dag — the │ - │ │ same LazyDeserializedDAG model the scheduler reads │ + │ │ Step 4: Compare the declarations against the parsed Python Dag — │ + │ │ union the handlers returned by every coordinator resolved │ + │ │ in Step 1 before comparing task ids │ │ │ │ - │ │ Python Dag "etl" (stub tasks) LazyDeserializedDAG "etl" │ - │ │ ─────────────────────────────── ───────────────────────── │ - │ │ task_id ↔ task_id (sets must match)│ - │ │ arg_bindings[*].name ↔ declared param name │ - │ │ arg_bindings[*].value_schema ↔ declared param schema │ + │ │ Python Dag "etl" (stub tasks) TaskHandlerDeclaration │ + │ │ ────────────────────────────── ────────────────────────────── │ + │ │ task_id ↔ task_id (sets must match) │ + │ │ arg_bindings[*].name ↔ params[*].name (in order) │ + │ │ arg_bindings[*].value_schema ↔ params[*].value_schema │ + │ │ compared only where neither side is null │ │ │ │ │ │ On mismatch → DagImportError │ │ └─────────────────────────────────────────────────────────────────────┘ @@ -198,62 +453,95 @@ DagModelOperation → PERSIST (Python Dag is the sole DB record) ### Decision Matrix -| Caller | Produced by | `mixed_language_dags_only` | Action | -|------------------------------------------------|------------------------|------------------------------|---------------------------------------------------------------------| -| `PythonDagImporter` | its own parsed Dag | n/a | PERSIST | -| `PythonDagImporter` (via stub → Coordinator) | `@MixedLangDag` | `True` | VALIDATE only — compared against the Python Dag, never persisted | -| `JavaDagImporter` | `@Dag` | `False` (default) | PERSIST | -| `JavaDagImporter` | `@MixedLangDag` | `False` (default) | never returned — filtered inside the JVM | +| Caller | Coordinator call | What comes back | Action | +|----------------------------------------------|----------------------------------------------|------------------------------------------|--------------------------------------------------| +| `PythonDagImporter` | — (parses the Python file itself) | its own parsed Dag | PERSIST | +| `PythonDagImporter` (via stub → Coordinator) | `parse_task_handler`, scoped to one `dag_id` | `TaskHandlerParsingResult` | VALIDATE only — not a Dag, so nothing to persist | +| `JavaDagImporter` | `parse_dag` | `DagFileParsingResult`, native Dags only | PERSIST | + +There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — +and nothing reading a `DagImporter`'s results — ever sees one. -### Why the Distinction Lives on the Annotation, Not the Bundle +### Why the Distinction Lives on the Interface, Not the Bundle -A single artifact can contain both native and mixed-language Dags: +A single artifact registers both kinds through one `register` call: ``` -analytics.jar (one BundleBuilder.getDags() call) -├── EtlTasks @MixedLangDag(id = "etl") (backs etl.py's stub tasks) -└── JavaReportPipeline @Dag(id = "java_report") (native) +analytics.jar +├── EtlTasks @Builder.TaskHandler(dagId = "etl", taskId = "extract") +│ @Builder.TaskHandler(dagId = "etl", taskId = "transform") +│ backs etl.py's stub tasks — no Dag on the Java side +└── JavaReportPipeline @Builder.Dag(id = "java_report") + native, persisted as "java_report" + +bundle.register(javaReport, etlExtract, etlTransform); // one verb, both kinds +bundle.serve(args); ``` -The importer decides per-Dag, not per-file or per-bundle — and it decides by -which annotation authored each class, not by inspecting a field on the -resulting Dag. +One artifact, one bundle, one `register` call — so neither the file nor the +bundle can tell a `DagImporter` what it is holding. The registration kind can, +and does: the split is per registration, not per file and not per bundle. ## Consequences - **Python leads, Lang-SDK follows.** `PythonDagImporter` persists the Dag and - drives validation via `queue → Coordinator → run_dag_parsing - (mixed_language_dags_only=True)`, comparing the resulting - `LazyDeserializedDAG` directly against its own parsed Dag. Lang-SDK importers - never receive an `@MixedLangDag`-produced Dag to begin with — the JVM - filters it out of the response before anything crosses the wire. -- Stub/implementation mismatches (missing task, extra task, parameter name or - schema mismatch) surface as `DagImportError` at parse time. + drives validation via `queue → Coordinator → parse_task_handler`, comparing + the returned declarations against its own parsed Dag. +- **A mixed-language `dag_id` never appears in Dag processing results.** The + artifact registers handlers only, so no `Dag` registration exists for a + `dag_id` a Python file already owns, and + `DagFileParsingResult.serialized_dags` carries native Dags exclusively. + Everything downstream — `DagImportResult`, import errors, the Dag list — sees + exactly one record per `dag_id`, with no flag to interpret. +- **The handler channel is typed where the Dag channel is opaque.** Every SDK + gets generated models for `TaskHandlerDeclaration` from the `schema.json` + snapshot, and nothing on this path requires a DagSerialization + implementation. +- Stub/implementation mismatches (missing handler, extra handler, parameter name + or order, incompatible schema) surface as `DagImportError` at parse time. + An unannotated stub argument is checked by name and position only. - The Python Dag and Lang-SDK artifact can live in different DagBundles — resolution goes through the coordinator registry, not the filesystem. - A single Dag can have stubs targeting different queues (some Java, some Go) — - each resolves to its own coordinator instance independently. + each resolves to its own coordinator instance independently, and validation + unions their declarations before comparing task ids. - **Mixed-language is Python-primary only.** Lang-SDK runtimes cannot define stub - operators — the reverse direction (a native Lang-SDK Dag delegating tasks to - Python) is not supported. -- **No `is_mixed_language_dag` flag.** This ADR supersedes that part of - [PR #71213](https://github.com/apache/airflow/pull/71213): the role is - encoded in which annotation authored the class (`@Dag` vs. - `@MixedLangDag`) and in the `DagFileParsingRequest.mixed_language_dags_only` - field, not in a value carried on the serialized Dag itself. -- No schema migration, no new `DagModel` column, no REST/UI change. + operators — the reverse direction (a native Dag delegating tasks to Python) is + not supported. +- **A Lang-SDK importer is never configured by hand.** `[sdk] coordinators` is + the one place a runtime is declared; the importer follows from it through + `get_dag_importer()`, so the same runtime has no second configuration site to + drift from. +- **One coordinator instance per DagBundle** becomes a deployment constraint: + two JDKs mean two `dag_bundle_name` values and two bundle-scoped + registries. It is what keeps extension-keyed importer registration + unambiguous, and it makes `NAMED_BUNDLE` the mode a multi-runtime + deployment has to use. +- **No per-Dag flag**, no schema migration, no new `DagModel` column, no + REST/UI change. +- Adding `ToRuntime` / `ToCoordinator` extends the set of unions the supervisor + schema package introspects, and regenerating the `schema.json` snapshot is + what propagates the new bodies into each SDK's generated models. Two prek + hooks guard that snapshot, and their interaction on a first-introduction body + needs checking against the hooks rather than the docs. +- Terms here track Language SDK spec `1.0`. A spec rename of `TaskHandler`, or + of the `register`/`serve` verbs, lands in this ADR too. ## References -- [ADR-0003](0003-pure-java-dags.md) — `BundleBuilder` / `BundleScanner` / - `BuilderProcessor` (`@Dag` / `@Task` annotation processing) -- [ADR-0004](0004-dag-parsing.md) — `BaseCoordinator` / `DagFileParsingResult` +- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) — `Dag` / + `TaskHandler` / `bundle` / `register` / `serve` and their per-SDK spellings +- `task-sdk/src/airflow/sdk/execution_time/schema/` — the supervisor schema + version bundle, the union registry, and the generated `schema.json` snapshot +- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor` + (`@Builder.Dag` / `@Builder.Task` annotation processing, build-time artifact + inventory) +- [ADR-0004](0004-dag-parsing.md) — coordinator subprocess bridge, + `DagFileParseRequest` / `DagFileParsingResult`, `can_handle_dag_file` - [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display for mixed-language Dags - [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / - `TaskArgBinding`, compared against `@MixedLangDag` task signatures in Flow C + `TaskArgBinding` / `ArgValueSchema`, compared against `TaskHandler` parameter + declarations in Flow C - [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs - [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter -- [PR #71213](https://github.com/apache/airflow/pull/71213) — introduced the - per-Dag `is_mixed_language_dag` flag, superseded by this ADR's type-level - distinction From ca15c5c3945683dfc3a7b2794abf76f3a5774a12 Mon Sep 17 00:00:00 2001 From: LIU ZHE YOU Date: Sun, 13 Sep 2026 04:04:00 +0000 Subject: [PATCH 4/9] Separate the mixed-language decisions from their implementation mechanics An ADR is read first to understand what was decided and why, and only later to build the thing. Exact message shapes, call sites, and the code that has to change were interleaved with that reasoning, so neither audience could skim its half. The decisions now stand on their own and the mechanics wait in an appendix. --- .../0010-mixed-language-dag-processing.md | 432 +++++++++--------- 1 file changed, 223 insertions(+), 209 deletions(-) diff --git a/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md index fcd647a00ef95..12f57875347f7 100644 --- a/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md @@ -42,12 +42,12 @@ which are not Dags, so no second definition for that `dag_id` is ever produced. Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec version `1.0`): `Dag`, `TaskHandler`, `DagRef`, `TaskHandlerRef`, `bundle`, -`register`, `serve`. The examples below use the Java and Go spellings +`register`, `serve`. The examples use the Java and Go spellings (`@Builder.TaskHandler` / `airflow.TaskHandler`); TypeScript spells the same terms in its own idiom. Note that the spec's `bundle` is the SDK-side registration container, not Airflow's `DagBundle` — both appear in this ADR. -This ADR uses `JavaDagImporter` and `JavaCoordinator` as the concrete examples. +`JavaDagImporter` and `JavaCoordinator` are the concrete examples throughout. `ExecutableDagImporter` (Go) and `NodeDagImporter` (TypeScript) follow the same flow, each pairing its own `SubprocessCoordinator` subclass with its own `AbstractDagImporter` implementation. @@ -74,12 +74,8 @@ flow, each pairing its own `SubprocessCoordinator` subclass with its own `TaskHandler` is a separate interface, not a second flavour of `Dag`. It names the `(dagId, taskId)` pair Python already declared and the `fn` that implements it, and returns a `TaskHandlerRef`: no schedule, no task graph, no `dag_id` of -its own to persist. `Dag(spec)` returns a `DagRef`, which owns all three. Java -spells the pair `@Builder.TaskHandler(dagId, taskId)` and `@Builder.Dag(id, -...)`, both driving the `BuilderProcessor` annotation processor -([ADR-0003](0003-pure-java-dags.md)); Go spells them -`airflow.TaskHandler(dagId, taskId, fn)` and `airflow.Dag(spec)`. The spelling -differs per SDK, the split does not. +its own to persist. `Dag(spec)` returns a `DagRef`, which owns all three. The +spelling differs per SDK; the split does not. The spec puts both kinds in one `bundle`, takes them through one `register` verb in any mixture, and serves the process with one `bundle.serve()`. So the bundle @@ -95,30 +91,205 @@ Dag. Taking the Dag out of the authoring interface answers that question once, for everyone. The only marker left in serialization is `is_stub`, and it is per-task. -### The Wire Contract: a Channel of Its Own +### Handler Declarations Travel on Their Own Channel -Handler declarations travel on their own message pair, in their own -discriminated unions — not as a field on `DagFileParseRequest`: +Handler declarations are carried by a dedicated message pair in their own +discriminated unions — `ToRuntime` / `ToCoordinator` — not by a field added to +`DagFileParseRequest`. + +Union membership is the only registration step in the supervisor schema +package: it is what puts a body into the generated snapshot, and therefore into +every SDK's generated models. So the choice of union *is* the wire-contract +decision. Neither existing pair fits, because the runtime plays two roles. When +it parses native Dags it **replaces** the parser process and answers the +manager, which the manager ↔ parser unions already describe. When it answers a +handler query it is a short-lived child of the Python parser, and the manager is +not a peer at all. Reusing the parser unions there would make one union mean two +recipients. + +The payload is deliberately a stronger contract than the one it parallels. +`DagFileParsingResult.serialized_dags` is `list[LazyDeserializedDAG]`, which +reduces in the snapshot to an opaque object — which is why every SDK +reimplements DagSerialization by hand and validates it against shared fixtures +([ADR-0004](0004-dag-parsing.md)). A handler declaration carries no Dag, so it +is fully typed, code-generated, and schema-validated in every SDK. **The +mixed-language path never asks a Lang SDK to implement DagSerialization at +all.** + +Parameter schemas reuse the existing `ArgValueSchema` definition that +`arg_bindings` already carries ([ADR-0007](0007-taskflow-across-language-boundary.md)), +so validation compares like against like rather than translating between two +vocabularies. Appendix A gives the message definitions and the two properties of +that field — nullability and ordering — that validation has to respect. + +### Two Parse Verbs, Because There Are Two Conversations + +`BaseCoordinator` gains `parse_dag` and `parse_task_handler` alongside the +shipped `execute_task`. They are **not symmetric siblings**, which is the +reason not to collapse them into one `parse(request)`: + +- `parse_dag` is **bridge mode**. The coordinator becomes the parser process + and raw-forwards the manager's request and the runtime's reply. It never + reads the payload. +- `parse_task_handler` is **query mode**. The coordinator is itself the peer: + it sends one request and decodes one reply on the channel above. + +One collapsed method would also erase a real deployment distinction — a +coordinator can serve mixed-language handlers with no interest in native Dag +parsing, and an absent method says so better than a runtime rejection. +Appendix B gives the layering and the command hooks. + +### A Lang-SDK DagImporter Comes From Its Coordinator + +`AbstractDagImporter` (AIP-85) is about source formats and returns +`DagImportResult.dags: list[DAG]`, i.e. `airflow.sdk.DAG` objects. The +coordinator is about processes. A Lang-SDK importer composes one, and the +coordinator is what hands it out — `JavaCoordinator.get_dag_importer()` returns +a `JavaDagImporter` already bound to that coordinator. An operator therefore +never configures an importer's coordinator: that wiring is an implementation +detail of the coordinator that produced it, and `[sdk] coordinators` stays the +one place a runtime is declared. + +Importers are keyed by file extension, one per extension, so two +`JavaCoordinator` instances on different JDKs would both claim `.jar`. The +boundary that resolves this is deployment-shaped rather than a tie-break: **a +coordinator instance owns a DagBundle.** `SubprocessCoordinator` already +classifies that ownership at construction, and only its `NAMED_BUNDLE` mode +names a bundle — which makes `NAMED_BUNDLE` the mode a deployment running two +runtimes of the same language has to use. Appendix C covers the registration +tiers, the three artifact-source modes, and extensionless artifacts. + +### Processing Flows + +Three flows, walked through in full in Appendix D. + +**Flow A — pure Python Dag.** No stub tasks, nothing to cross-validate. +`PythonDagImporter` parses and returns; the Dag is persisted. + +**Flow B — native Lang-SDK Dag.** `JavaDagImporter` calls `parse_dag`, and the +runtime serializes its `Dag` registrations. `TaskHandler` registrations have no +Dag to serialize, so no mixed-language `dag_id` is ever a candidate for +persistence — there is nothing to filter. Because `DagImportResult.dags` is +`list[DAG]`, the importer wraps each serialized entry as a +`LazyDeserializedDAG` and transforms it into an `airflow.sdk.DAG` before +returning. That transform is mechanical, but `LazyDeserializedDAG` lives in +`airflow-core` today, so where a Task SDK importer reaches it from is settled by +AIP-85's own `list[DAG]` / `list[LazyDeserializedDAG]` discussion rather than +here. + +**Flow C — mixed-language Dag.** The `PythonDagImporter` owns validation. When +it finds a stub task it resolves the stub's `queue` to a coordinator, calls +`parse_task_handler` for the same `dag_id`, and compares the returned +declarations against the Dag it just parsed. No `DagImporter` is involved — +this is a single request/response rather than a discovery operation, and what +comes back is not a Dag. Resolution goes through the coordinator registry, not +the filesystem, so the Python Dag and the Lang-SDK artifact **do not need to be +in the same DagBundle**. Flow C needs no `sdk.DAG` round-trip at all: it +returns the Dag the Python parser already built. + +### Decision Matrix + +| Caller | Coordinator call | What comes back | Action | +|----------------------------------------------|----------------------------------------------|------------------------------------------|--------------------------------------------------| +| `PythonDagImporter` | — (parses the Python file itself) | its own parsed Dag | PERSIST | +| `PythonDagImporter` (via stub → Coordinator) | `parse_task_handler`, scoped to one `dag_id` | `TaskHandlerParsingResult` | VALIDATE only — not a Dag, so nothing to persist | +| `JavaDagImporter` | `parse_dag` | `DagFileParsingResult`, native Dags only | PERSIST | + +There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — +and nothing reading a `DagImporter`'s results — ever sees one. + +### Why the Distinction Lives on the Interface, Not the Bundle + +A single artifact registers both kinds through one `register` call: ``` -ToRuntime = TaskHandlerParseRequest coordinator → runtime -ToCoordinator = TaskHandlerParsingResult runtime → coordinator +analytics.jar +├── EtlTasks @Builder.TaskHandler(dagId = "etl", taskId = "extract") +│ @Builder.TaskHandler(dagId = "etl", taskId = "transform") +│ backs etl.py's stub tasks — no Dag on the Java side +└── JavaReportPipeline @Builder.Dag(id = "java_report") + native, persisted as "java_report" + +bundle.register(javaReport, etlExtract, etlTransform); // one verb, both kinds +bundle.serve(args); ``` -`task-sdk/src/airflow/sdk/execution_time/schema/` introspects exactly the -unions it is given — `ToTask` / `ToSupervisor` (task-execution channel) and -`ToManager` / `ToDagProcessor` (manager ↔ parser channel) — and union -membership is the only registration step: it is what puts a body into the -generated `schema.json` snapshot and therefore into every SDK's generated -models. So the choice of union *is* the wire-contract decision. +One artifact, one bundle, one `register` call — so neither the file nor the +bundle can tell a `DagImporter` what it is holding. The registration kind can, +and does: the split is per registration, not per file and not per bundle. -Neither existing pair fits, because the runtime plays two different roles. When -it parses native Dags it **replaces** the parser process and answers the -manager, so `ToDagProcessor` / `ToManager` describe it exactly, and those -unions stay as ADR-0004 defined them. When it answers a handler query it is a -short-lived child of the Python parser, and the manager is not a peer at all. -Reusing `ToDagProcessor` there would make one union mean two recipients. A -fifth pair, named for the peers the coordinator already has, keeps both honest. +## Consequences + +- **Python leads, Lang-SDK follows.** `PythonDagImporter` persists the Dag and + drives validation via `queue → Coordinator → parse_task_handler`, comparing + the returned declarations against its own parsed Dag. +- **A mixed-language `dag_id` never appears in Dag processing results.** The + artifact registers handlers only, so no `Dag` registration exists for a + `dag_id` a Python file already owns. Everything downstream — `DagImportResult`, + import errors, the Dag list — sees exactly one record per `dag_id`, with no + flag to interpret. +- **The handler channel is typed where the Dag channel is opaque.** Every SDK + gets generated models for the declaration shape, and nothing on this path + requires a DagSerialization implementation. +- Stub/implementation mismatches (missing handler, extra handler, parameter name + or order, incompatible schema) surface as `DagImportError` at parse time. An + unannotated stub argument is checked by name and position only. +- The Python Dag and Lang-SDK artifact can live in different DagBundles — + resolution goes through the coordinator registry, not the filesystem. +- A single Dag can have stubs targeting different queues (some Java, some Go) — + each resolves to its own coordinator instance independently, and validation + unions their declarations before comparing task ids. +- **Mixed-language is Python-primary only.** Lang-SDK runtimes cannot define stub + operators — the reverse direction (a native Dag delegating tasks to Python) is + not supported. +- **A Lang-SDK importer is never configured by hand.** `[sdk] coordinators` is + the one place a runtime is declared; the importer follows from it through + `get_dag_importer()`, so the same runtime has no second configuration site to + drift from. +- **One coordinator instance per DagBundle** becomes a deployment constraint: + two JDKs mean two `dag_bundle_name` values and two bundle-scoped registries. + It is what keeps extension-keyed importer registration unambiguous, and it + makes `NAMED_BUNDLE` the mode a multi-runtime deployment has to use. +- **No per-Dag flag**, no schema migration, no new `DagModel` column, no + REST/UI change. +- Each Lang-SDK runtime implements a second request type rather than a new field + on the existing one. That is the cost of handlers not being Dags: a `DagRef` + and a `TaskHandlerRef` have different shapes, so one request/result pair could + not have carried both. +- Terms here track Language SDK spec `1.0`. A spec rename of `TaskHandler`, or + of the `register`/`serve` verbs, lands in this ADR too. + +## References + +- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) — `Dag` / + `TaskHandler` / `bundle` / `register` / `serve` and their per-SDK spellings +- `task-sdk/src/airflow/sdk/execution_time/schema/` — the supervisor schema + version bundle, the union registry, and the generated snapshot +- `task-sdk/src/airflow/sdk/importers/` — `AbstractDagImporter`, + `DagImportResult`, `DagImporterRegistry` +- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor` + (`@Builder.Dag` / `@Builder.Task` annotation processing, build-time artifact + inventory) +- [ADR-0004](0004-dag-parsing.md) — coordinator subprocess bridge, + `DagFileParseRequest` / `DagFileParsingResult`, `can_handle_dag_file` +- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display + for mixed-language Dags +- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / + `TaskArgBinding` / `ArgValueSchema` +- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs +- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter + +## Appendix — For Implementation + +Everything below is mechanics: exact shapes, exact call sites, and the code +that has to change. None of it is needed to follow the decision above. + +### Appendix A — Message Definitions + +``` +ToRuntime = TaskHandlerParseRequest coordinator → runtime +ToCoordinator = TaskHandlerParsingResult runtime → coordinator +``` ``` class TaskHandlerParseRequest: @@ -144,21 +315,7 @@ class TaskHandlerParam: required: bool # the handler declares no default ``` -`task_handlers` is this channel's answer to -`DagFileParsingResult.serialized_dags`, and it is deliberately a stronger -contract. `serialized_dags` is `list[LazyDeserializedDAG]`, which reduces in -the snapshot to `{"data": {"type": "object", "additionalProperties": true}}` — -an opaque blob, which is why every SDK reimplements DagSerialization v3 by hand -and validates it against shared fixtures ([ADR-0004](0004-dag-parsing.md)). -A handler declaration carries no Dag, so it can be fully typed, code-generated, -and schema-validated in every SDK. **The mixed-language path never asks a Lang -SDK to implement DagSerialization at all.** - -`value_schema` reuses the existing `ArgValueSchema` definition — the same one -`LiteralArgBinding.value_schema` and `XComArgBinding.value_schema` already -point at ([ADR-0007](0007-taskflow-across-language-boundary.md)). Validation -therefore compares like against like rather than translating between two -vocabularies. Two properties of that field are load-bearing: +Two properties of `value_schema` are load-bearing for validation: - **It is nullable on both sides.** An unannotated `@task.stub` parameter produces `value_schema: null` today, so validation compares schemas only @@ -168,11 +325,22 @@ vocabularies. Two properties of that field are load-bearing: documented as "one positional stub-task argument", so position is part of the contract rather than incidental. -A declaration carries no class, method, or source location. ADR-0006 already -rules out Lang-SDK source display, and putting it on the wire would invite a -consumer to render it. +A declaration carries no class, method, or source location. ADR-0006 rules out +Lang-SDK source display, and putting it on the wire would invite a consumer to +render it. -### The Coordinator Interface +Registration mechanics: `registered_models_by_name()` in +`task-sdk/src/airflow/sdk/execution_time/schema/` introspects a fixed set of +unions — `ToTask` / `ToSupervisor` and `ToManager` / `ToDagProcessor` — so +adding `ToRuntime` / `ToCoordinator` extends that set. Regenerating the +`schema.json` snapshot is what propagates the new bodies into each SDK's +generated models. Two prek hooks guard the snapshot, and their interaction on a +first-introduction body needs checking against the hooks rather than against +that package's `AGENTS.md`, which says no `VersionChange` is required while the +second hook fails when the snapshot moves with nothing under `versions/` +touched. + +### Appendix B — Coordinator Interface ``` BaseCoordinator execution_time/coordinator.py @@ -191,36 +359,14 @@ JavaCoordinator · ExecutableCoordinator · NodeCoordinator supply the three commands; no socket or protocol code ``` -The two parse methods are **not symmetric siblings**, which is the strongest -reason not to collapse them into one `parse(request)`: - -- `parse_dag` is **bridge mode**. The coordinator becomes the parser process; - the manager's `DagFileParseRequest` and the runtime's `DagFileParsingResult` - cross unchanged over the raw byte forwarder, on the manager ↔ parser unions - ([ADR-0004](0004-dag-parsing.md)). The coordinator never reads the payload. -- `parse_task_handler` is **query mode**. The coordinator is itself the peer: - it sends one `TaskHandlerParseRequest` and decodes one - `TaskHandlerParsingResult` on the channel above. - -One collapsed method would also erase a real deployment distinction — a -coordinator can serve mixed-language handlers with no interest in native Dag -parsing, and an absent method says so better than a runtime rejection. -`can_handle_dag_file` still gates Flow B only; Flow C resolves through the -coordinator registry instead. - Naming follows the shipped `execute_task` / `_build_execute_task_command` pair, and supersedes ADR-0004's `run_dag_parsing` / `dag_parsing_cmd`. Each hook returns its own `subprocess_schema_version`, so handler parsing negotiates the -supervisor schema through the mechanism task execution already uses, and -`None` disables migration exactly as it does today. +supervisor schema through the mechanism task execution already uses, and `None` +disables migration exactly as it does today. `can_handle_dag_file` still gates +Flow B only; Flow C resolves through the coordinator registry instead. -### DagImporter Registration - -`AbstractDagImporter` (AIP-85, `task-sdk/src/airflow/sdk/importers/`) is about -source formats — `can_handle`, `list_dag_definitions`, `import_definition`, -`get_source_code` — and returns `DagImportResult.dags: list[DAG]`, i.e. -`airflow.sdk.DAG` objects. The coordinator is about processes. A Lang-SDK -importer composes one, and the coordinator is what hands it out: +### Appendix C — DagImporter Registration ``` BaseCoordinator.get_dag_importer() -> AbstractDagImporter | None @@ -228,11 +374,6 @@ BaseCoordinator.get_dag_importer() -> AbstractDagImporter | None JavaCoordinator.get_dag_importer() -> JavaDagImporter(coordinator=self) ``` -Because the coordinator returns the importer already bound to itself, an -operator never configures an importer's coordinator: that wiring is an -implementation detail of the coordinator that produced it. -`DagImporterRegistry.from_config` gains one tier: - ``` DagImporterRegistry.from_config(bundle_name) ├── defaults PythonDagImporter, ZipImporter @@ -245,16 +386,9 @@ DagImporterRegistry.from_config(bundle_name) `dag_importer_configs` stays the door for importers with no runtime behind them — a YAML importer, say. A Lang SDK never arrives that way. -#### One coordinator instance per DagBundle - -The registry keys importers by file extension, one importer per extension, -evicting the previous claimant with a log warning. Two `JavaCoordinator` -instances on different JDKs would both claim `.jar`. +#### Artifact-source modes -The boundary that resolves this is deployment-shaped rather than a tie-break: -**a coordinator instance owns a DagBundle.** `SubprocessCoordinator` classifies -that ownership at construction into one of three artifact sources, and only one -of them names a bundle: +`SubprocessCoordinator` classifies artifact ownership at construction: ``` EXPLICIT_ROOT jars_root / executables_root / bundles_root @@ -268,13 +402,11 @@ TASK_BUNDLE neither set The first two are mutually exclusive, and both that conflict and a `dag_bundle_name` naming an unconfigured bundle are rejected at construction. - Which mode a coordinator is in decides whether it can back a Flow-B importer: -- **`NAMED_BUNDLE` registers into that bundle's registry.** This is the - unambiguous case, and the mode a deployment running two JDKs has to use: two - `dag_bundle_name` values, two bundle-scoped registries, `.jar` claimed once - in each. +- **`NAMED_BUNDLE` registers into that bundle's registry.** Two + `dag_bundle_name` values, two bundle-scoped registries, `.jar` claimed once in + each. - **`TASK_BUNDLE` has no fixed bundle**, so its importer registers into every bundle-scoped registry. Sound only while it is the sole claimant of its extension — the co-located single-runtime deployment. @@ -309,7 +441,7 @@ extensionless file; and `find_file_dag_definitions` filters on `path.suffix.lower()`. Empty has to pass through all three, with the guards testing `suffix is not None`. -### Processing Flows +### Appendix D — Flow Walkthroughs #### Flow A — Pure Python Dag (no `@task.stub`) @@ -325,9 +457,6 @@ DagModelOperation → PERSIST #### Flow B — Lang-SDK Importer (native Dags only) -A `TaskHandlerRef` carries no Dag, so a mixed-language `dag_id` is never a -candidate for persistence — there is nothing for `JavaDagImporter` to filter. - ``` JavaDagImporter.import_definition(definition, bundle=...) │ @@ -348,28 +477,8 @@ JavaDagImporter.import_definition(definition, bundle=...) DagModelOperation → PERSIST "java_report" only ``` -`DagImportResult.dags` is `list[DAG]`, so the importer wraps each serialized -entry as a `LazyDeserializedDAG` and transforms it into an `airflow.sdk.DAG` -before returning. The transform is mechanical — the serialized form encodes the -SDK Dag's own fields — but `LazyDeserializedDAG` lives in `airflow-core` today, -so where a Task SDK importer reaches it from is settled by AIP-85's own -`list[DAG]` / `list[LazyDeserializedDAG]` discussion rather than here. Flow C -needs none of this: it compares declarations and returns the `airflow.sdk.DAG` -the Python parser already built. - #### Flow C — Mixed-Language Dag (validation driven by PythonDagImporter) -The `PythonDagImporter` owns validation. When it finds a stub task, it resolves -the stub's `queue` to its Coordinator, then calls that Coordinator's -`parse_task_handler` **directly** — no `DagImporter` involved, since this is a -single request/response rather than a discovery operation, and what comes back -is not a Dag. - -Resolution goes through the coordinator registry, not the filesystem — the Dag -processor is file-at-a-time and the `PythonDagImporter` never sees the `.jar`. -This also means the Python Dag and the Lang-SDK artifact **do not need to be in -the same DagBundle**. - ``` PythonDagImporter.import_definition(definition, bundle=...) │ @@ -450,98 +559,3 @@ PythonDagImporter.import_definition(definition, bundle=...) ▼ DagModelOperation → PERSIST (Python Dag is the sole DB record) ``` - -### Decision Matrix - -| Caller | Coordinator call | What comes back | Action | -|----------------------------------------------|----------------------------------------------|------------------------------------------|--------------------------------------------------| -| `PythonDagImporter` | — (parses the Python file itself) | its own parsed Dag | PERSIST | -| `PythonDagImporter` (via stub → Coordinator) | `parse_task_handler`, scoped to one `dag_id` | `TaskHandlerParsingResult` | VALIDATE only — not a Dag, so nothing to persist | -| `JavaDagImporter` | `parse_dag` | `DagFileParsingResult`, native Dags only | PERSIST | - -There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — -and nothing reading a `DagImporter`'s results — ever sees one. - -### Why the Distinction Lives on the Interface, Not the Bundle - -A single artifact registers both kinds through one `register` call: - -``` -analytics.jar -├── EtlTasks @Builder.TaskHandler(dagId = "etl", taskId = "extract") -│ @Builder.TaskHandler(dagId = "etl", taskId = "transform") -│ backs etl.py's stub tasks — no Dag on the Java side -└── JavaReportPipeline @Builder.Dag(id = "java_report") - native, persisted as "java_report" - -bundle.register(javaReport, etlExtract, etlTransform); // one verb, both kinds -bundle.serve(args); -``` - -One artifact, one bundle, one `register` call — so neither the file nor the -bundle can tell a `DagImporter` what it is holding. The registration kind can, -and does: the split is per registration, not per file and not per bundle. - -## Consequences - -- **Python leads, Lang-SDK follows.** `PythonDagImporter` persists the Dag and - drives validation via `queue → Coordinator → parse_task_handler`, comparing - the returned declarations against its own parsed Dag. -- **A mixed-language `dag_id` never appears in Dag processing results.** The - artifact registers handlers only, so no `Dag` registration exists for a - `dag_id` a Python file already owns, and - `DagFileParsingResult.serialized_dags` carries native Dags exclusively. - Everything downstream — `DagImportResult`, import errors, the Dag list — sees - exactly one record per `dag_id`, with no flag to interpret. -- **The handler channel is typed where the Dag channel is opaque.** Every SDK - gets generated models for `TaskHandlerDeclaration` from the `schema.json` - snapshot, and nothing on this path requires a DagSerialization - implementation. -- Stub/implementation mismatches (missing handler, extra handler, parameter name - or order, incompatible schema) surface as `DagImportError` at parse time. - An unannotated stub argument is checked by name and position only. -- The Python Dag and Lang-SDK artifact can live in different DagBundles — - resolution goes through the coordinator registry, not the filesystem. -- A single Dag can have stubs targeting different queues (some Java, some Go) — - each resolves to its own coordinator instance independently, and validation - unions their declarations before comparing task ids. -- **Mixed-language is Python-primary only.** Lang-SDK runtimes cannot define stub - operators — the reverse direction (a native Dag delegating tasks to Python) is - not supported. -- **A Lang-SDK importer is never configured by hand.** `[sdk] coordinators` is - the one place a runtime is declared; the importer follows from it through - `get_dag_importer()`, so the same runtime has no second configuration site to - drift from. -- **One coordinator instance per DagBundle** becomes a deployment constraint: - two JDKs mean two `dag_bundle_name` values and two bundle-scoped - registries. It is what keeps extension-keyed importer registration - unambiguous, and it makes `NAMED_BUNDLE` the mode a multi-runtime - deployment has to use. -- **No per-Dag flag**, no schema migration, no new `DagModel` column, no - REST/UI change. -- Adding `ToRuntime` / `ToCoordinator` extends the set of unions the supervisor - schema package introspects, and regenerating the `schema.json` snapshot is - what propagates the new bodies into each SDK's generated models. Two prek - hooks guard that snapshot, and their interaction on a first-introduction body - needs checking against the hooks rather than the docs. -- Terms here track Language SDK spec `1.0`. A spec rename of `TaskHandler`, or - of the `register`/`serve` verbs, lands in this ADR too. - -## References - -- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) — `Dag` / - `TaskHandler` / `bundle` / `register` / `serve` and their per-SDK spellings -- `task-sdk/src/airflow/sdk/execution_time/schema/` — the supervisor schema - version bundle, the union registry, and the generated `schema.json` snapshot -- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor` - (`@Builder.Dag` / `@Builder.Task` annotation processing, build-time artifact - inventory) -- [ADR-0004](0004-dag-parsing.md) — coordinator subprocess bridge, - `DagFileParseRequest` / `DagFileParsingResult`, `can_handle_dag_file` -- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display - for mixed-language Dags -- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / - `TaskArgBinding` / `ArgValueSchema`, compared against `TaskHandler` parameter - declarations in Flow C -- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs -- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter From b7f137748fc547d9b5ab3c69545da91ebdf0baa3 Mon Sep 17 00:00:00 2001 From: LIU ZHE YOU Date: Sun, 13 Sep 2026 04:07:14 +0000 Subject: [PATCH 5/9] Split the Lang-SDK Dag processing decisions into one ADR each The single document answered three questions with different audiences and different review cycles: the wire protocol a runtime implements, how a Lang-SDK Dag source reaches the Dag processor, and how a Python stub is validated against the handler behind it. Readers had to take all three to act on one, and a change to any one of them forced the other two back through review. --- .../lang-sdk/0010-lang-sdk-parse-protocol.md | 214 +++++++ .../0010-mixed-language-dag-processing.md | 561 ------------------ .../lang-sdk/0011-native-dag-processing.md | 237 ++++++++ .../0012-mixed-language-dag-processing.md | 278 +++++++++ airflow-core/adr/lang-sdk/README.md | 4 +- 5 files changed, 732 insertions(+), 562 deletions(-) create mode 100644 airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md delete mode 100644 airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md create mode 100644 airflow-core/adr/lang-sdk/0011-native-dag-processing.md create mode 100644 airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md diff --git a/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md b/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md new file mode 100644 index 0000000000000..fed333fe0d778 --- /dev/null +++ b/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md @@ -0,0 +1,214 @@ + + +# ADR-0010: Lang-SDK Parse Protocol — Handler Messages and Coordinator Verbs + +## Status + +Proposed + +## Context + +The Dag processor needs two different answers from a Lang-SDK runtime, and they +are not variations of one question: + +- **Which Dags does this artifact define?** Answered by the runtime standing in + for the parser process, over the manager ↔ parser messages + [ADR-0004](0004-dag-parsing.md) already defines. +- **Which task handlers does this artifact register for a `dag_id` Python + already owns?** A `TaskHandler` registration carries no Dag + ([ADR-0012](0012-mixed-language-dag-processing.md)), so the answer is not a + serialized Dag and there is nothing in the existing messages to carry it. + +This ADR defines the channel and payload for the second question, and names +both parse-side entry points on the coordinator. + +Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec +version `1.0`): `Dag`, `TaskHandler`, `DagRef`, `TaskHandlerRef`, `bundle`, +`register`, `serve`. + +## Decision + +### Handler Declarations Travel on Their Own Channel + +Handler declarations are carried by a dedicated message pair in their own +discriminated unions — `ToRuntime` / `ToCoordinator` — not by a field added to +`DagFileParseRequest`. + +Union membership is the only registration step in the supervisor schema +package: it is what puts a body into the generated snapshot, and therefore into +every SDK's generated models. So the choice of union *is* the wire-contract +decision. Neither existing pair fits, because the runtime plays two roles. When +it parses native Dags it **replaces** the parser process and answers the +manager, which the manager ↔ parser unions already describe. When it answers a +handler query it is a short-lived child of the Python parser, and the manager is +not a peer at all. Reusing the parser unions there would make one union mean two +recipients. + +The payload is deliberately a stronger contract than the one it parallels. +`DagFileParsingResult.serialized_dags` is `list[LazyDeserializedDAG]`, which +reduces in the snapshot to an opaque object — which is why every SDK +reimplements DagSerialization by hand and validates it against shared fixtures +([ADR-0004](0004-dag-parsing.md)). A handler declaration carries no Dag, so it +is fully typed, code-generated, and schema-validated in every SDK. **The +mixed-language path never asks a Lang SDK to implement DagSerialization at +all.** + +Parameter schemas reuse the existing `ArgValueSchema` definition that +`arg_bindings` already carries +([ADR-0007](0007-taskflow-across-language-boundary.md)), so validation compares +like against like rather than translating between two vocabularies. Appendix A +gives the message definitions and the two properties of that field — +nullability and ordering — that validation has to respect. + +### Two Parse Verbs, Because There Are Two Conversations + +`BaseCoordinator` gains `parse_dag` and `parse_task_handler` alongside the +shipped `execute_task`. They are **not symmetric siblings**, which is the +reason not to collapse them into one `parse(request)`: + +- `parse_dag` is **bridge mode**. The coordinator becomes the parser process + and raw-forwards the manager's request and the runtime's reply. It never + reads the payload. +- `parse_task_handler` is **query mode**. The coordinator is itself the peer: + it sends one request and decodes one reply on the channel above. + +One collapsed method would also erase a real deployment distinction — a +coordinator can serve mixed-language handlers with no interest in native Dag +parsing, and an absent method says so better than a runtime rejection. +Appendix B gives the layering and the command hooks. + +## Consequences + +- **The handler channel is typed where the Dag channel is opaque.** Every SDK + gets generated models for the declaration shape, and nothing on this path + requires a DagSerialization implementation. +- Each Lang-SDK runtime implements a second request type rather than a new field + on the existing one. That is the cost of handlers not being Dags: a `DagRef` + and a `TaskHandlerRef` have different shapes, so one request/result pair could + not have carried both. +- Adding `ToRuntime` / `ToCoordinator` extends the set of unions the supervisor + schema package introspects, and regenerating the snapshot is what propagates + the new bodies into each SDK's generated models. +- Naming supersedes ADR-0004's `run_dag_parsing` / `dag_parsing_cmd`, matching + the shipped `execute_task` / `_build_execute_task_command` pair instead. +- Terms here track Language SDK spec `1.0`. A spec rename of `TaskHandler` + lands in this ADR too. + +## References + +- [ADR-0004](0004-dag-parsing.md) — coordinator subprocess bridge, + `DagFileParseRequest` / `DagFileParsingResult`, `can_handle_dag_file` +- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / + `TaskArgBinding` / `ArgValueSchema` +- [ADR-0011](0011-native-dag-processing.md) — who calls `parse_dag` +- [ADR-0012](0012-mixed-language-dag-processing.md) — who calls + `parse_task_handler`, and what it compares the reply against +- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) +- `task-sdk/src/airflow/sdk/execution_time/schema/` — the supervisor schema + version bundle, the union registry, and the generated snapshot +- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs + +## Appendix — For Implementation + +### Appendix A — Message Definitions + +``` +ToRuntime = TaskHandlerParseRequest coordinator → runtime +ToCoordinator = TaskHandlerParsingResult runtime → coordinator +``` + +``` +class TaskHandlerParseRequest: + file: str # the artifact the coordinator resolved + dag_id: str # scope: handlers bound to this dag_id only + type: Literal["TaskHandlerParseRequest"] + +class TaskHandlerParsingResult: + fileloc: str + task_handlers: list[TaskHandlerDeclaration] + import_errors: dict[str, str] | None = None + warnings: list | None = None + type: Literal["TaskHandlerParsingResult"] + +class TaskHandlerDeclaration: + dag_id: str + task_id: str + params: list[TaskHandlerParam] # ordered — arg_bindings are positional + +class TaskHandlerParam: + name: str + value_schema: ArgValueSchema | None = None + required: bool # the handler declares no default +``` + +Two properties of `value_schema` are load-bearing for validation: + +- **It is nullable on both sides.** An unannotated `@task.stub` parameter + produces `value_schema: null` today, so validation compares schemas only + where neither side is null and falls back to name-and-arity otherwise. + Without that, every untyped stub argument becomes a parse error. +- **`params` is ordered.** `LiteralArgBinding` and `XComArgBinding` are each + documented as "one positional stub-task argument", so position is part of the + contract rather than incidental. + +A declaration carries no class, method, or source location. +[ADR-0006](0006-no-lang-sdk-source-display.md) rules out Lang-SDK source +display, and putting it on the wire would invite a consumer to render it. + +Registration mechanics: `registered_models_by_name()` in +`task-sdk/src/airflow/sdk/execution_time/schema/` introspects a fixed set of +unions — `ToTask` / `ToSupervisor` and `ToManager` / `ToDagProcessor` — so +adding `ToRuntime` / `ToCoordinator` extends that set. Two prek hooks guard the +snapshot, and their interaction on a first-introduction body needs checking +against the hooks rather than against that package's `AGENTS.md`, which says no +`VersionChange` is required while the second hook fails when the snapshot moves +with nothing under `versions/` touched. + +### Appendix B — Coordinator Interface + +``` +BaseCoordinator execution_time/coordinator.py + ├── execute_task (shipped) + ├── parse_dag (new — native Dags, ADR-0011) + └── parse_task_handler (new — handlers, ADR-0012) + │ +SubprocessCoordinator coordinators/_subprocess.py + implements all three; each resolves (command, subprocess_schema_version) + from a hook and owns the socket lifecycle: + ├── _build_execute_task_command (shipped) + ├── _build_parse_dag_command (new) + └── _build_parse_task_handler_command (new) + │ +JavaCoordinator · ExecutableCoordinator · NodeCoordinator + supply the three commands; no socket or protocol code +``` + +Each hook returns its own `subprocess_schema_version`, so handler parsing +negotiates the supervisor schema through the mechanism task execution already +uses, and `None` disables migration exactly as it does today. +`can_handle_dag_file` still gates native Dag parsing only; handler queries +resolve through the coordinator registry instead. + +Artifact roots are resolved per mode by `_init_root_source`, but published +through `_get_scan_roots()`, which is scoped to an active task and raises +outside one. Both parse-side commands need the same roots with no +`TaskInstance` in hand, so the scope that publishes them has to open for a +parse as well as for a task. [ADR-0011](0011-native-dag-processing.md) covers +the modes themselves. diff --git a/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md deleted file mode 100644 index 12f57875347f7..0000000000000 --- a/airflow-core/adr/lang-sdk/0010-mixed-language-dag-processing.md +++ /dev/null @@ -1,561 +0,0 @@ - - -# ADR-0010: Mixed-Language Dag Processing — DagImporter Routing and Persistence - -## Status - -Proposed - -## Context - -A Lang-SDK artifact (JAR, packed Go executable) can serve either of the two -authoring features the Language SDK spec fixes, or both at once: - -| Feature | Who owns the graph | Author writes | Artifact contributes | -|---------------------------------|----------------------------|----------------------------------|----------------------| -| **Mixed Language Task Handler** | Python, via `@task.stub` | `TaskHandler(dagId, taskId, fn)` | Only task bodies | -| **Native Dag** | the Lang-SDK source itself | `Dag(spec)` | The entire Dag | - -In the mixed-language role the artifact used to author a Dag too — under the -same `dag_id` the Python file already owns — so two conflicting definitions -reached persistence and something downstream had to choose between them. This -ADR removes the conflict at the authoring interface instead of resolving it -afterwards: an artifact backing `@task.stub` tasks registers **task handlers**, -which are not Dags, so no second definition for that `dag_id` is ever produced. - -Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec -version `1.0`): `Dag`, `TaskHandler`, `DagRef`, `TaskHandlerRef`, `bundle`, -`register`, `serve`. The examples use the Java and Go spellings -(`@Builder.TaskHandler` / `airflow.TaskHandler`); TypeScript spells the same -terms in its own idiom. Note that the spec's `bundle` is the SDK-side -registration container, not Airflow's `DagBundle` — both appear in this ADR. - -`JavaDagImporter` and `JavaCoordinator` are the concrete examples throughout. -`ExecutableDagImporter` (Go) and `NodeDagImporter` (TypeScript) follow the same -flow, each pairing its own `SubprocessCoordinator` subclass with its own -`AbstractDagImporter` implementation. - -## Decision - -### Differentiate at the Authoring Interface - -``` -┌──────────────────────────────────────────────────────────────────────────┐ -│ Per-task (Python): is_stub │ -│ AbstractOperator.is_stub = False (default) │ -│ _StubOperator.is_stub = True │ -│ │ -│ Per-registration (Lang-SDK): which interface the author wrote │ -│ Dag(spec) → DagRef │ -│ TaskHandler(dagId, taskId, fn) → TaskHandlerRef │ -│ │ -│ Both kinds go into one bundle, through one verb: │ -│ bundle.register(dag, handler, ...) → bundle.serve() │ -└──────────────────────────────────────────────────────────────────────────┘ -``` - -`TaskHandler` is a separate interface, not a second flavour of `Dag`. It names -the `(dagId, taskId)` pair Python already declared and the `fn` that implements -it, and returns a `TaskHandlerRef`: no schedule, no task graph, no `dag_id` of -its own to persist. `Dag(spec)` returns a `DagRef`, which owns all three. The -spelling differs per SDK; the split does not. - -The spec puts both kinds in one `bundle`, takes them through one `register` verb -in any mixture, and serves the process with one `bundle.serve()`. So the bundle -cannot be the discriminator — and does not need to be. Dag parsing draws on the -`Dag` registrations; handler validation draws on the `TaskHandler` -registrations. Each registration already says which kind it is. - -The alternative was to keep authoring the Dag on both sides and mark the -Lang-SDK copy with a per-Dag flag (`is_mixed_language_dag`) so importers know -which copy to drop. That keeps producing the definition it then has to discard, -and pushes the question "is this Dag real?" onto every consumer of a serialized -Dag. Taking the Dag out of the authoring interface answers that question once, -for everyone. The only marker left in serialization is `is_stub`, and it is -per-task. - -### Handler Declarations Travel on Their Own Channel - -Handler declarations are carried by a dedicated message pair in their own -discriminated unions — `ToRuntime` / `ToCoordinator` — not by a field added to -`DagFileParseRequest`. - -Union membership is the only registration step in the supervisor schema -package: it is what puts a body into the generated snapshot, and therefore into -every SDK's generated models. So the choice of union *is* the wire-contract -decision. Neither existing pair fits, because the runtime plays two roles. When -it parses native Dags it **replaces** the parser process and answers the -manager, which the manager ↔ parser unions already describe. When it answers a -handler query it is a short-lived child of the Python parser, and the manager is -not a peer at all. Reusing the parser unions there would make one union mean two -recipients. - -The payload is deliberately a stronger contract than the one it parallels. -`DagFileParsingResult.serialized_dags` is `list[LazyDeserializedDAG]`, which -reduces in the snapshot to an opaque object — which is why every SDK -reimplements DagSerialization by hand and validates it against shared fixtures -([ADR-0004](0004-dag-parsing.md)). A handler declaration carries no Dag, so it -is fully typed, code-generated, and schema-validated in every SDK. **The -mixed-language path never asks a Lang SDK to implement DagSerialization at -all.** - -Parameter schemas reuse the existing `ArgValueSchema` definition that -`arg_bindings` already carries ([ADR-0007](0007-taskflow-across-language-boundary.md)), -so validation compares like against like rather than translating between two -vocabularies. Appendix A gives the message definitions and the two properties of -that field — nullability and ordering — that validation has to respect. - -### Two Parse Verbs, Because There Are Two Conversations - -`BaseCoordinator` gains `parse_dag` and `parse_task_handler` alongside the -shipped `execute_task`. They are **not symmetric siblings**, which is the -reason not to collapse them into one `parse(request)`: - -- `parse_dag` is **bridge mode**. The coordinator becomes the parser process - and raw-forwards the manager's request and the runtime's reply. It never - reads the payload. -- `parse_task_handler` is **query mode**. The coordinator is itself the peer: - it sends one request and decodes one reply on the channel above. - -One collapsed method would also erase a real deployment distinction — a -coordinator can serve mixed-language handlers with no interest in native Dag -parsing, and an absent method says so better than a runtime rejection. -Appendix B gives the layering and the command hooks. - -### A Lang-SDK DagImporter Comes From Its Coordinator - -`AbstractDagImporter` (AIP-85) is about source formats and returns -`DagImportResult.dags: list[DAG]`, i.e. `airflow.sdk.DAG` objects. The -coordinator is about processes. A Lang-SDK importer composes one, and the -coordinator is what hands it out — `JavaCoordinator.get_dag_importer()` returns -a `JavaDagImporter` already bound to that coordinator. An operator therefore -never configures an importer's coordinator: that wiring is an implementation -detail of the coordinator that produced it, and `[sdk] coordinators` stays the -one place a runtime is declared. - -Importers are keyed by file extension, one per extension, so two -`JavaCoordinator` instances on different JDKs would both claim `.jar`. The -boundary that resolves this is deployment-shaped rather than a tie-break: **a -coordinator instance owns a DagBundle.** `SubprocessCoordinator` already -classifies that ownership at construction, and only its `NAMED_BUNDLE` mode -names a bundle — which makes `NAMED_BUNDLE` the mode a deployment running two -runtimes of the same language has to use. Appendix C covers the registration -tiers, the three artifact-source modes, and extensionless artifacts. - -### Processing Flows - -Three flows, walked through in full in Appendix D. - -**Flow A — pure Python Dag.** No stub tasks, nothing to cross-validate. -`PythonDagImporter` parses and returns; the Dag is persisted. - -**Flow B — native Lang-SDK Dag.** `JavaDagImporter` calls `parse_dag`, and the -runtime serializes its `Dag` registrations. `TaskHandler` registrations have no -Dag to serialize, so no mixed-language `dag_id` is ever a candidate for -persistence — there is nothing to filter. Because `DagImportResult.dags` is -`list[DAG]`, the importer wraps each serialized entry as a -`LazyDeserializedDAG` and transforms it into an `airflow.sdk.DAG` before -returning. That transform is mechanical, but `LazyDeserializedDAG` lives in -`airflow-core` today, so where a Task SDK importer reaches it from is settled by -AIP-85's own `list[DAG]` / `list[LazyDeserializedDAG]` discussion rather than -here. - -**Flow C — mixed-language Dag.** The `PythonDagImporter` owns validation. When -it finds a stub task it resolves the stub's `queue` to a coordinator, calls -`parse_task_handler` for the same `dag_id`, and compares the returned -declarations against the Dag it just parsed. No `DagImporter` is involved — -this is a single request/response rather than a discovery operation, and what -comes back is not a Dag. Resolution goes through the coordinator registry, not -the filesystem, so the Python Dag and the Lang-SDK artifact **do not need to be -in the same DagBundle**. Flow C needs no `sdk.DAG` round-trip at all: it -returns the Dag the Python parser already built. - -### Decision Matrix - -| Caller | Coordinator call | What comes back | Action | -|----------------------------------------------|----------------------------------------------|------------------------------------------|--------------------------------------------------| -| `PythonDagImporter` | — (parses the Python file itself) | its own parsed Dag | PERSIST | -| `PythonDagImporter` (via stub → Coordinator) | `parse_task_handler`, scoped to one `dag_id` | `TaskHandlerParsingResult` | VALIDATE only — not a Dag, so nothing to persist | -| `JavaDagImporter` | `parse_dag` | `DagFileParsingResult`, native Dags only | PERSIST | - -There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — -and nothing reading a `DagImporter`'s results — ever sees one. - -### Why the Distinction Lives on the Interface, Not the Bundle - -A single artifact registers both kinds through one `register` call: - -``` -analytics.jar -├── EtlTasks @Builder.TaskHandler(dagId = "etl", taskId = "extract") -│ @Builder.TaskHandler(dagId = "etl", taskId = "transform") -│ backs etl.py's stub tasks — no Dag on the Java side -└── JavaReportPipeline @Builder.Dag(id = "java_report") - native, persisted as "java_report" - -bundle.register(javaReport, etlExtract, etlTransform); // one verb, both kinds -bundle.serve(args); -``` - -One artifact, one bundle, one `register` call — so neither the file nor the -bundle can tell a `DagImporter` what it is holding. The registration kind can, -and does: the split is per registration, not per file and not per bundle. - -## Consequences - -- **Python leads, Lang-SDK follows.** `PythonDagImporter` persists the Dag and - drives validation via `queue → Coordinator → parse_task_handler`, comparing - the returned declarations against its own parsed Dag. -- **A mixed-language `dag_id` never appears in Dag processing results.** The - artifact registers handlers only, so no `Dag` registration exists for a - `dag_id` a Python file already owns. Everything downstream — `DagImportResult`, - import errors, the Dag list — sees exactly one record per `dag_id`, with no - flag to interpret. -- **The handler channel is typed where the Dag channel is opaque.** Every SDK - gets generated models for the declaration shape, and nothing on this path - requires a DagSerialization implementation. -- Stub/implementation mismatches (missing handler, extra handler, parameter name - or order, incompatible schema) surface as `DagImportError` at parse time. An - unannotated stub argument is checked by name and position only. -- The Python Dag and Lang-SDK artifact can live in different DagBundles — - resolution goes through the coordinator registry, not the filesystem. -- A single Dag can have stubs targeting different queues (some Java, some Go) — - each resolves to its own coordinator instance independently, and validation - unions their declarations before comparing task ids. -- **Mixed-language is Python-primary only.** Lang-SDK runtimes cannot define stub - operators — the reverse direction (a native Dag delegating tasks to Python) is - not supported. -- **A Lang-SDK importer is never configured by hand.** `[sdk] coordinators` is - the one place a runtime is declared; the importer follows from it through - `get_dag_importer()`, so the same runtime has no second configuration site to - drift from. -- **One coordinator instance per DagBundle** becomes a deployment constraint: - two JDKs mean two `dag_bundle_name` values and two bundle-scoped registries. - It is what keeps extension-keyed importer registration unambiguous, and it - makes `NAMED_BUNDLE` the mode a multi-runtime deployment has to use. -- **No per-Dag flag**, no schema migration, no new `DagModel` column, no - REST/UI change. -- Each Lang-SDK runtime implements a second request type rather than a new field - on the existing one. That is the cost of handlers not being Dags: a `DagRef` - and a `TaskHandlerRef` have different shapes, so one request/result pair could - not have carried both. -- Terms here track Language SDK spec `1.0`. A spec rename of `TaskHandler`, or - of the `register`/`serve` verbs, lands in this ADR too. - -## References - -- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) — `Dag` / - `TaskHandler` / `bundle` / `register` / `serve` and their per-SDK spellings -- `task-sdk/src/airflow/sdk/execution_time/schema/` — the supervisor schema - version bundle, the union registry, and the generated snapshot -- `task-sdk/src/airflow/sdk/importers/` — `AbstractDagImporter`, - `DagImportResult`, `DagImporterRegistry` -- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor` - (`@Builder.Dag` / `@Builder.Task` annotation processing, build-time artifact - inventory) -- [ADR-0004](0004-dag-parsing.md) — coordinator subprocess bridge, - `DagFileParseRequest` / `DagFileParsingResult`, `can_handle_dag_file` -- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display - for mixed-language Dags -- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / - `TaskArgBinding` / `ArgValueSchema` -- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs -- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter - -## Appendix — For Implementation - -Everything below is mechanics: exact shapes, exact call sites, and the code -that has to change. None of it is needed to follow the decision above. - -### Appendix A — Message Definitions - -``` -ToRuntime = TaskHandlerParseRequest coordinator → runtime -ToCoordinator = TaskHandlerParsingResult runtime → coordinator -``` - -``` -class TaskHandlerParseRequest: - file: str # the artifact the coordinator resolved - dag_id: str # scope: handlers bound to this dag_id only - type: Literal["TaskHandlerParseRequest"] - -class TaskHandlerParsingResult: - fileloc: str - task_handlers: list[TaskHandlerDeclaration] - import_errors: dict[str, str] | None = None - warnings: list | None = None - type: Literal["TaskHandlerParsingResult"] - -class TaskHandlerDeclaration: - dag_id: str - task_id: str - params: list[TaskHandlerParam] # ordered — arg_bindings are positional - -class TaskHandlerParam: - name: str - value_schema: ArgValueSchema | None = None - required: bool # the handler declares no default -``` - -Two properties of `value_schema` are load-bearing for validation: - -- **It is nullable on both sides.** An unannotated `@task.stub` parameter - produces `value_schema: null` today, so validation compares schemas only - where neither side is null and falls back to name-and-arity otherwise. - Without that, every untyped stub argument becomes a parse error. -- **`params` is ordered.** `LiteralArgBinding` and `XComArgBinding` are each - documented as "one positional stub-task argument", so position is part of the - contract rather than incidental. - -A declaration carries no class, method, or source location. ADR-0006 rules out -Lang-SDK source display, and putting it on the wire would invite a consumer to -render it. - -Registration mechanics: `registered_models_by_name()` in -`task-sdk/src/airflow/sdk/execution_time/schema/` introspects a fixed set of -unions — `ToTask` / `ToSupervisor` and `ToManager` / `ToDagProcessor` — so -adding `ToRuntime` / `ToCoordinator` extends that set. Regenerating the -`schema.json` snapshot is what propagates the new bodies into each SDK's -generated models. Two prek hooks guard the snapshot, and their interaction on a -first-introduction body needs checking against the hooks rather than against -that package's `AGENTS.md`, which says no `VersionChange` is required while the -second hook fails when the snapshot moves with nothing under `versions/` -touched. - -### Appendix B — Coordinator Interface - -``` -BaseCoordinator execution_time/coordinator.py - ├── execute_task (shipped) - ├── parse_dag (new — Flow B) - └── parse_task_handler (new — Flow C) - │ -SubprocessCoordinator coordinators/_subprocess.py - implements all three; each resolves (command, subprocess_schema_version) - from a hook and owns the socket lifecycle: - ├── _build_execute_task_command (shipped) - ├── _build_parse_dag_command (new) - └── _build_parse_task_handler_command (new) - │ -JavaCoordinator · ExecutableCoordinator · NodeCoordinator - supply the three commands; no socket or protocol code -``` - -Naming follows the shipped `execute_task` / `_build_execute_task_command` pair, -and supersedes ADR-0004's `run_dag_parsing` / `dag_parsing_cmd`. Each hook -returns its own `subprocess_schema_version`, so handler parsing negotiates the -supervisor schema through the mechanism task execution already uses, and `None` -disables migration exactly as it does today. `can_handle_dag_file` still gates -Flow B only; Flow C resolves through the coordinator registry instead. - -### Appendix C — DagImporter Registration - -``` -BaseCoordinator.get_dag_importer() -> AbstractDagImporter | None - default: None — this coordinator contributes no importer - JavaCoordinator.get_dag_importer() -> JavaDagImporter(coordinator=self) -``` - -``` -DagImporterRegistry.from_config(bundle_name) - ├── defaults PythonDagImporter, ZipImporter - ├── coordinators serving bundle_name (new) - │ └── register(coordinator.get_dag_importer()) - ├── [dag_processor] dag_importer_configs (global, unchanged) - └── that bundle's own `importers` list (unchanged) -``` - -`dag_importer_configs` stays the door for importers with no runtime behind them -— a YAML importer, say. A Lang SDK never arrives that way. - -#### Artifact-source modes - -`SubprocessCoordinator` classifies artifact ownership at construction: - -``` -EXPLICIT_ROOT jars_root / executables_root / bundles_root - → no DagBundle at all -NAMED_BUNDLE dag_bundle_name - → that bundle, at the version current when work starts -TASK_BUNDLE neither set - → the bundle the delegating Dag itself lives in, - at the run's version -``` - -The first two are mutually exclusive, and both that conflict and a -`dag_bundle_name` naming an unconfigured bundle are rejected at construction. -Which mode a coordinator is in decides whether it can back a Flow-B importer: - -- **`NAMED_BUNDLE` registers into that bundle's registry.** Two - `dag_bundle_name` values, two bundle-scoped registries, `.jar` claimed once in - each. -- **`TASK_BUNDLE` has no fixed bundle**, so its importer registers into every - bundle-scoped registry. Sound only while it is the sole claimant of its - extension — the co-located single-runtime deployment. -- **`EXPLICIT_ROOT` has no DagBundle at all**, so the Dag processor never scans - its artifacts and it cannot produce a Dag to persist. It stays a Flow-C - coordinator, reached by queue. - -`get_importer_registry(bundle_name)` is already cached per bundle, so the -scoping mechanism is in place. What `CoordinatorManager` still needs is the -reverse lookup: it resolves coordinators by queue only today — -`for_queue(queue)`, driven by `[sdk] queue_to_coordinator` — with nothing that -answers "which coordinators serve this bundle". - -#### Resolving artifact roots at parse time - -`_init_root_source` already resolves roots for all three modes, but publishes -them through `_get_scan_roots()`, which is scoped to an active task and raises -outside one. Both parse-side commands need the same roots with no -`TaskInstance` in hand: `EXPLICIT_ROOT` and `NAMED_BUNDLE` resolve from the -coordinator's own configuration, and `TASK_BUNDLE` resolves against the bundle -the Dag processor is parsing. The scope that publishes the roots has to open -for a parse as well as for a task. - -#### Extensionless artifacts - -A packed Go bundle has no suffix, so `ExecutableDagImporter` claims the **empty** -extension as a first-class key, rather than depending on a `can_handle` scan -whose winner varies with registration order. Three places assume a non-empty -suffix today: `_normalize_extensions` rewrites `""` to `"."`; `get_importer` and -`can_handle` guard on `if suffix:`, which skips the extension map entirely for an -extensionless file; and `find_file_dag_definitions` filters on -`path.suffix.lower()`. Empty has to pass through all three, with the guards -testing `suffix is not None`. - -### Appendix D — Flow Walkthroughs - -#### Flow A — Pure Python Dag (no `@task.stub`) - -``` -PythonDagImporter.import_definition(definition, bundle=...) - │ - ├── Parse → DAG objects - ├── serialize_dag(dag) → no stub tasks, nothing to cross-validate - ├── Return DagImportResult(dags=[dag]) - ▼ -DagModelOperation → PERSIST -``` - -#### Flow B — Lang-SDK Importer (native Dags only) - -``` -JavaDagImporter.import_definition(definition, bundle=...) - │ - ├── JavaCoordinator.parse_dag(...) ← bridge mode - │ ├── _build_parse_dag_command() → (command, schema_version) - │ └── Spawn JVM, forward fd 0 ⇄ comm socket - │ manager ──DagFileParseRequest───▶ JVM - │ manager ◀─DagFileParsingResult─── JVM - │ ┌────────────────────────────────────────────────────┐ - │ │ serialized_dags: ["java_report"] (@Builder.Dag) │ - │ └────────────────────────────────────────────────────┘ - │ The TaskHandler registrations have no Dag to serialize — - │ no "etl" entry exists to be discarded. - │ - ├── LazyDeserializedDAG(data=...) → airflow.sdk.DAG - ├── Return DagImportResult(dags=[DAG("java_report")]) - ▼ -DagModelOperation → PERSIST "java_report" only -``` - -#### Flow C — Mixed-Language Dag (validation driven by PythonDagImporter) - -``` -PythonDagImporter.import_definition(definition, bundle=...) - │ - ├── Parse → DAG objects - ├── serialize_dag(dag) → is_stub tasks carry arg_bindings (ADR-0007) - │ - │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 1: Resolve stub tasks → Coordinator instances via │ - │ │ CoordinatorManager.for_queue(queue) │ - │ │ ([sdk] queue_to_coordinator) │ - │ │ │ - │ │ stub task queue Coordinator instance │ - │ │ ────────────────────────────────────────────────── │ - │ │ extract → "jdk-11" → JavaCoordinator(name="jdk-11") │ - │ │ transform → "jdk-17" → JavaCoordinator(name="jdk-17") │ - │ │ load → "jdk-11" → JavaCoordinator(name="jdk-11") │ - │ │ │ - │ │ Deduplicate → 2 distinct coordinator instances: │ - │ │ JavaCoordinator(name="jdk-11") │ - │ │ JavaCoordinator(name="jdk-17") │ - │ └─────────────────────────────────────────────────────────────────────┘ - │ - │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 2: Locate the artifact backing dag_id="etl", no JVM launch — │ - │ │ reuse BundleScanner (ADR-0003), whose build-time inventory │ - │ │ indexes registered (dagId, taskId) handler pairs alongside │ - │ │ the artifact's native Dag ids │ - │ │ │ - │ │ JavaCoordinator(name="jdk-11") │ - │ │ ├── resolve artifact roots for its mode │ - │ │ │ EXPLICIT_ROOT → jars_root │ - │ │ │ NAMED_BUNDLE → dag_bundle_name, pinned │ - │ │ │ TASK_BUNDLE → the bundle being parsed │ - │ │ │ │ - │ │ └── BundleScanner.scanBundles(roots) │ - │ │ → Map │ - │ │ → "etl" → ResolvedBundle(mainClass=..., classpath=...) │ - │ │ │ - │ │ JavaCoordinator(name="jdk-17") │ - │ │ └── (same lookup, its own roots) │ - │ └─────────────────────────────────────────────────────────────────────┘ - │ - │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 3: Query the Coordinator — one request, one response │ - │ │ │ - │ │ JavaCoordinator(name="jdk-11").parse_task_handler( │ - │ │ file=resolved.path, dag_id="etl") │ - │ │ │ │ - │ │ ├── _build_parse_task_handler_command() │ - │ │ │ → (command, subprocess_schema_version) │ - │ │ ├── Spawn JVM, send TaskHandlerParseRequest (ToRuntime) │ - │ │ ├── JVM answers from its own TaskHandler registrations │ - │ │ │ whose dagId is "etl" │ - │ │ ▼ │ - │ │ TaskHandlerParsingResult (ToCoordinator) │ - │ │ → [TaskHandlerDeclaration(task_id, params), ...] │ - │ │ │ - │ │ JavaCoordinator(name="jdk-17") │ - │ │ └── (same call, its own resolved path and handler set) │ - │ └─────────────────────────────────────────────────────────────────────┘ - │ - │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 4: Compare the declarations against the parsed Python Dag — │ - │ │ union the handlers returned by every coordinator resolved │ - │ │ in Step 1 before comparing task ids │ - │ │ │ - │ │ Python Dag "etl" (stub tasks) TaskHandlerDeclaration │ - │ │ ────────────────────────────── ────────────────────────────── │ - │ │ task_id ↔ task_id (sets must match) │ - │ │ arg_bindings[*].name ↔ params[*].name (in order) │ - │ │ arg_bindings[*].value_schema ↔ params[*].value_schema │ - │ │ compared only where neither side is null │ - │ │ │ - │ │ On mismatch → DagImportError │ - │ └─────────────────────────────────────────────────────────────────────┘ - │ - ├── Validation passed → Return DagImportResult(dags=[dag]) - ▼ -DagModelOperation → PERSIST (Python Dag is the sole DB record) -``` diff --git a/airflow-core/adr/lang-sdk/0011-native-dag-processing.md b/airflow-core/adr/lang-sdk/0011-native-dag-processing.md new file mode 100644 index 0000000000000..2e5a7ac72ed0a --- /dev/null +++ b/airflow-core/adr/lang-sdk/0011-native-dag-processing.md @@ -0,0 +1,237 @@ + + +# ADR-0011: Native Dag Processing — DagImporter Registration and Routing + +## Status + +Proposed + +## Context + +`AbstractDagImporter` (AIP-85, `task-sdk/src/airflow/sdk/importers/`) makes the +Dag processor aware of source formats beyond `.py`: `can_handle`, +`list_dag_definitions`, `import_definition`, `get_source_code`, returning +`DagImportResult.dags: list[DAG]` — `airflow.sdk.DAG` objects. + +A Lang-SDK artifact (JAR, packed Go executable) can be one of those sources: a +Dag authored entirely in Go or Java, with `Dag(spec)` on the SDK side. Parsing +it means launching a runtime, which is what a coordinator does. This ADR settles +where that importer comes from, which coordinator instance backs it, and how +the two are kept from being configured twice. + +`JavaDagImporter` and `JavaCoordinator` are the examples throughout; +`ExecutableDagImporter` (Go) and `NodeDagImporter` (TypeScript) follow the same +shape. + +## Decision + +### A Lang-SDK DagImporter Comes From Its Coordinator + +`AbstractDagImporter` is about source formats. The coordinator is about +processes. A Lang-SDK importer composes one, and the coordinator is what hands +it out — `JavaCoordinator.get_dag_importer()` returns a `JavaDagImporter` +already bound to that coordinator. + +An operator therefore never configures an importer's coordinator: that wiring is +an implementation detail of the coordinator that produced it, and +`[sdk] coordinators` stays the one place a runtime is declared. The existing +`[dag_processor] dag_importer_configs` path remains the door for importers with +no runtime behind them — a YAML importer, say. A Lang SDK never arrives that +way. Appendix A gives the registration tiers. + +### One Coordinator Instance Owns a DagBundle + +Importers are keyed by file extension, one per extension, with the later +registration evicting the earlier one and logging a warning. Two +`JavaCoordinator` instances on different JDKs would both claim `.jar`. + +The boundary that resolves this is deployment-shaped rather than a tie-break: a +coordinator instance owns a DagBundle. `SubprocessCoordinator` already +classifies that ownership at construction into three artifact sources, and only +`NAMED_BUNDLE` names a bundle — which makes `NAMED_BUNDLE` the mode a deployment +running two runtimes of the same language has to use. `EXPLICIT_ROOT` has no +DagBundle at all, so its artifacts are never scanned by the Dag processor and it +cannot produce a Dag to persist; it stays a coordinator reached by queue, for +mixed-language work only. Appendix B covers all three modes, the roots a parse +needs, and extensionless artifacts. + +### The Result Has to Be `airflow.sdk.DAG` + +`JavaDagImporter.import_definition` calls `parse_dag` +([ADR-0010](0010-lang-sdk-parse-protocol.md)) and the runtime serializes its +`Dag` registrations. `TaskHandler` registrations have no Dag to serialize, so a +mixed-language `dag_id` is never a candidate for persistence here — there is +nothing to filter out ([ADR-0012](0012-mixed-language-dag-processing.md)). + +Because `DagImportResult.dags` is `list[DAG]`, the importer wraps each +serialized entry as a `LazyDeserializedDAG` and transforms it into an +`airflow.sdk.DAG` before returning. The transform is mechanical — the serialized +form encodes the SDK Dag's own fields — but `LazyDeserializedDAG` lives in +`airflow-core` today, so where a Task SDK importer reaches it from is settled by +AIP-85's own `list[DAG]` / `list[LazyDeserializedDAG]` discussion rather than +here. + +Appendix C walks both flows: a pure Python file, and a Lang-SDK artifact. + +## Consequences + +- **A Lang-SDK importer is never configured by hand.** `[sdk] coordinators` is + the one place a runtime is declared; the importer follows from it through + `get_dag_importer()`, so the same runtime has no second configuration site to + drift from. +- **One coordinator instance per DagBundle** becomes a deployment constraint: + two JDKs mean two `dag_bundle_name` values and two bundle-scoped registries. + It is what keeps extension-keyed importer registration unambiguous, and it + makes `NAMED_BUNDLE` the mode a multi-runtime deployment has to use. +- A coordinator in `EXPLICIT_ROOT` mode cannot back a Dag importer at all. Its + artifacts live outside any DagBundle, so nothing scans them. +- `CoordinatorManager` needs a reverse lookup it does not have: it resolves + coordinators by queue only today, with nothing that answers "which + coordinators serve this bundle". +- A packed Go bundle claims the **empty** extension, which three call sites + currently treat as absent rather than as a key. +- `get_source_code` is abstract, so every Lang-SDK importer must implement it, + and a native Lang-SDK Dag has no Python source to return. What it should + return, and how that squares with + [ADR-0006](0006-no-lang-sdk-source-display.md), is not settled here. + +## References + +- [ADR-0010](0010-lang-sdk-parse-protocol.md) — `parse_dag` and the coordinator + interface it belongs to +- [ADR-0012](0012-mixed-language-dag-processing.md) — why `TaskHandler` + registrations never reach a `DagImporter` +- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor`, + build-time artifact inventory +- [ADR-0004](0004-dag-parsing.md) — `can_handle_dag_file`, the subprocess bridge +- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display +- `task-sdk/src/airflow/sdk/importers/` — `AbstractDagImporter`, + `DagImportResult`, `DagImporterRegistry` +- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter +- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs + +## Appendix — For Implementation + +### Appendix A — Registration + +``` +BaseCoordinator.get_dag_importer() -> AbstractDagImporter | None + default: None — this coordinator contributes no importer + JavaCoordinator.get_dag_importer() -> JavaDagImporter(coordinator=self) +``` + +``` +DagImporterRegistry.from_config(bundle_name) + ├── defaults PythonDagImporter, ZipImporter + ├── coordinators serving bundle_name (new) + │ └── register(coordinator.get_dag_importer()) + ├── [dag_processor] dag_importer_configs (global, unchanged) + └── that bundle's own `importers` list (unchanged) +``` + +`get_importer_registry(bundle_name)` is already cached per bundle, so the +scoping mechanism is in place. What `CoordinatorManager` still needs is the +reverse lookup: it resolves coordinators by queue only today — +`for_queue(queue)`, driven by `[sdk] queue_to_coordinator` — with nothing that +answers "which coordinators serve this bundle". + +### Appendix B — Artifact Sources + +`SubprocessCoordinator` classifies artifact ownership at construction: + +``` +EXPLICIT_ROOT jars_root / executables_root / bundles_root + → no DagBundle at all +NAMED_BUNDLE dag_bundle_name + → that bundle, at the version current when work starts +TASK_BUNDLE neither set + → the bundle the delegating Dag itself lives in, + at the run's version +``` + +The first two are mutually exclusive, and both that conflict and a +`dag_bundle_name` naming an unconfigured bundle are rejected at construction. +Which mode a coordinator is in decides whether it can back an importer: + +- **`NAMED_BUNDLE` registers into that bundle's registry.** Two + `dag_bundle_name` values, two bundle-scoped registries, `.jar` claimed once in + each. +- **`TASK_BUNDLE` has no fixed bundle**, so its importer registers into every + bundle-scoped registry. Sound only while it is the sole claimant of its + extension — the co-located single-runtime deployment. +- **`EXPLICIT_ROOT` has no DagBundle at all**, so the Dag processor never scans + its artifacts and it cannot produce a Dag to persist. + +#### Resolving artifact roots at parse time + +`_init_root_source` already resolves roots for all three modes, but publishes +them through `_get_scan_roots()`, which is scoped to an active task and raises +outside one. Both parse-side commands need the same roots with no +`TaskInstance` in hand: `EXPLICIT_ROOT` and `NAMED_BUNDLE` resolve from the +coordinator's own configuration, and `TASK_BUNDLE` resolves against the bundle +the Dag processor is parsing. The scope that publishes the roots has to open +for a parse as well as for a task. + +#### Extensionless artifacts + +A packed Go bundle has no suffix, so `ExecutableDagImporter` claims the **empty** +extension as a first-class key, rather than depending on a `can_handle` scan +whose winner varies with registration order. Three places assume a non-empty +suffix today: `_normalize_extensions` rewrites `""` to `"."`; `get_importer` and +`can_handle` guard on `if suffix:`, which skips the extension map entirely for an +extensionless file; and `find_file_dag_definitions` filters on +`path.suffix.lower()`. Empty has to pass through all three, with the guards +testing `suffix is not None`. + +### Appendix C — Flow Walkthroughs + +#### Pure Python Dag (no `@task.stub`) + +``` +PythonDagImporter.import_definition(definition, bundle=...) + │ + ├── Parse → DAG objects + ├── serialize_dag(dag) → no stub tasks, nothing to cross-validate + ├── Return DagImportResult(dags=[dag]) + ▼ +DagModelOperation → PERSIST +``` + +#### Native Lang-SDK Dag + +``` +JavaDagImporter.import_definition(definition, bundle=...) + │ + ├── JavaCoordinator.parse_dag(...) ← bridge mode + │ ├── _build_parse_dag_command() → (command, schema_version) + │ └── Spawn JVM, forward fd 0 ⇄ comm socket + │ manager ──DagFileParseRequest───▶ JVM + │ manager ◀─DagFileParsingResult─── JVM + │ ┌────────────────────────────────────────────────────┐ + │ │ serialized_dags: ["java_report"] (@Builder.Dag) │ + │ └────────────────────────────────────────────────────┘ + │ The TaskHandler registrations have no Dag to serialize — + │ no "etl" entry exists to be discarded. + │ + ├── LazyDeserializedDAG(data=...) → airflow.sdk.DAG + ├── Return DagImportResult(dags=[DAG("java_report")]) + ▼ +DagModelOperation → PERSIST "java_report" only +``` diff --git a/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md new file mode 100644 index 0000000000000..1b82e3f61523c --- /dev/null +++ b/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md @@ -0,0 +1,278 @@ + + +# ADR-0012: Mixed-Language Dag Processing — Task Handlers Are Not Dags + +## Status + +Proposed + +## Context + +A Lang-SDK artifact (JAR, packed Go executable) can serve either of the two +authoring features the Language SDK spec fixes, or both at once: + +| Feature | Who owns the graph | Author writes | Artifact contributes | +|---------------------------------|----------------------------|----------------------------------|----------------------| +| **Mixed Language Task Handler** | Python, via `@task.stub` | `TaskHandler(dagId, taskId, fn)` | Only task bodies | +| **Native Dag** | the Lang-SDK source itself | `Dag(spec)` | The entire Dag | + +In the mixed-language role the artifact used to author a Dag too — under the +same `dag_id` the Python file already owns — so two conflicting definitions +reached persistence and something downstream had to choose between them. This +ADR removes the conflict at the authoring interface instead of resolving it +afterwards, and defines how the Python importer validates a stub task against +the handler that implements it. + +Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec +version `1.0`): `Dag`, `TaskHandler`, `DagRef`, `TaskHandlerRef`, `bundle`, +`register`, `serve`. The examples use the Java and Go spellings +(`@Builder.TaskHandler` / `airflow.TaskHandler`); TypeScript spells the same +terms in its own idiom. Note that the spec's `bundle` is the SDK-side +registration container, not Airflow's `DagBundle` — both appear below. + +## Decision + +### Differentiate at the Authoring Interface + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ Per-task (Python): is_stub │ +│ AbstractOperator.is_stub = False (default) │ +│ _StubOperator.is_stub = True │ +│ │ +│ Per-registration (Lang-SDK): which interface the author wrote │ +│ Dag(spec) → DagRef │ +│ TaskHandler(dagId, taskId, fn) → TaskHandlerRef │ +│ │ +│ Both kinds go into one bundle, through one verb: │ +│ bundle.register(dag, handler, ...) → bundle.serve() │ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +`TaskHandler` is a separate interface, not a second flavour of `Dag`. It names +the `(dagId, taskId)` pair Python already declared and the `fn` that implements +it, and returns a `TaskHandlerRef`: no schedule, no task graph, no `dag_id` of +its own to persist. `Dag(spec)` returns a `DagRef`, which owns all three. The +spelling differs per SDK; the split does not. + +The spec puts both kinds in one `bundle`, takes them through one `register` verb +in any mixture, and serves the process with one `bundle.serve()`. So the bundle +cannot be the discriminator — and does not need to be. Dag parsing draws on the +`Dag` registrations ([ADR-0011](0011-native-dag-processing.md)); handler +validation draws on the `TaskHandler` registrations. Each registration already +says which kind it is. + +The alternative was to keep authoring the Dag on both sides and mark the +Lang-SDK copy with a per-Dag flag (`is_mixed_language_dag`) so importers know +which copy to drop. That keeps producing the definition it then has to discard, +and pushes the question "is this Dag real?" onto every consumer of a serialized +Dag. Taking the Dag out of the authoring interface answers that question once, +for everyone. The only marker left in serialization is `is_stub`, and it is +per-task. + +### Validation Is Driven by the Python Importer + +The `PythonDagImporter` owns validation. When it finds a stub task it resolves +the stub's `queue` to a coordinator, calls `parse_task_handler` +([ADR-0010](0010-lang-sdk-parse-protocol.md)) for the same `dag_id`, and +compares the returned declarations against the Dag it just parsed. No +`DagImporter` is involved — this is a single request/response rather than a +discovery operation, and what comes back is not a Dag. + +Resolution goes through the coordinator registry, not the filesystem: the Dag +processor is file-at-a-time and the `PythonDagImporter` never sees the `.jar`. +This also means the Python Dag and the Lang-SDK artifact **do not need to be in +the same DagBundle**. Nothing here needs an `airflow.sdk.DAG` round-trip either +— validation returns the Dag the Python parser already built. Appendix A walks +the four steps and Appendix B states what is compared. + +### Decision Matrix + +| Caller | Coordinator call | What comes back | Action | +|----------------------------------------------|----------------------------------------------|------------------------------------------|--------------------------------------------------| +| `PythonDagImporter` | — (parses the Python file itself) | its own parsed Dag | PERSIST | +| `PythonDagImporter` (via stub → Coordinator) | `parse_task_handler`, scoped to one `dag_id` | `TaskHandlerParsingResult` | VALIDATE only — not a Dag, so nothing to persist | +| `JavaDagImporter` | `parse_dag` | `DagFileParsingResult`, native Dags only | PERSIST | + +There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — +and nothing reading a `DagImporter`'s results — ever sees one. + +### Why the Distinction Lives on the Interface, Not the Bundle + +A single artifact registers both kinds through one `register` call: + +``` +analytics.jar +├── EtlTasks @Builder.TaskHandler(dagId = "etl", taskId = "extract") +│ @Builder.TaskHandler(dagId = "etl", taskId = "transform") +│ backs etl.py's stub tasks — no Dag on the Java side +└── JavaReportPipeline @Builder.Dag(id = "java_report") + native, persisted as "java_report" + +bundle.register(javaReport, etlExtract, etlTransform); // one verb, both kinds +bundle.serve(args); +``` + +One artifact, one bundle, one `register` call — so neither the file nor the +bundle can tell a `DagImporter` what it is holding. The registration kind can, +and does: the split is per registration, not per file and not per bundle. + +## Consequences + +- **Python leads, Lang-SDK follows.** `PythonDagImporter` persists the Dag and + drives validation via `queue → Coordinator → parse_task_handler`, comparing + the returned declarations against its own parsed Dag. +- **A mixed-language `dag_id` never appears in Dag processing results.** The + artifact registers handlers only, so no `Dag` registration exists for a + `dag_id` a Python file already owns. Everything downstream — `DagImportResult`, + import errors, the Dag list — sees exactly one record per `dag_id`, with no + flag to interpret. +- Stub/implementation mismatches (missing handler, extra handler, parameter name + or order, incompatible schema) surface as `DagImportError` at parse time. An + unannotated stub argument is checked by name and position only. +- The Python Dag and Lang-SDK artifact can live in different DagBundles — + resolution goes through the coordinator registry, not the filesystem. +- A single Dag can have stubs targeting different queues (some Java, some Go) — + each resolves to its own coordinator instance independently, and validation + unions their declarations before comparing task ids. +- **Mixed-language is Python-primary only.** Lang-SDK runtimes cannot define stub + operators — the reverse direction (a native Dag delegating tasks to Python) is + not supported. +- **No per-Dag flag**, no schema migration, no new `DagModel` column, no + REST/UI change. +- Terms here track Language SDK spec `1.0`. A spec rename of `TaskHandler`, or + of the `register`/`serve` verbs, lands in this ADR too. + +## References + +- [ADR-0010](0010-lang-sdk-parse-protocol.md) — `parse_task_handler` and the + `TaskHandlerParsingResult` shape this ADR compares against +- [ADR-0011](0011-native-dag-processing.md) — the `Dag`-registration half, and + the importer that persists it +- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor`, + build-time artifact inventory +- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display + for mixed-language Dags +- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / + `TaskArgBinding` / `ArgValueSchema` +- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) +- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs +- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter + +## Appendix — For Implementation + +### Appendix A — Validation Walkthrough + +``` +PythonDagImporter.import_definition(definition, bundle=...) + │ + ├── Parse → DAG objects + ├── serialize_dag(dag) → is_stub tasks carry arg_bindings (ADR-0007) + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 1: Resolve stub tasks → Coordinator instances via │ + │ │ CoordinatorManager.for_queue(queue) │ + │ │ ([sdk] queue_to_coordinator) │ + │ │ │ + │ │ stub task queue Coordinator instance │ + │ │ ────────────────────────────────────────────────── │ + │ │ extract → "jdk-11" → JavaCoordinator(name="jdk-11") │ + │ │ transform → "jdk-17" → JavaCoordinator(name="jdk-17") │ + │ │ load → "jdk-11" → JavaCoordinator(name="jdk-11") │ + │ │ │ + │ │ Deduplicate → 2 distinct coordinator instances: │ + │ │ JavaCoordinator(name="jdk-11") │ + │ │ JavaCoordinator(name="jdk-17") │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 2: Locate the artifact backing dag_id="etl", no JVM launch — │ + │ │ reuse BundleScanner (ADR-0003), whose build-time inventory │ + │ │ indexes registered (dagId, taskId) handler pairs alongside │ + │ │ the artifact's native Dag ids │ + │ │ │ + │ │ JavaCoordinator(name="jdk-11") │ + │ │ ├── resolve artifact roots for its mode │ + │ │ │ EXPLICIT_ROOT → jars_root │ + │ │ │ NAMED_BUNDLE → dag_bundle_name, pinned │ + │ │ │ TASK_BUNDLE → the bundle being parsed │ + │ │ │ │ + │ │ └── BundleScanner.scanBundles(roots) │ + │ │ → Map │ + │ │ → "etl" → ResolvedBundle(mainClass=..., classpath=...) │ + │ │ │ + │ │ JavaCoordinator(name="jdk-17") │ + │ │ └── (same lookup, its own roots) │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 3: Query the Coordinator — one request, one response │ + │ │ │ + │ │ JavaCoordinator(name="jdk-11").parse_task_handler( │ + │ │ file=resolved.path, dag_id="etl") │ + │ │ │ │ + │ │ ├── _build_parse_task_handler_command() │ + │ │ │ → (command, subprocess_schema_version) │ + │ │ ├── Spawn JVM, send TaskHandlerParseRequest (ToRuntime) │ + │ │ ├── JVM answers from its own TaskHandler registrations │ + │ │ │ whose dagId is "etl" │ + │ │ ▼ │ + │ │ TaskHandlerParsingResult (ToCoordinator) │ + │ │ → [TaskHandlerDeclaration(task_id, params), ...] │ + │ │ │ + │ │ JavaCoordinator(name="jdk-17") │ + │ │ └── (same call, its own resolved path and handler set) │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 4: Compare the declarations against the parsed Python Dag — │ + │ │ union the handlers returned by every coordinator resolved │ + │ │ in Step 1 before comparing task ids │ + │ │ │ + │ │ Python Dag "etl" (stub tasks) TaskHandlerDeclaration │ + │ │ ────────────────────────────── ────────────────────────────── │ + │ │ task_id ↔ task_id (sets must match) │ + │ │ arg_bindings[*].name ↔ params[*].name (in order) │ + │ │ arg_bindings[*].value_schema ↔ params[*].value_schema │ + │ │ compared only where neither side is null │ + │ │ │ + │ │ On mismatch → DagImportError │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + ├── Validation passed → Return DagImportResult(dags=[dag]) + ▼ +DagModelOperation → PERSIST (Python Dag is the sole DB record) +``` + +### Appendix B — What Is Compared + +Handler declarations from every coordinator resolved in Step 1 are unioned +before comparison, since one Dag's stubs can target several queues. + +- `task_id` sets must match exactly — a missing or extra handler is an error. +- `arg_bindings[*].name` against `params[*].name`, **in order**: both sides bind + positionally. +- `arg_bindings[*].value_schema` against `params[*].value_schema`, compared only + where neither side is null. An unannotated `@task.stub` parameter produces + `null` today, so a strict comparison would make every untyped stub argument a + parse error. + +Any mismatch is raised as `DagImportError` against the Python file, which is the +definition the author can act on. diff --git a/airflow-core/adr/lang-sdk/README.md b/airflow-core/adr/lang-sdk/README.md index b96437fe1ac00..a438acc3488bf 100644 --- a/airflow-core/adr/lang-sdk/README.md +++ b/airflow-core/adr/lang-sdk/README.md @@ -35,7 +35,9 @@ bind core interfaces and apply to every language SDK, not just the Java SDK. - [ADR-0007](0007-taskflow-across-language-boundary.md): TaskFlow across the language boundary — argument binding for Lang-SDK tasks. - [ADR-0008](0008-control-flow-constructs.md): control-flow constructs — grouping, conditional skipping, branching, and triggering a Dag run. - [ADR-0009](0009-provider-operators-as-generated-dsl.md): provider operators as generated, serialization-only DSL in every Lang SDK. -- [ADR-0010](0010-mixed-language-dag-processing.md): mixed-language Dag processing — DagImporter routing and persistence. +- [ADR-0010](0010-lang-sdk-parse-protocol.md): Lang-SDK parse protocol — task handler messages and coordinator verbs. +- [ADR-0011](0011-native-dag-processing.md): native Dag processing — DagImporter registration and routing. +- [ADR-0012](0012-mixed-language-dag-processing.md): mixed-language Dag processing — task handlers are not Dags. Decisions specific to a single SDK stay next to that SDK — for example, the Go SDK's bundle-format decisions live in [`go-sdk/adr/`](../../../go-sdk/adr). Java-SDK-only interface-design decisions — From 57740e70ff592cd922df4a63cac161c87f810d67 Mon Sep 17 00:00:00 2001 From: LIU ZHE YOU Date: Sun, 13 Sep 2026 14:24:31 +0000 Subject: [PATCH 6/9] Lead each Lang-SDK ADR with its diagrams and state the decisions plainly The diagrams carry the decision faster than the prose around them did, so they belong where a reader lands rather than behind an appendix. The argument for each choice is still worth keeping, but it is what someone reads second. Coordinator lookup gains a named answer: for_bundle, beside the existing for_queue, rather than an open question about how a registry finds the coordinators serving a bundle. --- .../lang-sdk/0010-lang-sdk-parse-protocol.md | 230 +++++++-------- .../lang-sdk/0011-native-dag-processing.md | 267 +++++++----------- .../0012-mixed-language-dag-processing.md | 198 +++++-------- 3 files changed, 276 insertions(+), 419 deletions(-) diff --git a/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md b/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md index fed333fe0d778..f6723b1af8df2 100644 --- a/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md +++ b/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md @@ -25,119 +25,51 @@ Proposed ## Context -The Dag processor needs two different answers from a Lang-SDK runtime, and they -are not variations of one question: +The Dag processor asks a Lang-SDK runtime two different questions. "Which Dags does this artifact define?" is answered by the runtime standing in for the parser process, over the +messages [ADR-0004](0004-dag-parsing.md) already defines. "Which task handlers does this artifact register for a `dag_id` Python already owns?" has no answer in those messages, +because a `TaskHandler` registration carries no Dag ([ADR-0012](0012-mixed-language-dag-processing.md)). -- **Which Dags does this artifact define?** Answered by the runtime standing in - for the parser process, over the manager ↔ parser messages - [ADR-0004](0004-dag-parsing.md) already defines. -- **Which task handlers does this artifact register for a `dag_id` Python - already owns?** A `TaskHandler` registration carries no Dag - ([ADR-0012](0012-mixed-language-dag-processing.md)), so the answer is not a - serialized Dag and there is nothing in the existing messages to carry it. +This ADR defines the channel and payload for the second question, and names both parse-side entry points on the coordinator. -This ADR defines the channel and payload for the second question, and names -both parse-side entry points on the coordinator. - -Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec -version `1.0`): `Dag`, `TaskHandler`, `DagRef`, `TaskHandlerRef`, `bundle`, -`register`, `serve`. +Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec version `1.0`). ## Decision -### Handler Declarations Travel on Their Own Channel - -Handler declarations are carried by a dedicated message pair in their own -discriminated unions — `ToRuntime` / `ToCoordinator` — not by a field added to -`DagFileParseRequest`. - -Union membership is the only registration step in the supervisor schema -package: it is what puts a body into the generated snapshot, and therefore into -every SDK's generated models. So the choice of union *is* the wire-contract -decision. Neither existing pair fits, because the runtime plays two roles. When -it parses native Dags it **replaces** the parser process and answers the -manager, which the manager ↔ parser unions already describe. When it answers a -handler query it is a short-lived child of the Python parser, and the manager is -not a peer at all. Reusing the parser unions there would make one union mean two -recipients. - -The payload is deliberately a stronger contract than the one it parallels. -`DagFileParsingResult.serialized_dags` is `list[LazyDeserializedDAG]`, which -reduces in the snapshot to an opaque object — which is why every SDK -reimplements DagSerialization by hand and validates it against shared fixtures -([ADR-0004](0004-dag-parsing.md)). A handler declaration carries no Dag, so it -is fully typed, code-generated, and schema-validated in every SDK. **The -mixed-language path never asks a Lang SDK to implement DagSerialization at -all.** - -Parameter schemas reuse the existing `ArgValueSchema` definition that -`arg_bindings` already carries -([ADR-0007](0007-taskflow-across-language-boundary.md)), so validation compares -like against like rather than translating between two vocabularies. Appendix A -gives the message definitions and the two properties of that field — -nullability and ordering — that validation has to respect. - -### Two Parse Verbs, Because There Are Two Conversations - -`BaseCoordinator` gains `parse_dag` and `parse_task_handler` alongside the -shipped `execute_task`. They are **not symmetric siblings**, which is the -reason not to collapse them into one `parse(request)`: - -- `parse_dag` is **bridge mode**. The coordinator becomes the parser process - and raw-forwards the manager's request and the runtime's reply. It never - reads the payload. -- `parse_task_handler` is **query mode**. The coordinator is itself the peer: - it sends one request and decodes one reply on the channel above. - -One collapsed method would also erase a real deployment distinction — a -coordinator can serve mixed-language handlers with no interest in native Dag -parsing, and an absent method says so better than a runtime rejection. -Appendix B gives the layering and the command hooks. - -## Consequences - -- **The handler channel is typed where the Dag channel is opaque.** Every SDK - gets generated models for the declaration shape, and nothing on this path - requires a DagSerialization implementation. -- Each Lang-SDK runtime implements a second request type rather than a new field - on the existing one. That is the cost of handlers not being Dags: a `DagRef` - and a `TaskHandlerRef` have different shapes, so one request/result pair could - not have carried both. -- Adding `ToRuntime` / `ToCoordinator` extends the set of unions the supervisor - schema package introspects, and regenerating the snapshot is what propagates - the new bodies into each SDK's generated models. -- Naming supersedes ADR-0004's `run_dag_parsing` / `dag_parsing_cmd`, matching - the shipped `execute_task` / `_build_execute_task_command` pair instead. -- Terms here track Language SDK spec `1.0`. A spec rename of `TaskHandler` - lands in this ADR too. - -## References +### Two conversations, two verbs -- [ADR-0004](0004-dag-parsing.md) — coordinator subprocess bridge, - `DagFileParseRequest` / `DagFileParsingResult`, `can_handle_dag_file` -- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / - `TaskArgBinding` / `ArgValueSchema` -- [ADR-0011](0011-native-dag-processing.md) — who calls `parse_dag` -- [ADR-0012](0012-mixed-language-dag-processing.md) — who calls - `parse_task_handler`, and what it compares the reply against -- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) -- `task-sdk/src/airflow/sdk/execution_time/schema/` — the supervisor schema - version bundle, the union registry, and the generated snapshot -- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs +``` +parse_dag — bridge mode parse_task_handler — query mode + + manager coordinator + │ DagFileParseRequest │ TaskHandlerParseRequest + ▼ ▼ + coordinator (raw byte forward) runtime + ▼ │ TaskHandlerParsingResult + runtime ▼ + │ DagFileParsingResult coordinator + ▼ + manager +``` -## Appendix — For Implementation +`parse_dag` forwards bytes and never reads the payload. `parse_task_handler` is itself the peer: it sends one request and decodes one reply. They are two methods, not one +`parse(request)`, because a coordinator can serve handlers without serving native Dag parsing. Appendix A has the longer argument. -### Appendix A — Message Definitions +### Handler declarations get their own unions ``` -ToRuntime = TaskHandlerParseRequest coordinator → runtime -ToCoordinator = TaskHandlerParsingResult runtime → coordinator +ToRuntime = TaskHandlerParseRequest coordinator → runtime +ToCoordinator = TaskHandlerParsingResult runtime → coordinator ``` +Not a field on `DagFileParseRequest`. Union membership is the only registration step in the supervisor schema package, so picking the union is the wire-contract decision. The +existing pairs describe a runtime answering the manager; here the runtime answers the coordinator. + +### Message shapes + ``` class TaskHandlerParseRequest: - file: str # the artifact the coordinator resolved - dag_id: str # scope: handlers bound to this dag_id only + file: str # the artifact the coordinator resolved + dag_id: str # scope: handlers bound to this dag_id only type: Literal["TaskHandlerParseRequest"] class TaskHandlerParsingResult: @@ -158,30 +90,13 @@ class TaskHandlerParam: required: bool # the handler declares no default ``` -Two properties of `value_schema` are load-bearing for validation: - -- **It is nullable on both sides.** An unannotated `@task.stub` parameter - produces `value_schema: null` today, so validation compares schemas only - where neither side is null and falls back to name-and-arity otherwise. - Without that, every untyped stub argument becomes a parse error. -- **`params` is ordered.** `LiteralArgBinding` and `XComArgBinding` are each - documented as "one positional stub-task argument", so position is part of the - contract rather than incidental. - -A declaration carries no class, method, or source location. -[ADR-0006](0006-no-lang-sdk-source-display.md) rules out Lang-SDK source -display, and putting it on the wire would invite a consumer to render it. +`value_schema` reuses the `ArgValueSchema` definition `arg_bindings` already carries ([ADR-0007](0007-taskflow-across-language-boundary.md)), so both sides of a comparison are the +same type. Two properties matter to validation: the field is nullable on both sides, and `params` is ordered. Appendix B says what that forces. -Registration mechanics: `registered_models_by_name()` in -`task-sdk/src/airflow/sdk/execution_time/schema/` introspects a fixed set of -unions — `ToTask` / `ToSupervisor` and `ToManager` / `ToDagProcessor` — so -adding `ToRuntime` / `ToCoordinator` extends that set. Two prek hooks guard the -snapshot, and their interaction on a first-introduction body needs checking -against the hooks rather than against that package's `AGENTS.md`, which says no -`VersionChange` is required while the second hook fails when the snapshot moves -with nothing under `versions/` touched. +`task_handlers` is the counterpart to `DagFileParsingResult.serialized_dags`, but fully typed. `serialized_dags` is `list[LazyDeserializedDAG]`, which is an opaque object in the +schema snapshot. A handler declaration carries no Dag, so it code-generates and schema-validates in every SDK, and nothing on this path needs a DagSerialization implementation. -### Appendix B — Coordinator Interface +### Coordinator interface ``` BaseCoordinator execution_time/coordinator.py @@ -200,15 +115,64 @@ JavaCoordinator · ExecutableCoordinator · NodeCoordinator supply the three commands; no socket or protocol code ``` -Each hook returns its own `subprocess_schema_version`, so handler parsing -negotiates the supervisor schema through the mechanism task execution already -uses, and `None` disables migration exactly as it does today. -`can_handle_dag_file` still gates native Dag parsing only; handler queries -resolve through the coordinator registry instead. - -Artifact roots are resolved per mode by `_init_root_source`, but published -through `_get_scan_roots()`, which is scoped to an active task and raises -outside one. Both parse-side commands need the same roots with no -`TaskInstance` in hand, so the scope that publishes them has to open for a -parse as well as for a task. [ADR-0011](0011-native-dag-processing.md) covers -the modes themselves. +Names follow the shipped `execute_task` / `_build_execute_task_command` pair and supersede ADR-0004's `run_dag_parsing` / `dag_parsing_cmd`. Each hook returns its own +`subprocess_schema_version`, so handler parsing negotiates the schema the same way task execution does. + +## Consequences + +- The handler channel is typed; the Dag channel is opaque. Every SDK gets generated models for the declaration shape. +- Each runtime implements a second request type instead of a new field on the existing one. A `DagRef` and a `TaskHandlerRef` have different shapes, so one request/result pair + could not carry both. +- `ToRuntime` / `ToCoordinator` extends the set of unions the supervisor schema package introspects. Regenerating the snapshot is what propagates the bodies into each SDK's + generated models. +- `can_handle_dag_file` still gates native Dag parsing only. Handler queries resolve through the coordinator registry. +- Terms track Language SDK spec `1.0`. A spec rename of `TaskHandler` lands here too. + +## References + +- [ADR-0004](0004-dag-parsing.md) — coordinator subprocess bridge, `DagFileParseRequest` / `DagFileParsingResult`, `can_handle_dag_file` +- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display +- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / `TaskArgBinding` / `ArgValueSchema` +- [ADR-0011](0011-native-dag-processing.md) — who calls `parse_dag` +- [ADR-0012](0012-mixed-language-dag-processing.md) — who calls `parse_task_handler`, and what it compares the reply against +- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) +- `task-sdk/src/airflow/sdk/execution_time/schema/` — supervisor schema version bundle, union registry, generated snapshot +- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs + +## Appendix + +### Appendix A — Why two verbs and a separate channel + +The runtime plays two roles, and they differ in who it is talking to. Parsing native Dags, it replaces the parser process and answers the manager, which is exactly what the manager +↔ parser unions describe. Answering a handler query, it is a short-lived child of the Python parser, and the manager is not a peer at all. Reusing the parser unions for both would +make one union mean two different recipients, and a runtime could not tell from the union which role it was in. + +The same split is why the verbs stay separate. Collapsing them into one `parse(request)` would hide a real deployment distinction: a coordinator can serve mixed-language handlers +with no interest in native Dag parsing, and an absent method states that better than a runtime rejection does. The two also differ in mechanism — one is a byte forwarder that never +decodes the payload, the other decodes a reply it asked for — so the shared code is the subprocess plumbing underneath, not the public method. + +A boolean on `DagFileParseRequest` was the alternative. It cannot work: a `DagRef` and a `TaskHandlerRef` are different payloads, not two subsets of one, so the flag would select +between shapes the result type cannot both hold. + +### Appendix B — What the nullable, ordered parameter list forces + +`value_schema` is nullable on both sides. An unannotated `@task.stub` parameter produces `value_schema: null` today, so validation compares schemas only where neither side is null, +and falls back to name-and-arity otherwise. A strict comparison would turn every untyped stub argument into a parse error. + +`params` is ordered because `LiteralArgBinding` and `XComArgBinding` are each documented as "one positional stub-task argument". Position is part of the contract, not incidental, +and both sides bind positionally. + +A declaration carries no class, method, or source location. [ADR-0006](0006-no-lang-sdk-source-display.md) rules out Lang-SDK source display, and putting it on the wire would +invite a consumer to render it. + +### Appendix C — Schema registration mechanics + +`registered_models_by_name()` in `task-sdk/src/airflow/sdk/execution_time/schema/` introspects a fixed set of unions — `ToTask` / `ToSupervisor` and `ToManager` / `ToDagProcessor` +— so adding `ToRuntime` / `ToCoordinator` extends that set. + +Two prek hooks guard the generated snapshot. Their interaction on a first-introduction body needs checking against the hooks rather than against that package's `AGENTS.md`: the doc +says no `VersionChange` is required for a new body, while `check-supervisor-schemas-versions` fails when the snapshot moves and nothing under `versions/` was touched. + +Artifact roots are resolved per mode by `_init_root_source`, but published through `_get_scan_roots()`, which is scoped to an active task and raises outside one. Both parse-side +commands need those roots with no `TaskInstance` in hand, so the scope that publishes them has to open for a parse as well as for a task. [ADR-0011](0011-native-dag-processing.md) +covers the modes themselves. diff --git a/airflow-core/adr/lang-sdk/0011-native-dag-processing.md b/airflow-core/adr/lang-sdk/0011-native-dag-processing.md index 2e5a7ac72ed0a..4bf4b028dcec8 100644 --- a/airflow-core/adr/lang-sdk/0011-native-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0011-native-dag-processing.md @@ -25,111 +25,17 @@ Proposed ## Context -`AbstractDagImporter` (AIP-85, `task-sdk/src/airflow/sdk/importers/`) makes the -Dag processor aware of source formats beyond `.py`: `can_handle`, -`list_dag_definitions`, `import_definition`, `get_source_code`, returning -`DagImportResult.dags: list[DAG]` — `airflow.sdk.DAG` objects. +`AbstractDagImporter` (AIP-85, `task-sdk/src/airflow/sdk/importers/`) makes the Dag processor aware of source formats beyond `.py`. It exposes `can_handle`, `list_dag_definitions`, +`import_definition` and `get_source_code`, and returns `DagImportResult.dags: list[DAG]` — `airflow.sdk.DAG` objects. -A Lang-SDK artifact (JAR, packed Go executable) can be one of those sources: a -Dag authored entirely in Go or Java, with `Dag(spec)` on the SDK side. Parsing -it means launching a runtime, which is what a coordinator does. This ADR settles -where that importer comes from, which coordinator instance backs it, and how -the two are kept from being configured twice. +A Lang-SDK artifact can be one of those sources: a Dag authored entirely in Go or Java, with `Dag(spec)` on the SDK side. Parsing it means launching a runtime, which is what a +coordinator does. This ADR settles where that importer comes from, which coordinator instance backs it, and how the two avoid being configured twice. -`JavaDagImporter` and `JavaCoordinator` are the examples throughout; -`ExecutableDagImporter` (Go) and `NodeDagImporter` (TypeScript) follow the same -shape. +`JavaDagImporter` and `JavaCoordinator` are the examples throughout. `ExecutableDagImporter` (Go) and `NodeDagImporter` (TypeScript) follow the same shape. ## Decision -### A Lang-SDK DagImporter Comes From Its Coordinator - -`AbstractDagImporter` is about source formats. The coordinator is about -processes. A Lang-SDK importer composes one, and the coordinator is what hands -it out — `JavaCoordinator.get_dag_importer()` returns a `JavaDagImporter` -already bound to that coordinator. - -An operator therefore never configures an importer's coordinator: that wiring is -an implementation detail of the coordinator that produced it, and -`[sdk] coordinators` stays the one place a runtime is declared. The existing -`[dag_processor] dag_importer_configs` path remains the door for importers with -no runtime behind them — a YAML importer, say. A Lang SDK never arrives that -way. Appendix A gives the registration tiers. - -### One Coordinator Instance Owns a DagBundle - -Importers are keyed by file extension, one per extension, with the later -registration evicting the earlier one and logging a warning. Two -`JavaCoordinator` instances on different JDKs would both claim `.jar`. - -The boundary that resolves this is deployment-shaped rather than a tie-break: a -coordinator instance owns a DagBundle. `SubprocessCoordinator` already -classifies that ownership at construction into three artifact sources, and only -`NAMED_BUNDLE` names a bundle — which makes `NAMED_BUNDLE` the mode a deployment -running two runtimes of the same language has to use. `EXPLICIT_ROOT` has no -DagBundle at all, so its artifacts are never scanned by the Dag processor and it -cannot produce a Dag to persist; it stays a coordinator reached by queue, for -mixed-language work only. Appendix B covers all three modes, the roots a parse -needs, and extensionless artifacts. - -### The Result Has to Be `airflow.sdk.DAG` - -`JavaDagImporter.import_definition` calls `parse_dag` -([ADR-0010](0010-lang-sdk-parse-protocol.md)) and the runtime serializes its -`Dag` registrations. `TaskHandler` registrations have no Dag to serialize, so a -mixed-language `dag_id` is never a candidate for persistence here — there is -nothing to filter out ([ADR-0012](0012-mixed-language-dag-processing.md)). - -Because `DagImportResult.dags` is `list[DAG]`, the importer wraps each -serialized entry as a `LazyDeserializedDAG` and transforms it into an -`airflow.sdk.DAG` before returning. The transform is mechanical — the serialized -form encodes the SDK Dag's own fields — but `LazyDeserializedDAG` lives in -`airflow-core` today, so where a Task SDK importer reaches it from is settled by -AIP-85's own `list[DAG]` / `list[LazyDeserializedDAG]` discussion rather than -here. - -Appendix C walks both flows: a pure Python file, and a Lang-SDK artifact. - -## Consequences - -- **A Lang-SDK importer is never configured by hand.** `[sdk] coordinators` is - the one place a runtime is declared; the importer follows from it through - `get_dag_importer()`, so the same runtime has no second configuration site to - drift from. -- **One coordinator instance per DagBundle** becomes a deployment constraint: - two JDKs mean two `dag_bundle_name` values and two bundle-scoped registries. - It is what keeps extension-keyed importer registration unambiguous, and it - makes `NAMED_BUNDLE` the mode a multi-runtime deployment has to use. -- A coordinator in `EXPLICIT_ROOT` mode cannot back a Dag importer at all. Its - artifacts live outside any DagBundle, so nothing scans them. -- `CoordinatorManager` needs a reverse lookup it does not have: it resolves - coordinators by queue only today, with nothing that answers "which - coordinators serve this bundle". -- A packed Go bundle claims the **empty** extension, which three call sites - currently treat as absent rather than as a key. -- `get_source_code` is abstract, so every Lang-SDK importer must implement it, - and a native Lang-SDK Dag has no Python source to return. What it should - return, and how that squares with - [ADR-0006](0006-no-lang-sdk-source-display.md), is not settled here. - -## References - -- [ADR-0010](0010-lang-sdk-parse-protocol.md) — `parse_dag` and the coordinator - interface it belongs to -- [ADR-0012](0012-mixed-language-dag-processing.md) — why `TaskHandler` - registrations never reach a `DagImporter` -- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor`, - build-time artifact inventory -- [ADR-0004](0004-dag-parsing.md) — `can_handle_dag_file`, the subprocess bridge -- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display -- `task-sdk/src/airflow/sdk/importers/` — `AbstractDagImporter`, - `DagImportResult`, `DagImporterRegistry` -- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter -- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs - -## Appendix — For Implementation - -### Appendix A — Registration +### The coordinator hands out its importer ``` BaseCoordinator.get_dag_importer() -> AbstractDagImporter | None @@ -137,89 +43,54 @@ BaseCoordinator.get_dag_importer() -> AbstractDagImporter | None JavaCoordinator.get_dag_importer() -> JavaDagImporter(coordinator=self) ``` +The importer comes back already bound to the coordinator, so an operator never configures which coordinator an importer uses. `[sdk] coordinators` stays the one place a runtime is +declared. + +### Registration order + ``` DagImporterRegistry.from_config(bundle_name) ├── defaults PythonDagImporter, ZipImporter - ├── coordinators serving bundle_name (new) + ├── CoordinatorManager.for_bundle(bundle_name) (new) │ └── register(coordinator.get_dag_importer()) ├── [dag_processor] dag_importer_configs (global, unchanged) └── that bundle's own `importers` list (unchanged) ``` -`get_importer_registry(bundle_name)` is already cached per bundle, so the -scoping mechanism is in place. What `CoordinatorManager` still needs is the -reverse lookup: it resolves coordinators by queue only today — -`for_queue(queue)`, driven by `[sdk] queue_to_coordinator` — with nothing that -answers "which coordinators serve this bundle". +`dag_importer_configs` remains the door for importers with no runtime behind them — a YAML importer, say. A Lang SDK never arrives that way. + +### `CoordinatorManager.for_bundle` + +``` +CoordinatorManager + ├── for_queue(queue) → one coordinator (shipped — task execution) + └── for_bundle(name) → the coordinators serving that bundle (new) +``` -### Appendix B — Artifact Sources +`for_queue` answers "who runs this task". `for_bundle` answers "who can parse Dags in this bundle", which is what the registry tier above needs. It reads the same `[sdk] +coordinators` specs, selecting by the artifact source below. -`SubprocessCoordinator` classifies artifact ownership at construction: +### One coordinator instance owns a DagBundle ``` EXPLICIT_ROOT jars_root / executables_root / bundles_root - → no DagBundle at all + → no DagBundle at all → not returned by for_bundle NAMED_BUNDLE dag_bundle_name → that bundle, at the version current when work starts TASK_BUNDLE neither set - → the bundle the delegating Dag itself lives in, - at the run's version + → the bundle the delegating Dag lives in, at the run's version ``` -The first two are mutually exclusive, and both that conflict and a -`dag_bundle_name` naming an unconfigured bundle are rejected at construction. -Which mode a coordinator is in decides whether it can back an importer: +Importers are keyed by file extension, one per extension, so two `JavaCoordinator` instances on different JDKs would both claim `.jar`. Only `NAMED_BUNDLE` names a bundle, which +makes it the mode a deployment running two runtimes of the same language has to use. `EXPLICIT_ROOT` has no DagBundle, so nothing scans its artifacts and it cannot produce a Dag to +persist; it serves mixed-language work only. Appendix A covers the three modes in full. -- **`NAMED_BUNDLE` registers into that bundle's registry.** Two - `dag_bundle_name` values, two bundle-scoped registries, `.jar` claimed once in - each. -- **`TASK_BUNDLE` has no fixed bundle**, so its importer registers into every - bundle-scoped registry. Sound only while it is the sole claimant of its - extension — the co-located single-runtime deployment. -- **`EXPLICIT_ROOT` has no DagBundle at all**, so the Dag processor never scans - its artifacts and it cannot produce a Dag to persist. - -#### Resolving artifact roots at parse time - -`_init_root_source` already resolves roots for all three modes, but publishes -them through `_get_scan_roots()`, which is scoped to an active task and raises -outside one. Both parse-side commands need the same roots with no -`TaskInstance` in hand: `EXPLICIT_ROOT` and `NAMED_BUNDLE` resolve from the -coordinator's own configuration, and `TASK_BUNDLE` resolves against the bundle -the Dag processor is parsing. The scope that publishes the roots has to open -for a parse as well as for a task. - -#### Extensionless artifacts - -A packed Go bundle has no suffix, so `ExecutableDagImporter` claims the **empty** -extension as a first-class key, rather than depending on a `can_handle` scan -whose winner varies with registration order. Three places assume a non-empty -suffix today: `_normalize_extensions` rewrites `""` to `"."`; `get_importer` and -`can_handle` guard on `if suffix:`, which skips the extension map entirely for an -extensionless file; and `find_file_dag_definitions` filters on -`path.suffix.lower()`. Empty has to pass through all three, with the guards -testing `suffix is not None`. - -### Appendix C — Flow Walkthroughs - -#### Pure Python Dag (no `@task.stub`) - -``` -PythonDagImporter.import_definition(definition, bundle=...) - │ - ├── Parse → DAG objects - ├── serialize_dag(dag) → no stub tasks, nothing to cross-validate - ├── Return DagImportResult(dags=[dag]) - ▼ -DagModelOperation → PERSIST -``` - -#### Native Lang-SDK Dag +### The result has to be `airflow.sdk.DAG` ``` JavaDagImporter.import_definition(definition, bundle=...) │ - ├── JavaCoordinator.parse_dag(...) ← bridge mode + ├── JavaCoordinator.parse_dag(...) ← bridge mode (ADR-0010) │ ├── _build_parse_dag_command() → (command, schema_version) │ └── Spawn JVM, forward fd 0 ⇄ comm socket │ manager ──DagFileParseRequest───▶ JVM @@ -227,7 +98,7 @@ JavaDagImporter.import_definition(definition, bundle=...) │ ┌────────────────────────────────────────────────────┐ │ │ serialized_dags: ["java_report"] (@Builder.Dag) │ │ └────────────────────────────────────────────────────┘ - │ The TaskHandler registrations have no Dag to serialize — + │ TaskHandler registrations have no Dag to serialize — │ no "etl" entry exists to be discarded. │ ├── LazyDeserializedDAG(data=...) → airflow.sdk.DAG @@ -235,3 +106,77 @@ JavaDagImporter.import_definition(definition, bundle=...) ▼ DagModelOperation → PERSIST "java_report" only ``` + +For comparison, a pure Python file with no stub tasks: + +``` +PythonDagImporter.import_definition(definition, bundle=...) + │ + ├── Parse → DAG objects + ├── serialize_dag(dag) → no stub tasks, nothing to cross-validate + ├── Return DagImportResult(dags=[dag]) + ▼ +DagModelOperation → PERSIST +``` + +`DagImportResult.dags` is `list[DAG]`, so the importer wraps each serialized entry as a `LazyDeserializedDAG` and transforms it into an `airflow.sdk.DAG`. `LazyDeserializedDAG` +lives in `airflow-core` today, so where a Task SDK importer reaches it from is settled by AIP-85's own `list[DAG]` / `list[LazyDeserializedDAG]` discussion, not here. + +## Consequences + +- A Lang-SDK importer is never configured by hand. The runtime is declared once, in `[sdk] coordinators`, and the importer follows from it. +- One coordinator instance per DagBundle becomes a deployment constraint: two JDKs mean two `dag_bundle_name` values and two bundle-scoped registries. This is what keeps + extension-keyed registration unambiguous. +- A coordinator in `EXPLICIT_ROOT` mode cannot back a Dag importer. Its artifacts live outside any DagBundle, so nothing scans them. +- `CoordinatorManager` gains `for_bundle`, a second lookup axis beside `for_queue`. +- A packed Go bundle claims the empty extension, which three call sites currently treat as absent rather than as a key. Appendix B lists them. +- `get_source_code` is abstract, so every Lang-SDK importer must implement it, and a native Lang-SDK Dag has no Python source to return. What it should return, and how that squares + with [ADR-0006](0006-no-lang-sdk-source-display.md), is not settled here. + +## References + +- [ADR-0010](0010-lang-sdk-parse-protocol.md) — `parse_dag` and the coordinator interface it belongs to +- [ADR-0012](0012-mixed-language-dag-processing.md) — why `TaskHandler` registrations never reach a `DagImporter` +- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor`, build-time artifact inventory +- [ADR-0004](0004-dag-parsing.md) — `can_handle_dag_file`, the subprocess bridge +- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display +- `task-sdk/src/airflow/sdk/importers/` — `AbstractDagImporter`, `DagImportResult`, `DagImporterRegistry` +- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter +- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs + +## Appendix + +### Appendix A — Artifact sources and what `for_bundle` returns + +`SubprocessCoordinator` classifies artifact ownership at construction. The explicit root and `dag_bundle_name` are mutually exclusive, and both that conflict and a +`dag_bundle_name` naming an unconfigured bundle are rejected there. + +`NAMED_BUNDLE` is the unambiguous case. `for_bundle(name)` returns it when its `dag_bundle_name` matches, so its importer lands in exactly one bundle-scoped registry. Two +`dag_bundle_name` values give two registries, and `.jar` is claimed once in each. + +`TASK_BUNDLE` has no fixed bundle — its artifacts ride along with whichever Dag delegates to them — so `for_bundle` returns it for every bundle and its importer registers +everywhere. That is sound only while it is the sole claimant of its extension, which is the co-located single-runtime deployment. + +`EXPLICIT_ROOT` points at a filesystem path outside any DagBundle. The Dag processor never scans it, so there is no file for an importer to claim and `for_bundle` never returns it. +Such a coordinator is reachable only by queue, for mixed-language work. + +`get_importer_registry(bundle_name)` is already cached per bundle, and `CoordinatorManager` caches instances separately, so the two caches have to be reset together. + +### Appendix B — Extensionless artifacts + +A packed Go bundle has no suffix. `ExecutableDagImporter` claims the empty extension as a first-class key rather than depending on a `can_handle` scan, whose winner varies with +registration order because `_ordered_importers` is scanned in reverse. + +Three places assume a non-empty suffix today: + +- `_normalize_extensions` rewrites `""` to `"."`. +- `get_importer` and `can_handle` guard on `if suffix:`, which skips the extension map entirely for an extensionless file. +- `find_file_dag_definitions` filters on `path.suffix.lower()`. + +Empty has to pass through all three, with the guards testing `suffix is not None`. + +### Appendix C — Resolving artifact roots at parse time + +`_init_root_source` already resolves roots for all three modes, but publishes them through `_get_scan_roots()`, which is scoped to an active task and raises outside one. Both +parse-side commands need the same roots with no `TaskInstance` in hand: `EXPLICIT_ROOT` and `NAMED_BUNDLE` resolve from the coordinator's own configuration, and `TASK_BUNDLE` +resolves against the bundle the Dag processor is parsing. The scope that publishes the roots has to open for a parse as well as for a task. diff --git a/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md index 1b82e3f61523c..b2625ced61365 100644 --- a/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md @@ -25,31 +25,23 @@ Proposed ## Context -A Lang-SDK artifact (JAR, packed Go executable) can serve either of the two -authoring features the Language SDK spec fixes, or both at once: +A Lang-SDK artifact can serve either of the two authoring features the Language SDK spec fixes, or both at once: | Feature | Who owns the graph | Author writes | Artifact contributes | |---------------------------------|----------------------------|----------------------------------|----------------------| | **Mixed Language Task Handler** | Python, via `@task.stub` | `TaskHandler(dagId, taskId, fn)` | Only task bodies | | **Native Dag** | the Lang-SDK source itself | `Dag(spec)` | The entire Dag | -In the mixed-language role the artifact used to author a Dag too — under the -same `dag_id` the Python file already owns — so two conflicting definitions -reached persistence and something downstream had to choose between them. This -ADR removes the conflict at the authoring interface instead of resolving it -afterwards, and defines how the Python importer validates a stub task against -the handler that implements it. +In the mixed-language role the artifact used to author a Dag too, under the same `dag_id` the Python file already owns. Two conflicting definitions reached persistence and +something downstream had to choose between them. This ADR removes the conflict at the authoring interface, and defines how the Python importer validates a stub task against the +handler that implements it. -Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec -version `1.0`): `Dag`, `TaskHandler`, `DagRef`, `TaskHandlerRef`, `bundle`, -`register`, `serve`. The examples use the Java and Go spellings -(`@Builder.TaskHandler` / `airflow.TaskHandler`); TypeScript spells the same -terms in its own idiom. Note that the spec's `bundle` is the SDK-side -registration container, not Airflow's `DagBundle` — both appear below. +Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec version `1.0`). Note that the spec's `bundle` is the SDK-side registration container, not Airflow's +`DagBundle` — both appear below. ## Decision -### Differentiate at the Authoring Interface +### Two registration kinds, one bundle ``` ┌──────────────────────────────────────────────────────────────────────────┐ @@ -66,57 +58,11 @@ registration container, not Airflow's `DagBundle` — both appear below. └──────────────────────────────────────────────────────────────────────────┘ ``` -`TaskHandler` is a separate interface, not a second flavour of `Dag`. It names -the `(dagId, taskId)` pair Python already declared and the `fn` that implements -it, and returns a `TaskHandlerRef`: no schedule, no task graph, no `dag_id` of -its own to persist. `Dag(spec)` returns a `DagRef`, which owns all three. The -spelling differs per SDK; the split does not. - -The spec puts both kinds in one `bundle`, takes them through one `register` verb -in any mixture, and serves the process with one `bundle.serve()`. So the bundle -cannot be the discriminator — and does not need to be. Dag parsing draws on the -`Dag` registrations ([ADR-0011](0011-native-dag-processing.md)); handler -validation draws on the `TaskHandler` registrations. Each registration already -says which kind it is. - -The alternative was to keep authoring the Dag on both sides and mark the -Lang-SDK copy with a per-Dag flag (`is_mixed_language_dag`) so importers know -which copy to drop. That keeps producing the definition it then has to discard, -and pushes the question "is this Dag real?" onto every consumer of a serialized -Dag. Taking the Dag out of the authoring interface answers that question once, -for everyone. The only marker left in serialization is `is_stub`, and it is -per-task. - -### Validation Is Driven by the Python Importer - -The `PythonDagImporter` owns validation. When it finds a stub task it resolves -the stub's `queue` to a coordinator, calls `parse_task_handler` -([ADR-0010](0010-lang-sdk-parse-protocol.md)) for the same `dag_id`, and -compares the returned declarations against the Dag it just parsed. No -`DagImporter` is involved — this is a single request/response rather than a -discovery operation, and what comes back is not a Dag. - -Resolution goes through the coordinator registry, not the filesystem: the Dag -processor is file-at-a-time and the `PythonDagImporter` never sees the `.jar`. -This also means the Python Dag and the Lang-SDK artifact **do not need to be in -the same DagBundle**. Nothing here needs an `airflow.sdk.DAG` round-trip either -— validation returns the Dag the Python parser already built. Appendix A walks -the four steps and Appendix B states what is compared. - -### Decision Matrix +A `TaskHandlerRef` has no schedule, no task graph, and no `dag_id` of its own to persist. A `DagRef` has all three. Dag parsing draws on the `Dag` registrations +([ADR-0011](0011-native-dag-processing.md)); validation draws on the `TaskHandler` registrations. Because both kinds share one bundle and one `register` verb, the bundle cannot be +the discriminator — the registration kind is. Appendix A gives the rejected alternative and why. -| Caller | Coordinator call | What comes back | Action | -|----------------------------------------------|----------------------------------------------|------------------------------------------|--------------------------------------------------| -| `PythonDagImporter` | — (parses the Python file itself) | its own parsed Dag | PERSIST | -| `PythonDagImporter` (via stub → Coordinator) | `parse_task_handler`, scoped to one `dag_id` | `TaskHandlerParsingResult` | VALIDATE only — not a Dag, so nothing to persist | -| `JavaDagImporter` | `parse_dag` | `DagFileParsingResult`, native Dags only | PERSIST | - -There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — -and nothing reading a `DagImporter`'s results — ever sees one. - -### Why the Distinction Lives on the Interface, Not the Bundle - -A single artifact registers both kinds through one `register` call: +### One artifact, both kinds ``` analytics.jar @@ -130,55 +76,9 @@ bundle.register(javaReport, etlExtract, etlTransform); // one verb, both kinds bundle.serve(args); ``` -One artifact, one bundle, one `register` call — so neither the file nor the -bundle can tell a `DagImporter` what it is holding. The registration kind can, -and does: the split is per registration, not per file and not per bundle. +The split is per registration, not per file and not per bundle. -## Consequences - -- **Python leads, Lang-SDK follows.** `PythonDagImporter` persists the Dag and - drives validation via `queue → Coordinator → parse_task_handler`, comparing - the returned declarations against its own parsed Dag. -- **A mixed-language `dag_id` never appears in Dag processing results.** The - artifact registers handlers only, so no `Dag` registration exists for a - `dag_id` a Python file already owns. Everything downstream — `DagImportResult`, - import errors, the Dag list — sees exactly one record per `dag_id`, with no - flag to interpret. -- Stub/implementation mismatches (missing handler, extra handler, parameter name - or order, incompatible schema) surface as `DagImportError` at parse time. An - unannotated stub argument is checked by name and position only. -- The Python Dag and Lang-SDK artifact can live in different DagBundles — - resolution goes through the coordinator registry, not the filesystem. -- A single Dag can have stubs targeting different queues (some Java, some Go) — - each resolves to its own coordinator instance independently, and validation - unions their declarations before comparing task ids. -- **Mixed-language is Python-primary only.** Lang-SDK runtimes cannot define stub - operators — the reverse direction (a native Dag delegating tasks to Python) is - not supported. -- **No per-Dag flag**, no schema migration, no new `DagModel` column, no - REST/UI change. -- Terms here track Language SDK spec `1.0`. A spec rename of `TaskHandler`, or - of the `register`/`serve` verbs, lands in this ADR too. - -## References - -- [ADR-0010](0010-lang-sdk-parse-protocol.md) — `parse_task_handler` and the - `TaskHandlerParsingResult` shape this ADR compares against -- [ADR-0011](0011-native-dag-processing.md) — the `Dag`-registration half, and - the importer that persists it -- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor`, - build-time artifact inventory -- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display - for mixed-language Dags -- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / - `TaskArgBinding` / `ArgValueSchema` -- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) -- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs -- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter - -## Appendix — For Implementation - -### Appendix A — Validation Walkthrough +### Validation is driven by the Python importer ``` PythonDagImporter.import_definition(definition, bundle=...) @@ -261,18 +161,66 @@ PythonDagImporter.import_definition(definition, bundle=...) DagModelOperation → PERSIST (Python Dag is the sole DB record) ``` -### Appendix B — What Is Compared +The `PythonDagImporter` owns validation. It resolves each stub's `queue` to a coordinator, calls `parse_task_handler` ([ADR-0010](0010-lang-sdk-parse-protocol.md)) for the same +`dag_id`, and compares the reply against the Dag it just parsed. No `DagImporter` is involved: this is one request/response, and what comes back is not a Dag. + +Resolution goes through the coordinator registry, not the filesystem, so the Python Dag and the Lang-SDK artifact **do not need to be in the same DagBundle**. Nothing here needs an +`airflow.sdk.DAG` round-trip either — validation returns the Dag the Python parser already built. Appendix B states exactly what is compared. + +### Decision matrix + +| Caller | Coordinator call | What comes back | Action | +|----------------------------------------------|----------------------------------------------|------------------------------------------|--------------------------------------------------| +| `PythonDagImporter` | — (parses the Python file itself) | its own parsed Dag | PERSIST | +| `PythonDagImporter` (via stub → Coordinator) | `parse_task_handler`, scoped to one `dag_id` | `TaskHandlerParsingResult` | VALIDATE only — not a Dag, so nothing to persist | +| `JavaDagImporter` | `parse_dag` | `DagFileParsingResult`, native Dags only | PERSIST | + +There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — and nothing reading a `DagImporter`'s results — ever sees one. + +## Consequences + +- Python leads, Lang-SDK follows. `PythonDagImporter` persists the Dag and drives validation via `queue → Coordinator → parse_task_handler`. +- A mixed-language `dag_id` never appears in Dag processing results. No `Dag` registration exists for a `dag_id` a Python file already owns, so everything downstream sees exactly + one record per `dag_id`, with no flag to interpret. +- Stub/implementation mismatches — missing handler, extra handler, parameter name or order, incompatible schema — surface as `DagImportError` at parse time. An unannotated stub + argument is checked by name and position only. +- The Python Dag and Lang-SDK artifact can live in different DagBundles. +- A single Dag can have stubs targeting different queues, some Java, some Go. Each resolves to its own coordinator instance, and validation unions their declarations before + comparing task ids. +- Mixed-language is Python-primary only. Lang-SDK runtimes cannot define stub operators; a native Dag cannot delegate tasks to Python. +- No per-Dag flag, no schema migration, no new `DagModel` column, no REST/UI change. +- Terms track Language SDK spec `1.0`. A spec rename of `TaskHandler`, or of the `register` / `serve` verbs, lands here too. + +## References + +- [ADR-0010](0010-lang-sdk-parse-protocol.md) — `parse_task_handler` and the `TaskHandlerParsingResult` shape this ADR compares against +- [ADR-0011](0011-native-dag-processing.md) — the `Dag`-registration half, and the importer that persists it +- [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor`, build-time artifact inventory +- [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display for mixed-language Dags +- [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / `TaskArgBinding` / `ArgValueSchema` +- Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) +- [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs +- [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter + +## Appendix + +### Appendix A — Why the split lives on the interface + +The alternative was to keep authoring the Dag on both sides and mark the Lang-SDK copy with a per-Dag flag, `is_mixed_language_dag`, so importers knew which copy to drop. That +keeps producing the definition it then has to discard, and it pushes the question "is this Dag real?" onto every consumer of a serialized Dag — the importer, the persistence layer, +anything later reading the record. Taking the Dag out of the authoring interface answers the question once, at the only point where the answer is known for free: the author already +chose which interface to write against. The only marker left in serialization is `is_stub`, and it is per-task. + +The bundle cannot carry the distinction either. The Language SDK spec puts both kinds in one `bundle`, takes them through one `register` verb in any mixture, and serves the process +with one `bundle.serve()`. One artifact can hold both, so neither the file nor the bundle tells a `DagImporter` what it is holding. + +### Appendix B — What is compared -Handler declarations from every coordinator resolved in Step 1 are unioned -before comparison, since one Dag's stubs can target several queues. +Handler declarations from every coordinator resolved in Step 1 are unioned before comparison, since one Dag's stubs can target several queues. -- `task_id` sets must match exactly — a missing or extra handler is an error. -- `arg_bindings[*].name` against `params[*].name`, **in order**: both sides bind - positionally. -- `arg_bindings[*].value_schema` against `params[*].value_schema`, compared only - where neither side is null. An unannotated `@task.stub` parameter produces - `null` today, so a strict comparison would make every untyped stub argument a - parse error. +- `task_id` sets must match exactly. A missing or extra handler is an error. +- `arg_bindings[*].name` against `params[*].name`, in order — both sides bind positionally. +- `arg_bindings[*].value_schema` against `params[*].value_schema`, compared only where neither side is null. An unannotated `@task.stub` parameter produces `null` today, so a + strict comparison would make every untyped stub argument a parse error. -Any mismatch is raised as `DagImportError` against the Python file, which is the -definition the author can act on. +Any mismatch is raised as `DagImportError` against the Python file, which is the definition the author can act on. From be52702059250fbec26c9a9e48b558a688e7eeae Mon Sep 17 00:00:00 2001 From: LIU ZHE YOU Date: Mon, 14 Sep 2026 05:35:55 +0000 Subject: [PATCH 7/9] Route Lang-SDK parsing through one parse channel and one validation point MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An earlier draft gave task-handler parsing its own message pair on the theory that the runtime answered the coordinator. It does not: the coordinator forwards bytes and decodes nothing, so the peer is whichever process started the parse. Two reply unions would have meant two decoder configurations and two relay paths for identical traffic. Validation likewise had no home. It needs the parsed Dags and the argument bindings that only exist after serialization, and the only place holding both is the Dag-file parse itself — not an importer, which should not have to know what a coordinator or a queue is. Making a coordinator's importer optional restated in a return type what the artifact source already decides, leaving two answers free to disagree. --- .../lang-sdk/0010-lang-sdk-parse-protocol.md | 137 ++++++++---- .../lang-sdk/0011-native-dag-processing.md | 76 +++++-- .../0012-mixed-language-dag-processing.md | 211 ++++++++++-------- 3 files changed, 264 insertions(+), 160 deletions(-) diff --git a/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md b/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md index f6723b1af8df2..ad4acde2b4907 100644 --- a/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md +++ b/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md @@ -25,62 +25,76 @@ Proposed ## Context -The Dag processor asks a Lang-SDK runtime two different questions. "Which Dags does this artifact define?" is answered by the runtime standing in for the parser process, over the -messages [ADR-0004](0004-dag-parsing.md) already defines. "Which task handlers does this artifact register for a `dag_id` Python already owns?" has no answer in those messages, -because a `TaskHandler` registration carries no Dag ([ADR-0012](0012-mixed-language-dag-processing.md)). +The Dag processor asks a Lang-SDK runtime two different questions. "Which Dags does this artifact define?" is answered over the messages [ADR-0004](0004-dag-parsing.md) already +defines. "Which task handlers does this artifact register for a `dag_id` Python already owns?" has no answer in those messages, because a `TaskHandler` registration carries no Dag +([ADR-0012](0012-mixed-language-dag-processing.md)). -This ADR defines the channel and payload for the second question, and names both parse-side entry points on the coordinator. +This ADR defines the request that carries the second question, the subprocess classes that carry both, and the two parse-side entry points on the coordinator. Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec version `1.0`). ## Decision -### Two conversations, two verbs +### One channel shape, two request types ``` -parse_dag — bridge mode parse_task_handler — query mode - - manager coordinator - │ DagFileParseRequest │ TaskHandlerParseRequest - ▼ ▼ - coordinator (raw byte forward) runtime - ▼ │ TaskHandlerParsingResult - runtime ▼ - │ DagFileParsingResult coordinator - ▼ - manager +parse_dag parse_task_handler + + parent process parent process + │ DagFileParseRequest │ TaskHandlerParseRequest + ▼ (ToDagProcessor) ▼ (ToSDKTaskHandlerProcessor) + coordinator (raw byte forward) coordinator (raw byte forward) + ▼ ▼ + runtime runtime + │ DagFileParsingResult │ TaskHandlerParsingResult + ▼ (ToManager) ▼ (ToManager) + parent process parent process ``` -`parse_dag` forwards bytes and never reads the payload. `parse_task_handler` is itself the peer: it sends one request and decodes one reply. They are two methods, not one -`parse(request)`, because a coordinator can serve handlers without serving native Dag parsing. Appendix A has the longer argument. +Both verbs are byte forwarders: the coordinator spawns the runtime, wires `fd 0` to the comm socket, and never decodes the payload. The process that spawned the parse decodes the +reply. They stay two methods, not one `parse(request)`, because a coordinator can serve handlers without serving native Dag parsing. Appendix A has the longer argument. -### Handler declarations get their own unions +### The reply travels on `ToManager` ``` -ToRuntime = TaskHandlerParseRequest coordinator → runtime -ToCoordinator = TaskHandlerParsingResult runtime → coordinator +_ParseSideResponses = shared tail — same members, same `type` discriminator + ConnectionResult | VariableResult | VariableKeysResult | TaskStatesResult + | PreviousDagRunResult | PreviousTIResult | PrevSuccessfulDagRunResult + | ErrorResponse | OKResponse | XComCountResponse | XComResult + | XComSequenceIndexResult | XComSequenceSliceResult + +ToDagProcessor = DagFileParseRequest | _ParseSideResponses parent → child +ToSDKTaskHandlerProcessor = TaskHandlerParseRequest | _ParseSideResponses parent → child (new) + +ToManager = DagFileParsingResult | TaskHandlerParsingResult child → parent + | GetConnection | GetVariable | … | MaskSecret ``` -Not a field on `DagFileParseRequest`. Union membership is the only registration step in the supervisor schema package, so picking the union is the wire-contract decision. The -existing pairs describe a runtime answering the manager; here the runtime answers the coordinator. +`ToSDKTaskHandlerProcessor` is the only new union; `ToManager` gains one member. The two parent → child unions differ in exactly one member, because the child's questions about +connections, variables and XComs do not depend on which parse it was asked for. + +`ToManager` is named for the process that usually holds the other end, but the role it describes is "whoever spawned this parse". The Dag processor manager fills it for +`DagFileProcessorProcess`; a Dag-parsing child fills it for the two processes below, relaying anything that is not a parsing result up its own `ToManager` channel unchanged. That +relay is only type-safe because both hops speak the same pair, which is the reason not to mint a separate `ToCoordinator`. ### Message shapes ``` class TaskHandlerParseRequest: - file: str # the artifact the coordinator resolved - dag_id: str # scope: handlers bound to this dag_id only + file: str # the artifact resolved for this coordinator + dag_ids: list[str] # every Dag in the parsed file with stub tasks that resolved here + bundle_path: Path + bundle_name: str type: Literal["TaskHandlerParseRequest"] class TaskHandlerParsingResult: fileloc: str - task_handlers: list[TaskHandlerDeclaration] + task_handlers: dict[str, list[TaskHandlerDeclaration]] # dag_id → its declarations import_errors: dict[str, str] | None = None warnings: list | None = None type: Literal["TaskHandlerParsingResult"] class TaskHandlerDeclaration: - dag_id: str task_id: str params: list[TaskHandlerParam] # ordered — arg_bindings are positional @@ -90,12 +104,46 @@ class TaskHandlerParam: required: bool # the handler declares no default ``` +One request carries every `dag_id` that resolved to the same artifact under the same coordinator, so a file whose stubs all target one runtime costs one process. A `dag_id` the +artifact registers nothing for is **omitted** from `task_handlers` rather than returned empty: the key set is not required to match `dag_ids`, because it is the union across +coordinators that has to cover the stubs ([ADR-0012](0012-mixed-language-dag-processing.md)). + `value_schema` reuses the `ArgValueSchema` definition `arg_bindings` already carries ([ADR-0007](0007-taskflow-across-language-boundary.md)), so both sides of a comparison are the same type. Two properties matter to validation: the field is nullable on both sides, and `params` is ordered. Appendix B says what that forces. `task_handlers` is the counterpart to `DagFileParsingResult.serialized_dags`, but fully typed. `serialized_dags` is `list[LazyDeserializedDAG]`, which is an opaque object in the schema snapshot. A handler declaration carries no Dag, so it code-generates and schema-validates in every SDK, and nothing on this path needs a DagSerialization implementation. +### Parse processes + +``` +WatchedSubprocess + └── BaseParsingProcess socket lifecycle · ToManager decoding · Get* handling + │ · log forwarding under dag_processor.* + ├── DagFileProcessorProcess (shipped, now a subclass) + │ │ target = _parse_file_entrypoint + │ │ DagFileParseRequest → DagFileParsingResult + │ │ + │ └── LangSDKDagFileProcessorProcess (new — ADR-0011) + │ target = _parse_lang_sdk_dag_entrypoint + │ └── coordinator.parse_dag() — spawn runtime, forward fd 0 ⇄ comm socket + │ same request and result types as its base class + │ + └── SDKTaskHandlerProcessorProcess (new — ADR-0012) + target = _parse_task_handler_entrypoint + └── coordinator.parse_task_handler() — same forwarding + TaskHandlerParseRequest → TaskHandlerParsingResult +``` + +`BaseParsingProcess` is `DagFileProcessorProcess` minus the Dag-specific request and result: the comm socket, the `ToManager` decode, the `Get*` dispatch, and the +`task.` → `dag_processor.` log-forwarder rename. The subclasses supply the first message they send, the result they collect, and the target the child runs. + +`LangSDKDagFileProcessorProcess` differs from its base in the target callable alone. Everything else — the request, the result, the socket, the logging — is inherited, because a +native Lang-SDK Dag answers the same question a Python file does. + +Answering `Get*` needs a `Client`, which only the manager holds. `BaseParsingProcess` therefore resolves a request one of two ways: directly against `self.client` when the manager +is the parent, or by relaying it up `SUPERVISOR_COMMS` when a Dag-parsing child is. + ### Coordinator interface ``` @@ -123,9 +171,14 @@ Names follow the shipped `execute_task` / `_build_execute_task_command` pair and - The handler channel is typed; the Dag channel is opaque. Every SDK gets generated models for the declaration shape. - Each runtime implements a second request type instead of a new field on the existing one. A `DagRef` and a `TaskHandlerRef` have different shapes, so one request/result pair could not carry both. -- `ToRuntime` / `ToCoordinator` extends the set of unions the supervisor schema package introspects. Regenerating the snapshot is what propagates the bodies into each SDK's - generated models. -- `can_handle_dag_file` still gates native Dag parsing only. Handler queries resolve through the coordinator registry. +- `ToSDKTaskHandlerProcessor` extends the set of unions the supervisor schema package introspects, from four to five. It adds two union members — `TaskHandlerParseRequest` and + `TaskHandlerParsingResult`, the latter carrying `TaskHandlerDeclaration` and `TaskHandlerParam` as nested definitions — because the shared responses are classes the registry + already holds. +- Nothing in the protocol distinguishes a coordinator-backed parse from a Python one. A runtime's `Get*` request is answered by the same handlers that answer a Python parser's, + through however many relay hops lie between it and the manager. +- `DagFileProcessorProcess` becomes a subclass. Its public surface does not move, but the shipped `_handle_request` and socket code shifts to `BaseParsingProcess`. +- Neither verb is reached through ADR-0004's `can_handle_dag_file` scan. `parse_dag` is reached through the importer registered for the artifact's extension + ([ADR-0011](0011-native-dag-processing.md)); `parse_task_handler` through `queue → coordinator` ([ADR-0012](0012-mixed-language-dag-processing.md)). - Terms track Language SDK spec `1.0`. A spec rename of `TaskHandler` lands here too. ## References @@ -136,23 +189,24 @@ Names follow the shipped `execute_task` / `_build_execute_task_command` pair and - [ADR-0011](0011-native-dag-processing.md) — who calls `parse_dag` - [ADR-0012](0012-mixed-language-dag-processing.md) — who calls `parse_task_handler`, and what it compares the reply against - Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) +- `airflow-core/src/airflow/dag_processing/processor.py` — `DagFileProcessorProcess`, `ToManager` / `ToDagProcessor` - `task-sdk/src/airflow/sdk/execution_time/schema/` — supervisor schema version bundle, union registry, generated snapshot - [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs ## Appendix -### Appendix A — Why two verbs and a separate channel +### Appendix A — Why two verbs, and why one reply union -The runtime plays two roles, and they differ in who it is talking to. Parsing native Dags, it replaces the parser process and answers the manager, which is exactly what the manager -↔ parser unions describe. Answering a handler query, it is a short-lived child of the Python parser, and the manager is not a peer at all. Reusing the parser unions for both would -make one union mean two different recipients, and a runtime could not tell from the union which role it was in. +The two verbs differ in what they ask for and in which command starts the runtime, not in how they talk. Collapsing them into one `parse(request)` would hide a real deployment +distinction: a coordinator can serve mixed-language handlers with no interest in native Dag parsing, and an absent method states that better than a runtime rejection does. -The same split is why the verbs stay separate. Collapsing them into one `parse(request)` would hide a real deployment distinction: a coordinator can serve mixed-language handlers -with no interest in native Dag parsing, and an absent method states that better than a runtime rejection does. The two also differ in mechanism — one is a byte forwarder that never -decodes the payload, the other decodes a reply it asked for — so the shared code is the subprocess plumbing underneath, not the public method. +The reply direction does not need the same split. An earlier draft gave handler parsing its own `ToRuntime` / `ToCoordinator` pair on the theory that the runtime was answering the +coordinator rather than the manager. It is not: the coordinator forwards bytes in both directions and decodes nothing, so the peer at the far end of the socket is whichever process +spawned the parse. Giving that peer two unions to decode would mean two `CommsDecoder` configurations, two relay paths for the identical `Get*` traffic, and two registry entries for +bodies that never differ. Reusing `ToManager` leaves one reply union with one new member. -A boolean on `DagFileParseRequest` was the alternative. It cannot work: a `DagRef` and a `TaskHandlerRef` are different payloads, not two subsets of one, so the flag would select -between shapes the result type cannot both hold. +A boolean on `DagFileParseRequest` was the other alternative. It cannot work: a `DagRef` and a `TaskHandlerRef` are different payloads, not two subsets of one, so the flag would +select between shapes the result type cannot both hold. ### Appendix B — What the nullable, ordered parameter list forces @@ -163,12 +217,13 @@ and falls back to name-and-arity otherwise. A strict comparison would turn every and both sides bind positionally. A declaration carries no class, method, or source location. [ADR-0006](0006-no-lang-sdk-source-display.md) rules out Lang-SDK source display, and putting it on the wire would -invite a consumer to render it. +invite a consumer to render it. It carries no `dag_id` either — the `task_handlers` key supplies it, so a declaration cannot disagree with the bucket it arrived in. ### Appendix C — Schema registration mechanics `registered_models_by_name()` in `task-sdk/src/airflow/sdk/execution_time/schema/` introspects a fixed set of unions — `ToTask` / `ToSupervisor` and `ToManager` / `ToDagProcessor` -— so adding `ToRuntime` / `ToCoordinator` extends that set. +— so adding `ToSDKTaskHandlerProcessor` extends that set to five. The shared responses appear in two unions now; the registry keys by class name and rejects two *distinct* classes +under one name, so a member reached twice is not a clash. Two prek hooks guard the generated snapshot. Their interaction on a first-introduction body needs checking against the hooks rather than against that package's `AGENTS.md`: the doc says no `VersionChange` is required for a new body, while `check-supervisor-schemas-versions` fails when the snapshot moves and nothing under `versions/` was touched. diff --git a/airflow-core/adr/lang-sdk/0011-native-dag-processing.md b/airflow-core/adr/lang-sdk/0011-native-dag-processing.md index 4bf4b028dcec8..48edf9dbc1e99 100644 --- a/airflow-core/adr/lang-sdk/0011-native-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0011-native-dag-processing.md @@ -38,14 +38,17 @@ coordinator does. This ADR settles where that importer comes from, which coordin ### The coordinator hands out its importer ``` -BaseCoordinator.get_dag_importer() -> AbstractDagImporter | None - default: None — this coordinator contributes no importer +BaseCoordinator.get_dag_importer() -> AbstractDagImporter + abstract — a coordinator names the importer for the artifacts it parses JavaCoordinator.get_dag_importer() -> JavaDagImporter(coordinator=self) ``` The importer comes back already bound to the coordinator, so an operator never configures which coordinator an importer uses. `[sdk] coordinators` stays the one place a runtime is declared. +The return is not optional. A coordinator that never contributes an importer to any bundle already says so by the artifact source it was configured with — `for_bundle` below +answers that, and answering it a second time with a `None` return would let the two disagree. What a coordinator can parse and where it gets registered are separate questions. + ### Registration order ``` @@ -85,28 +88,48 @@ Importers are keyed by file extension, one per extension, so two `JavaCoordinato makes it the mode a deployment running two runtimes of the same language has to use. `EXPLICIT_ROOT` has no DagBundle, so nothing scans its artifacts and it cannot produce a Dag to persist; it serves mixed-language work only. Appendix A covers the three modes in full. -### The result has to be `airflow.sdk.DAG` +### The integration point is `import_definition` + +An importer runs inside the Dag-parsing child, so a native Lang-SDK Dag is parsed by a process the importer itself starts. That process is +`LangSDKDagFileProcessorProcess`, which differs from the one the manager started only in the target callable it runs +([ADR-0010](0010-lang-sdk-parse-protocol.md)). ``` -JavaDagImporter.import_definition(definition, bundle=...) - │ - ├── JavaCoordinator.parse_dag(...) ← bridge mode (ADR-0010) - │ ├── _build_parse_dag_command() → (command, schema_version) - │ └── Spawn JVM, forward fd 0 ⇄ comm socket - │ manager ──DagFileParseRequest───▶ JVM - │ manager ◀─DagFileParsingResult─── JVM - │ ┌────────────────────────────────────────────────────┐ - │ │ serialized_dags: ["java_report"] (@Builder.Dag) │ - │ └────────────────────────────────────────────────────┘ - │ TaskHandler registrations have no Dag to serialize — - │ no "etl" entry exists to be discarded. - │ - ├── LazyDeserializedDAG(data=...) → airflow.sdk.DAG - ├── Return DagImportResult(dags=[DAG("java_report")]) - ▼ -DagModelOperation → PERSIST "java_report" only +DagFileProcessorProcess(analytics.jar) ← manager spawns, as for any file + └── _parse_file_entrypoint → _parse_file + └── BundleDagBag → DagImporterRegistry.get_importer(".jar") → JavaDagImporter + │ + └── JavaDagImporter.import_definition(definition, bundle=...) + │ + ├── LangSDKDagFileProcessorProcess.start( + │ target=_parse_lang_sdk_dag_entrypoint, + │ coordinator=JavaCoordinator(...), path=analytics.jar) + │ │ + │ ├── in the child: _build_parse_dag_command() → (command, schema_version) + │ │ coordinator.parse_dag() — spawn JVM, fd 0 ⇄ comm socket + │ │ + │ │ ──DagFileParseRequest───▶ JVM (ToDagProcessor) + │ │ ◀─DagFileParsingResult─── JVM (ToManager) + │ │ ┌────────────────────────────────────────────────────┐ + │ │ │ serialized_dags: ["java_report"] (@Builder.Dag) │ + │ │ └────────────────────────────────────────────────────┘ + │ │ TaskHandler registrations have no Dag to serialize — + │ │ no "etl" entry exists to be discarded. + │ │ + │ └── Get* from the JVM relayed up to the manager unchanged + │ + ├── LazyDeserializedDAG(data=...) → airflow.sdk.DAG + └── DagImportResult(dags=[DAG("java_report")]) + ▼ + _serialize_dags(bag) → DagFileParsingResult → manager + ▼ + DagModelOperation → PERSIST "java_report" only ``` +The coordinator is reached through the importer, never through the manager's file-to-process routing. ADR-0004's `_resolve_processor_target` scan, which picks a coordinator by +asking each one `can_handle_dag_file`, is superseded here: extension-keyed importer registration has already decided that `.jar` belongs to this coordinator, and two mechanisms +claiming the same file would have to agree. + For comparison, a pure Python file with no stub tasks: ``` @@ -119,19 +142,25 @@ PythonDagImporter.import_definition(definition, bundle=...) DagModelOperation → PERSIST ``` -`DagImportResult.dags` is `list[DAG]`, so the importer wraps each serialized entry as a `LazyDeserializedDAG` and transforms it into an `airflow.sdk.DAG`. `LazyDeserializedDAG` -lives in `airflow-core` today, so where a Task SDK importer reaches it from is settled by AIP-85's own `list[DAG]` / `list[LazyDeserializedDAG]` discussion, not here. +`DagImportResult.dags` is `list[DAG]`, so the importer wraps each serialized entry as a `LazyDeserializedDAG` and transforms it into an `airflow.sdk.DAG`. ## Consequences - A Lang-SDK importer is never configured by hand. The runtime is declared once, in `[sdk] coordinators`, and the importer follows from it. +- Routing a Lang-SDK artifact to its runtime becomes the importer registry's job. ADR-0004's `can_handle_dag_file` / `_resolve_processor_target` scan no longer decides which + process parses a file, and the coordinator method it drove is replaced by `parse_dag` ([ADR-0010](0010-lang-sdk-parse-protocol.md)). - One coordinator instance per DagBundle becomes a deployment constraint: two JDKs mean two `dag_bundle_name` values and two bundle-scoped registries. This is what keeps extension-keyed registration unambiguous. -- A coordinator in `EXPLICIT_ROOT` mode cannot back a Dag importer. Its artifacts live outside any DagBundle, so nothing scans them. +- A coordinator in `EXPLICIT_ROOT` mode cannot back a Dag importer. Its artifacts live outside any DagBundle, so nothing scans them. It still implements `get_dag_importer`; the + importer is simply never asked for, because `for_bundle` does not return that coordinator. - `CoordinatorManager` gains `for_bundle`, a second lookup axis beside `for_queue`. - A packed Go bundle claims the empty extension, which three call sites currently treat as absent rather than as a key. Appendix B lists them. - `get_source_code` is abstract, so every Lang-SDK importer must implement it, and a native Lang-SDK Dag has no Python source to return. What it should return, and how that squares with [ADR-0006](0006-no-lang-sdk-source-display.md), is not settled here. +- A native Dag crosses serialization twice over: the runtime serializes it, the importer deserializes it into an `airflow.sdk.DAG`, and the enclosing parse serializes it again. + That round trip is the price of `DagImportResult.dags` being `list[DAG]` — the same cost AIP-85's `list[DAG]` / `list[LazyDeserializedDAG]` question is about. +- Parsing a native Dag costs two processes, not one: the Dag-processing child the manager already starts, plus the runtime the importer starts inside it. The inner one inherits its + request, result and logging from `DagFileProcessorProcess`, so the only Lang-SDK-specific code is the target callable. ## References @@ -141,6 +170,7 @@ lives in `airflow-core` today, so where a Task SDK importer reaches it from is s - [ADR-0004](0004-dag-parsing.md) — `can_handle_dag_file`, the subprocess bridge - [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display - `task-sdk/src/airflow/sdk/importers/` — `AbstractDagImporter`, `DagImportResult`, `DagImporterRegistry` +- `airflow-core/src/airflow/dag_processing/processor.py` — `DagFileProcessorProcess`, the class `LangSDKDagFileProcessorProcess` extends - [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter - [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs diff --git a/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md index b2625ced61365..c4e9f975bdee9 100644 --- a/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md @@ -33,8 +33,8 @@ A Lang-SDK artifact can serve either of the two authoring features the Language | **Native Dag** | the Lang-SDK source itself | `Dag(spec)` | The entire Dag | In the mixed-language role the artifact used to author a Dag too, under the same `dag_id` the Python file already owns. Two conflicting definitions reached persistence and -something downstream had to choose between them. This ADR removes the conflict at the authoring interface, and defines how the Python importer validates a stub task against the -handler that implements it. +something downstream had to choose between them. This ADR removes the conflict at the authoring interface, and defines where a stub task is validated against the handler that +implements it. Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec version `1.0`). Note that the spec's `bundle` is the SDK-side registration container, not Airflow's `DagBundle` — both appear below. @@ -78,115 +78,132 @@ bundle.serve(args); The split is per registration, not per file and not per bundle. -### Validation is driven by the Python importer +### Validation runs in the Dag-file parse, after the Dags exist + +Comparing a stub against its handler needs two things at once: the `airflow.sdk.DAG` objects the Python file produced, and the `arg_bindings` that only appear once those Dags are +serialized. `_parse_file` holds both, between `_serialize_dags` and the `DagFileParsingResult` it returns. That is where the handler query is issued. ``` -PythonDagImporter.import_definition(definition, bundle=...) - │ - ├── Parse → DAG objects - ├── serialize_dag(dag) → is_stub tasks carry arg_bindings (ADR-0007) - │ - │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 1: Resolve stub tasks → Coordinator instances via │ - │ │ CoordinatorManager.for_queue(queue) │ - │ │ ([sdk] queue_to_coordinator) │ - │ │ │ - │ │ stub task queue Coordinator instance │ - │ │ ────────────────────────────────────────────────── │ - │ │ extract → "jdk-11" → JavaCoordinator(name="jdk-11") │ - │ │ transform → "jdk-17" → JavaCoordinator(name="jdk-17") │ - │ │ load → "jdk-11" → JavaCoordinator(name="jdk-11") │ - │ │ │ - │ │ Deduplicate → 2 distinct coordinator instances: │ - │ │ JavaCoordinator(name="jdk-11") │ - │ │ JavaCoordinator(name="jdk-17") │ - │ └─────────────────────────────────────────────────────────────────────┘ - │ - │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 2: Locate the artifact backing dag_id="etl", no JVM launch — │ - │ │ reuse BundleScanner (ADR-0003), whose build-time inventory │ - │ │ indexes registered (dagId, taskId) handler pairs alongside │ - │ │ the artifact's native Dag ids │ - │ │ │ - │ │ JavaCoordinator(name="jdk-11") │ - │ │ ├── resolve artifact roots for its mode │ - │ │ │ EXPLICIT_ROOT → jars_root │ - │ │ │ NAMED_BUNDLE → dag_bundle_name, pinned │ - │ │ │ TASK_BUNDLE → the bundle being parsed │ - │ │ │ │ - │ │ └── BundleScanner.scanBundles(roots) │ - │ │ → Map │ - │ │ → "etl" → ResolvedBundle(mainClass=..., classpath=...) │ - │ │ │ - │ │ JavaCoordinator(name="jdk-17") │ - │ │ └── (same lookup, its own roots) │ - │ └─────────────────────────────────────────────────────────────────────┘ - │ - │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 3: Query the Coordinator — one request, one response │ - │ │ │ - │ │ JavaCoordinator(name="jdk-11").parse_task_handler( │ - │ │ file=resolved.path, dag_id="etl") │ - │ │ │ │ - │ │ ├── _build_parse_task_handler_command() │ - │ │ │ → (command, subprocess_schema_version) │ - │ │ ├── Spawn JVM, send TaskHandlerParseRequest (ToRuntime) │ - │ │ ├── JVM answers from its own TaskHandler registrations │ - │ │ │ whose dagId is "etl" │ - │ │ ▼ │ - │ │ TaskHandlerParsingResult (ToCoordinator) │ - │ │ → [TaskHandlerDeclaration(task_id, params), ...] │ - │ │ │ - │ │ JavaCoordinator(name="jdk-17") │ - │ │ └── (same call, its own resolved path and handler set) │ - │ └─────────────────────────────────────────────────────────────────────┘ - │ - │ ┌─────────────────────────────────────────────────────────────────────┐ - │ │ Step 4: Compare the declarations against the parsed Python Dag — │ - │ │ union the handlers returned by every coordinator resolved │ - │ │ in Step 1 before comparing task ids │ - │ │ │ - │ │ Python Dag "etl" (stub tasks) TaskHandlerDeclaration │ - │ │ ────────────────────────────── ────────────────────────────── │ - │ │ task_id ↔ task_id (sets must match) │ - │ │ arg_bindings[*].name ↔ params[*].name (in order) │ - │ │ arg_bindings[*].value_schema ↔ params[*].value_schema │ - │ │ compared only where neither side is null │ - │ │ │ - │ │ On mismatch → DagImportError │ - │ └─────────────────────────────────────────────────────────────────────┘ - │ - ├── Validation passed → Return DagImportResult(dags=[dag]) - ▼ -DagModelOperation → PERSIST (Python Dag is the sole DB record) +DagFileProcessorProcess(etl.py) ← manager spawns, as for any file + └── _parse_file_entrypoint → _parse_file + ├── BundleDagBag → PythonDagImporter → airflow.sdk.DAG objects ← the Dags now exist + ├── _serialize_dags(bag) → is_stub tasks carry arg_bindings (ADR-0007) + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 1: Collect the dag_ids to ask about — every Dag in this │ + │ │ file with at least one is_stub task → ["etl"] │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 2: Resolve stub tasks → Coordinator instances via │ + │ │ CoordinatorManager.for_queue(queue) │ + │ │ ([sdk] queue_to_coordinator) │ + │ │ │ + │ │ stub task queue Coordinator instance │ + │ │ ────────────────────────────────────────────────── │ + │ │ extract → "jdk-11" → JavaCoordinator(name="jdk-11") │ + │ │ transform → "jdk-17" → JavaCoordinator(name="jdk-17") │ + │ │ load → "jdk-11" → JavaCoordinator(name="jdk-11") │ + │ │ │ + │ │ Deduplicate → 2 distinct coordinator instances │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 3: Locate the artifact backing each dag_id, no JVM launch — │ + │ │ reuse BundleScanner (ADR-0003), whose build-time inventory │ + │ │ indexes registered (dagId, taskId) handler pairs alongside │ + │ │ the artifact's native Dag ids │ + │ │ │ + │ │ JavaCoordinator(name="jdk-11") │ + │ │ ├── resolve artifact roots for its mode │ + │ │ │ EXPLICIT_ROOT → jars_root │ + │ │ │ NAMED_BUNDLE → dag_bundle_name, pinned │ + │ │ │ TASK_BUNDLE → the bundle being parsed │ + │ │ │ │ + │ │ └── BundleScanner.scanBundles(roots) │ + │ │ → "etl" → ResolvedBundle(analytics.jar, mainClass, ...) │ + │ │ │ + │ │ Group the dag_ids by (coordinator, artifact) — one group, one │ + │ │ process, one request │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 4: Query each group — one request, one response │ + │ │ │ + │ │ SDKTaskHandlerProcessorProcess.start( │ + │ │ target=_parse_task_handler_entrypoint, │ + │ │ coordinator=JavaCoordinator("jdk-11"), │ + │ │ path=analytics.jar) │ + │ │ │ │ + │ │ ├── in the child: _build_parse_task_handler_command() │ + │ │ │ coordinator.parse_task_handler() — spawn JVM │ + │ │ │ │ + │ │ │ ──TaskHandlerParseRequest(file=analytics.jar, │ + │ │ │ dag_ids=["etl"])─────▶ JVM │ + │ │ │ (ToSDKTaskHandlerProcessor) │ + │ │ │ │ + │ │ │ JVM answers from its own TaskHandler registrations │ + │ │ │ whose dagId is one of the requested ids │ + │ │ │ │ + │ │ │ ◀─TaskHandlerParsingResult(task_handlers={ │ + │ │ │ "etl": [extract, transform, load]})─────── JVM │ + │ │ │ (ToManager) │ + │ │ │ │ + │ │ └── Get* from the JVM relayed up to the manager unchanged │ + │ │ │ + │ │ JavaCoordinator(name="jdk-17") │ + │ │ └── its own process, its own resolved artifact and handler set │ + │ └─────────────────────────────────────────────────────────────────────┘ + │ + │ ┌─────────────────────────────────────────────────────────────────────┐ + │ │ Step 5: Compare per dag_id against the Dag just serialized — │ + │ │ union task_handlers[dag_id] across every process first │ + │ │ │ + │ │ Python Dag "etl" (stub tasks) TaskHandlerDeclaration │ + │ │ ────────────────────────────── ────────────────────────────── │ + │ │ task_id ↔ task_id (sets must match) │ + │ │ arg_bindings[*].name ↔ params[*].name (in order) │ + │ │ arg_bindings[*].value_schema ↔ params[*].value_schema │ + │ │ compared only where neither side is null │ + │ │ │ + │ │ On mismatch → import_errors["etl.py"] │ + │ └─────────────────────────────────────────────────────────────────────┘ + ▼ + DagFileParsingResult(serialized_dags=["etl"], import_errors={...}) + ▼ + manager → DagModelOperation → PERSIST (the Python Dag is the sole DB record) ``` -The `PythonDagImporter` owns validation. It resolves each stub's `queue` to a coordinator, calls `parse_task_handler` ([ADR-0010](0010-lang-sdk-parse-protocol.md)) for the same -`dag_id`, and compares the reply against the Dag it just parsed. No `DagImporter` is involved: this is one request/response, and what comes back is not a Dag. +The parse owns validation, not an importer. `PythonDagImporter` returns `airflow.sdk.DAG` objects and knows nothing about coordinators or queues, so `@task.stub` keeps working for +any importer that can produce a Dag carrying stub tasks. `_parse_file` is also the only place where the whole file's Dags are visible at once, which is what lets one request cover +every `dag_id` that resolved to the same artifact ([ADR-0010](0010-lang-sdk-parse-protocol.md)). Resolution goes through the coordinator registry, not the filesystem, so the Python Dag and the Lang-SDK artifact **do not need to be in the same DagBundle**. Nothing here needs an -`airflow.sdk.DAG` round-trip either — validation returns the Dag the Python parser already built. Appendix B states exactly what is compared. +`airflow.sdk.DAG` round-trip either — validation compares against the Dag the Python parser already built. Appendix B states exactly what is compared. ### Decision matrix -| Caller | Coordinator call | What comes back | Action | -|----------------------------------------------|----------------------------------------------|------------------------------------------|--------------------------------------------------| -| `PythonDagImporter` | — (parses the Python file itself) | its own parsed Dag | PERSIST | -| `PythonDagImporter` (via stub → Coordinator) | `parse_task_handler`, scoped to one `dag_id` | `TaskHandlerParsingResult` | VALIDATE only — not a Dag, so nothing to persist | -| `JavaDagImporter` | `parse_dag` | `DagFileParsingResult`, native Dags only | PERSIST | +| Caller | Coordinator call | What comes back | Action | +|--------------------------------------------|------------------------------------------------------|------------------------------------------|--------------------------------------------------| +| `_parse_file` → `PythonDagImporter` | — (the Python file is parsed in process) | its own parsed Dags | PERSIST | +| `_parse_file`, per (coordinator, artifact) | `parse_task_handler`, scoped to that group's dag_ids | `TaskHandlerParsingResult` | VALIDATE only — not a Dag, so nothing to persist | +| `_parse_file` → `JavaDagImporter` | `parse_dag` | `DagFileParsingResult`, native Dags only | PERSIST | There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — and nothing reading a `DagImporter`'s results — ever sees one. ## Consequences -- Python leads, Lang-SDK follows. `PythonDagImporter` persists the Dag and drives validation via `queue → Coordinator → parse_task_handler`. +- Python leads, Lang-SDK follows. The Dag-file parse persists the Dag and drives validation via `queue → Coordinator → parse_task_handler`. +- No importer knows about coordinators. `PythonDagImporter` is unchanged by this ADR; the stub-to-handler comparison sits in `_parse_file`, above every importer. - A mixed-language `dag_id` never appears in Dag processing results. No `Dag` registration exists for a `dag_id` a Python file already owns, so everything downstream sees exactly one record per `dag_id`, with no flag to interpret. -- Stub/implementation mismatches — missing handler, extra handler, parameter name or order, incompatible schema — surface as `DagImportError` at parse time. An unannotated stub - argument is checked by name and position only. +- Stub/implementation mismatches — missing handler, extra handler, parameter name or order, incompatible schema — surface as import errors against the Python file at parse time, + alongside the errors the parse already reports. An unannotated stub argument is checked by name and position only. - The Python Dag and Lang-SDK artifact can live in different DagBundles. -- A single Dag can have stubs targeting different queues, some Java, some Go. Each resolves to its own coordinator instance, and validation unions their declarations before - comparing task ids. +- A single Dag can have stubs targeting different queues, some Java, some Go. Each resolves to its own coordinator instance, and validation unions their declarations per `dag_id` + before comparing task ids. +- Validating a file costs one extra process per (coordinator, artifact) pair its stubs resolve to — one for the common case of a file whose stubs all target a single runtime, and + none at all for a file with no stub tasks. - Mixed-language is Python-primary only. Lang-SDK runtimes cannot define stub operators; a native Dag cannot delegate tasks to Python. - No per-Dag flag, no schema migration, no new `DagModel` column, no REST/UI change. - Terms track Language SDK spec `1.0`. A spec rename of `TaskHandler`, or of the `register` / `serve` verbs, lands here too. @@ -201,6 +218,7 @@ There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — a - Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) - [AIP-108](https://cwiki.apache.org/confluence/x/pY4mGQ) — Language SDKs - [AIP-85](https://cwiki.apache.org/confluence/x/_Q7OEg) — DagImporter +- `airflow-core/src/airflow/dag_processing/processor.py` — `_parse_file`, `_serialize_dags`, `DagFileParsingResult` ## Appendix @@ -216,11 +234,12 @@ with one `bundle.serve()`. One artifact can hold both, so neither the file nor t ### Appendix B — What is compared -Handler declarations from every coordinator resolved in Step 1 are unioned before comparison, since one Dag's stubs can target several queues. +Handler declarations from every process spawned in Step 4 are unioned per `dag_id` before comparison, since one Dag's stubs can target several queues. - `task_id` sets must match exactly. A missing or extra handler is an error. - `arg_bindings[*].name` against `params[*].name`, in order — both sides bind positionally. - `arg_bindings[*].value_schema` against `params[*].value_schema`, compared only where neither side is null. An unannotated `@task.stub` parameter produces `null` today, so a strict comparison would make every untyped stub argument a parse error. -Any mismatch is raised as `DagImportError` against the Python file, which is the definition the author can act on. +Any mismatch is reported against the Python file, which is the definition the author can act on, and travels back on `DagFileParsingResult.import_errors` with everything else the +parse found. From f684ee0f957c79059b43f492b4ae7cda836d3c04 Mon Sep 17 00:00:00 2001 From: LIU ZHE YOU Date: Mon, 14 Sep 2026 05:52:37 +0000 Subject: [PATCH 8/9] Order the Lang-SDK Dag processing ADRs decisions-first A reader meeting native Dag processing and mixed-language handlers for the first time needs to know what each one decides before the message shapes and subprocess classes that carry them mean anything. The parse protocol is the mechanics both decisions share, so it reads better after them than in front of them. --- ...ssing.md => 0010-native-dag-processing.md} | 10 ++++----- ... => 0011-mixed-language-dag-processing.md} | 10 ++++----- ...col.md => 0012-lang-sdk-parse-protocol.md} | 22 +++++++++---------- airflow-core/adr/lang-sdk/README.md | 6 ++--- 4 files changed, 24 insertions(+), 24 deletions(-) rename airflow-core/adr/lang-sdk/{0011-native-dag-processing.md => 0010-native-dag-processing.md} (97%) rename airflow-core/adr/lang-sdk/{0012-mixed-language-dag-processing.md => 0011-mixed-language-dag-processing.md} (98%) rename airflow-core/adr/lang-sdk/{0010-lang-sdk-parse-protocol.md => 0012-lang-sdk-parse-protocol.md} (96%) diff --git a/airflow-core/adr/lang-sdk/0011-native-dag-processing.md b/airflow-core/adr/lang-sdk/0010-native-dag-processing.md similarity index 97% rename from airflow-core/adr/lang-sdk/0011-native-dag-processing.md rename to airflow-core/adr/lang-sdk/0010-native-dag-processing.md index 48edf9dbc1e99..ff2acb006031c 100644 --- a/airflow-core/adr/lang-sdk/0011-native-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0010-native-dag-processing.md @@ -17,7 +17,7 @@ under the License. --> -# ADR-0011: Native Dag Processing — DagImporter Registration and Routing +# ADR-0010: Native Dag Processing — DagImporter Registration and Routing ## Status @@ -92,7 +92,7 @@ persist; it serves mixed-language work only. Appendix A covers the three modes i An importer runs inside the Dag-parsing child, so a native Lang-SDK Dag is parsed by a process the importer itself starts. That process is `LangSDKDagFileProcessorProcess`, which differs from the one the manager started only in the target callable it runs -([ADR-0010](0010-lang-sdk-parse-protocol.md)). +([ADR-0012](0012-lang-sdk-parse-protocol.md)). ``` DagFileProcessorProcess(analytics.jar) ← manager spawns, as for any file @@ -148,7 +148,7 @@ DagModelOperation → PERSIST - A Lang-SDK importer is never configured by hand. The runtime is declared once, in `[sdk] coordinators`, and the importer follows from it. - Routing a Lang-SDK artifact to its runtime becomes the importer registry's job. ADR-0004's `can_handle_dag_file` / `_resolve_processor_target` scan no longer decides which - process parses a file, and the coordinator method it drove is replaced by `parse_dag` ([ADR-0010](0010-lang-sdk-parse-protocol.md)). + process parses a file, and the coordinator method it drove is replaced by `parse_dag` ([ADR-0012](0012-lang-sdk-parse-protocol.md)). - One coordinator instance per DagBundle becomes a deployment constraint: two JDKs mean two `dag_bundle_name` values and two bundle-scoped registries. This is what keeps extension-keyed registration unambiguous. - A coordinator in `EXPLICIT_ROOT` mode cannot back a Dag importer. Its artifacts live outside any DagBundle, so nothing scans them. It still implements `get_dag_importer`; the @@ -164,8 +164,8 @@ DagModelOperation → PERSIST ## References -- [ADR-0010](0010-lang-sdk-parse-protocol.md) — `parse_dag` and the coordinator interface it belongs to -- [ADR-0012](0012-mixed-language-dag-processing.md) — why `TaskHandler` registrations never reach a `DagImporter` +- [ADR-0012](0012-lang-sdk-parse-protocol.md) — `parse_dag` and the coordinator interface it belongs to +- [ADR-0011](0011-mixed-language-dag-processing.md) — why `TaskHandler` registrations never reach a `DagImporter` - [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor`, build-time artifact inventory - [ADR-0004](0004-dag-parsing.md) — `can_handle_dag_file`, the subprocess bridge - [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display diff --git a/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0011-mixed-language-dag-processing.md similarity index 98% rename from airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md rename to airflow-core/adr/lang-sdk/0011-mixed-language-dag-processing.md index c4e9f975bdee9..8cf6ac986a625 100644 --- a/airflow-core/adr/lang-sdk/0012-mixed-language-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0011-mixed-language-dag-processing.md @@ -17,7 +17,7 @@ under the License. --> -# ADR-0012: Mixed-Language Dag Processing — Task Handlers Are Not Dags +# ADR-0011: Mixed-Language Dag Processing — Task Handlers Are Not Dags ## Status @@ -59,7 +59,7 @@ Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec vers ``` A `TaskHandlerRef` has no schedule, no task graph, and no `dag_id` of its own to persist. A `DagRef` has all three. Dag parsing draws on the `Dag` registrations -([ADR-0011](0011-native-dag-processing.md)); validation draws on the `TaskHandler` registrations. Because both kinds share one bundle and one `register` verb, the bundle cannot be +([ADR-0010](0010-native-dag-processing.md)); validation draws on the `TaskHandler` registrations. Because both kinds share one bundle and one `register` verb, the bundle cannot be the discriminator — the registration kind is. Appendix A gives the rejected alternative and why. ### One artifact, both kinds @@ -176,7 +176,7 @@ DagFileProcessorProcess(etl.py) ← manager spawn The parse owns validation, not an importer. `PythonDagImporter` returns `airflow.sdk.DAG` objects and knows nothing about coordinators or queues, so `@task.stub` keeps working for any importer that can produce a Dag carrying stub tasks. `_parse_file` is also the only place where the whole file's Dags are visible at once, which is what lets one request cover -every `dag_id` that resolved to the same artifact ([ADR-0010](0010-lang-sdk-parse-protocol.md)). +every `dag_id` that resolved to the same artifact ([ADR-0012](0012-lang-sdk-parse-protocol.md)). Resolution goes through the coordinator registry, not the filesystem, so the Python Dag and the Lang-SDK artifact **do not need to be in the same DagBundle**. Nothing here needs an `airflow.sdk.DAG` round-trip either — validation compares against the Dag the Python parser already built. Appendix B states exactly what is compared. @@ -210,8 +210,8 @@ There is no fourth row. A `TaskHandlerRef` has no Dag, so no `DagImporter` — a ## References -- [ADR-0010](0010-lang-sdk-parse-protocol.md) — `parse_task_handler` and the `TaskHandlerParsingResult` shape this ADR compares against -- [ADR-0011](0011-native-dag-processing.md) — the `Dag`-registration half, and the importer that persists it +- [ADR-0012](0012-lang-sdk-parse-protocol.md) — `parse_task_handler` and the `TaskHandlerParsingResult` shape this ADR compares against +- [ADR-0010](0010-native-dag-processing.md) — the `Dag`-registration half, and the importer that persists it - [ADR-0003](0003-pure-java-dags.md) — `BundleScanner` / `BuilderProcessor`, build-time artifact inventory - [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display for mixed-language Dags - [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / `TaskArgBinding` / `ArgValueSchema` diff --git a/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md b/airflow-core/adr/lang-sdk/0012-lang-sdk-parse-protocol.md similarity index 96% rename from airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md rename to airflow-core/adr/lang-sdk/0012-lang-sdk-parse-protocol.md index ad4acde2b4907..e144b87445b58 100644 --- a/airflow-core/adr/lang-sdk/0010-lang-sdk-parse-protocol.md +++ b/airflow-core/adr/lang-sdk/0012-lang-sdk-parse-protocol.md @@ -17,7 +17,7 @@ under the License. --> -# ADR-0010: Lang-SDK Parse Protocol — Handler Messages and Coordinator Verbs +# ADR-0012: Lang-SDK Parse Protocol — Handler Messages and Coordinator Verbs ## Status @@ -27,7 +27,7 @@ Proposed The Dag processor asks a Lang-SDK runtime two different questions. "Which Dags does this artifact define?" is answered over the messages [ADR-0004](0004-dag-parsing.md) already defines. "Which task handlers does this artifact register for a `dag_id` Python already owns?" has no answer in those messages, because a `TaskHandler` registration carries no Dag -([ADR-0012](0012-mixed-language-dag-processing.md)). +([ADR-0011](0011-mixed-language-dag-processing.md)). This ADR defines the request that carries the second question, the subprocess classes that carry both, and the two parse-side entry points on the coordinator. @@ -106,7 +106,7 @@ class TaskHandlerParam: One request carries every `dag_id` that resolved to the same artifact under the same coordinator, so a file whose stubs all target one runtime costs one process. A `dag_id` the artifact registers nothing for is **omitted** from `task_handlers` rather than returned empty: the key set is not required to match `dag_ids`, because it is the union across -coordinators that has to cover the stubs ([ADR-0012](0012-mixed-language-dag-processing.md)). +coordinators that has to cover the stubs ([ADR-0011](0011-mixed-language-dag-processing.md)). `value_schema` reuses the `ArgValueSchema` definition `arg_bindings` already carries ([ADR-0007](0007-taskflow-across-language-boundary.md)), so both sides of a comparison are the same type. Two properties matter to validation: the field is nullable on both sides, and `params` is ordered. Appendix B says what that forces. @@ -124,12 +124,12 @@ WatchedSubprocess │ │ target = _parse_file_entrypoint │ │ DagFileParseRequest → DagFileParsingResult │ │ - │ └── LangSDKDagFileProcessorProcess (new — ADR-0011) + │ └── LangSDKDagFileProcessorProcess (new — ADR-0010) │ target = _parse_lang_sdk_dag_entrypoint │ └── coordinator.parse_dag() — spawn runtime, forward fd 0 ⇄ comm socket │ same request and result types as its base class │ - └── SDKTaskHandlerProcessorProcess (new — ADR-0012) + └── SDKTaskHandlerProcessorProcess (new — ADR-0011) target = _parse_task_handler_entrypoint └── coordinator.parse_task_handler() — same forwarding TaskHandlerParseRequest → TaskHandlerParsingResult @@ -149,8 +149,8 @@ is the parent, or by relaying it up `SUPERVISOR_COMMS` when a Dag-parsing child ``` BaseCoordinator execution_time/coordinator.py ├── execute_task (shipped) - ├── parse_dag (new — native Dags, ADR-0011) - └── parse_task_handler (new — handlers, ADR-0012) + ├── parse_dag (new — native Dags, ADR-0010) + └── parse_task_handler (new — handlers, ADR-0011) │ SubprocessCoordinator coordinators/_subprocess.py implements all three; each resolves (command, subprocess_schema_version) @@ -178,7 +178,7 @@ Names follow the shipped `execute_task` / `_build_execute_task_command` pair and through however many relay hops lie between it and the manager. - `DagFileProcessorProcess` becomes a subclass. Its public surface does not move, but the shipped `_handle_request` and socket code shifts to `BaseParsingProcess`. - Neither verb is reached through ADR-0004's `can_handle_dag_file` scan. `parse_dag` is reached through the importer registered for the artifact's extension - ([ADR-0011](0011-native-dag-processing.md)); `parse_task_handler` through `queue → coordinator` ([ADR-0012](0012-mixed-language-dag-processing.md)). + ([ADR-0010](0010-native-dag-processing.md)); `parse_task_handler` through `queue → coordinator` ([ADR-0011](0011-mixed-language-dag-processing.md)). - Terms track Language SDK spec `1.0`. A spec rename of `TaskHandler` lands here too. ## References @@ -186,8 +186,8 @@ Names follow the shipped `execute_task` / `_build_execute_task_command` pair and - [ADR-0004](0004-dag-parsing.md) — coordinator subprocess bridge, `DagFileParseRequest` / `DagFileParsingResult`, `can_handle_dag_file` - [ADR-0006](0006-no-lang-sdk-source-display.md) — no Lang-SDK source display - [ADR-0007](0007-taskflow-across-language-boundary.md) — `arg_bindings` / `TaskArgBinding` / `ArgValueSchema` -- [ADR-0011](0011-native-dag-processing.md) — who calls `parse_dag` -- [ADR-0012](0012-mixed-language-dag-processing.md) — who calls `parse_task_handler`, and what it compares the reply against +- [ADR-0010](0010-native-dag-processing.md) — who calls `parse_dag` +- [ADR-0011](0011-mixed-language-dag-processing.md) — who calls `parse_task_handler`, and what it compares the reply against - Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`) - `airflow-core/src/airflow/dag_processing/processor.py` — `DagFileProcessorProcess`, `ToManager` / `ToDagProcessor` - `task-sdk/src/airflow/sdk/execution_time/schema/` — supervisor schema version bundle, union registry, generated snapshot @@ -229,5 +229,5 @@ Two prek hooks guard the generated snapshot. Their interaction on a first-introd says no `VersionChange` is required for a new body, while `check-supervisor-schemas-versions` fails when the snapshot moves and nothing under `versions/` was touched. Artifact roots are resolved per mode by `_init_root_source`, but published through `_get_scan_roots()`, which is scoped to an active task and raises outside one. Both parse-side -commands need those roots with no `TaskInstance` in hand, so the scope that publishes them has to open for a parse as well as for a task. [ADR-0011](0011-native-dag-processing.md) +commands need those roots with no `TaskInstance` in hand, so the scope that publishes them has to open for a parse as well as for a task. [ADR-0010](0010-native-dag-processing.md) covers the modes themselves. diff --git a/airflow-core/adr/lang-sdk/README.md b/airflow-core/adr/lang-sdk/README.md index a438acc3488bf..ec80aa587326c 100644 --- a/airflow-core/adr/lang-sdk/README.md +++ b/airflow-core/adr/lang-sdk/README.md @@ -35,9 +35,9 @@ bind core interfaces and apply to every language SDK, not just the Java SDK. - [ADR-0007](0007-taskflow-across-language-boundary.md): TaskFlow across the language boundary — argument binding for Lang-SDK tasks. - [ADR-0008](0008-control-flow-constructs.md): control-flow constructs — grouping, conditional skipping, branching, and triggering a Dag run. - [ADR-0009](0009-provider-operators-as-generated-dsl.md): provider operators as generated, serialization-only DSL in every Lang SDK. -- [ADR-0010](0010-lang-sdk-parse-protocol.md): Lang-SDK parse protocol — task handler messages and coordinator verbs. -- [ADR-0011](0011-native-dag-processing.md): native Dag processing — DagImporter registration and routing. -- [ADR-0012](0012-mixed-language-dag-processing.md): mixed-language Dag processing — task handlers are not Dags. +- [ADR-0010](0010-native-dag-processing.md): native Dag processing — DagImporter registration and routing. +- [ADR-0011](0011-mixed-language-dag-processing.md): mixed-language Dag processing — task handlers are not Dags. +- [ADR-0012](0012-lang-sdk-parse-protocol.md): Lang-SDK parse protocol — task handler messages and coordinator verbs. Decisions specific to a single SDK stay next to that SDK — for example, the Go SDK's bundle-format decisions live in [`go-sdk/adr/`](../../../go-sdk/adr). Java-SDK-only interface-design decisions — From dcc256cd327fb40e54cda347d31c688b1c4b3a2a Mon Sep 17 00:00:00 2001 From: ZHE YOU LIU Date: Tue, 29 Sep 2026 07:20:30 +0000 Subject: [PATCH 9/9] Use dag and task as the TaskHandler argument names in ADR-0011 --- .../adr/lang-sdk/0011-mixed-language-dag-processing.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/airflow-core/adr/lang-sdk/0011-mixed-language-dag-processing.md b/airflow-core/adr/lang-sdk/0011-mixed-language-dag-processing.md index 8cf6ac986a625..c2acaadbedf2a 100644 --- a/airflow-core/adr/lang-sdk/0011-mixed-language-dag-processing.md +++ b/airflow-core/adr/lang-sdk/0011-mixed-language-dag-processing.md @@ -27,10 +27,10 @@ Proposed A Lang-SDK artifact can serve either of the two authoring features the Language SDK spec fixes, or both at once: -| Feature | Who owns the graph | Author writes | Artifact contributes | -|---------------------------------|----------------------------|----------------------------------|----------------------| -| **Mixed Language Task Handler** | Python, via `@task.stub` | `TaskHandler(dagId, taskId, fn)` | Only task bodies | -| **Native Dag** | the Lang-SDK source itself | `Dag(spec)` | The entire Dag | +| Feature | Who owns the graph | Author writes | Artifact contributes | +|---------------------------------|----------------------------|------------------------------|----------------------| +| **Mixed Language Task Handler** | Python, via `@task.stub` | `TaskHandler(dag, task, fn)` | Only task bodies | +| **Native Dag** | the Lang-SDK source itself | `Dag(spec)` | The entire Dag | In the mixed-language role the artifact used to author a Dag too, under the same `dag_id` the Python file already owns. Two conflicting definitions reached persistence and something downstream had to choose between them. This ADR removes the conflict at the authoring interface, and defines where a stub task is validated against the handler that @@ -51,7 +51,7 @@ Terms follow the Language SDK spec (`task-sdk/docs/lang-sdk-spec.rst`, spec vers │ │ │ Per-registration (Lang-SDK): which interface the author wrote │ │ Dag(spec) → DagRef │ -│ TaskHandler(dagId, taskId, fn) → TaskHandlerRef │ +│ TaskHandler(dag, task, fn) → TaskHandlerRef │ │ │ │ Both kinds go into one bundle, through one verb: │ │ bundle.register(dag, handler, ...) → bundle.serve() │