Skip to content

Java SDK: Generate Dag and task configuration from the serialization schema - #73595

Merged
henry3260 merged 2 commits into
apache:mainfrom
jason810496:feature/java-sdk-dag-schema-config
Sep 30, 2026
Merged

henry3260 merged 2 commits into
apache:mainfrom
jason810496:feature/java-sdk-dag-schema-config

Conversation

@jason810496

@jason810496 jason810496 commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Merge order:

  1. Java SDK: Generate Dag and task configuration from the serialization schema #73595 — Generate Dag and task configuration from the serialization schema (current one)
  2. Java SDK: Resolve a task's arguments from the Dag's own wiring #73596 — Resolve a task's arguments from the Dag's own wiring
  3. Java SDK: Declare a Dag's task graph with @Builder.Deps #71189 — Declare a Dag's task graph with @Builder.Deps
  4. Java SDK: Serialize native Dags to DagSerialization v3 #71190 — Serialize native Dags to DagSerialization v3

Why

Add the Dag and task level arguments, and those arguments are generated from the Airflow's own Dag serialization schema.

Example

@Builder.Dag(id = "java_etl", schedule = "@daily", catchup = false, tags = {"example"})
public class EtlPipeline {

  @Builder.Task(id = "extract", retries = 2, retryDelay = "PT5M")
  public long extract() { return 42L; }
}

The same keys on the interface surface:

var dag = new DagDef("java_etl").config("schedule", "@daily");
dag.task("extract", Extract.class).config("retry_delay", Duration.ofMinutes(5));

How

  • :sdk:generateDagDsl emits org.apache.airflow.sdk.Builder — the @Builder.Dag / @Builder.Task / @Builder.TaskHandler annotations — and the key-to-type table that DagDef.config and TaskDef.config validate against.
  • Field selection mirrors the Go SDK's TaskSpec generator: scalar properties only, serializer-owned keys skipped, and a documented exclusion list for Python-only concerns. An exclusion entry that stops matching an eligible key fails generation, so the list cannot go stale.
  • The schema is vendored at sdk/schema/dag-schema.json so a source release builds standalone; an in-repo build refreshes it from airflow-core, and sync-java-sdk-dag-schema prek hook re-vendors it on demand.
  • Annotation attributes are camelCase (retryDelay); config keys are Airflow's own names ("retry_delay").
  • The generated build() constructs the DagDef itself and chains the config calls onto it.
  • Durations and date-times are ISO-8601 strings in annotations, validated at compile time, and java.time values in config calls. An unknown key or a mismatched value type fails the build or Dag parsing.

Was generative AI tooling used to co-author this PR?

A Dag authored entirely in Java has no Python in the picture, so teaching
its edge verbs as "Java's spelling of >> and <<" asks the reader to know a
language the feature does not involve. The mixed-language surface keeps its
Python references, because there the Python Dag file is part of what the
reader has to understand.
@jason810496
jason810496 force-pushed the feature/java-sdk-dag-schema-config branch from c36610b to 6b03721 Compare September 29, 2026 02:41
@jason810496
jason810496 marked this pull request as ready for review September 29, 2026 07:27
@jason810496 jason810496 self-assigned this Sep 29, 2026

@FrankYang0529 FrankYang0529 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. Thanks for the PR.

Comment thread java-sdk/sdk/src/test/kotlin/org/apache/airflow/sdk/DagDefTest.kt
…schema

A Java-authored Dag could carry no schedule, retries, tags or any other
Airflow setting, so it could be defined but not configured. Hand-listing
those settings in Java would drift from the Python semantics they mirror,
so they are recorded and validated against Airflow's own Dag serialization
schema instead, and take effect once the runtime answers parse requests.

The `Builder` annotations are generated from a copy of that schema vendored
at java-sdk/sdk/schema/dag-schema.json, so a source release builds without
the monorepo; a prek hook keeps the copy in sync.
@jason810496
jason810496 force-pushed the feature/java-sdk-dag-schema-config branch from 6b03721 to 8b8bccd Compare September 30, 2026 02:47
@henry3260
henry3260 merged commit 21bbf2f into apache:main Sep 30, 2026
100 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants