Skip to content

Java SDK: Group a native Dag's tasks with task groups - #74230

Merged
jason810496 merged 2 commits into
apache:mainfrom
jason810496:feature/java-sdk-task-group
Oct 8, 2026
Merged

jason810496 merged 2 commits into
apache:mainfrom
jason810496:feature/java-sdk-task-group

Conversation

@jason810496

@jason810496 jason810496 commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Merge order:

  1. Java SDK: Group a native Dag's tasks with task groups #74230 — Group a native Dag's tasks with task groups (current one)
  2. Java SDK: Serialize native Dags to DagSerialization v3 #71190 — Serialize native Dags to DagSerialization v3
  3. Java SDK: Pack each native Dag's source file into the bundle JAR #74096 — Pack each native Dag's source file into the bundle JAR

Example

With annotations, a @Builder.TaskGroup class holds the tasks of one group, and nesting one in
another nests the groups:

@Builder.Dag(id = "java_etl", schedule = "@daily")
public class EtlPipeline {

  @Builder.Task
  public long extract() { return 42L; }

  @Builder.TaskGroup                      // the group "Staging", after the class
  static class Staging {
    @Builder.Task
    public long stage(long rows) { return rows; }   // the task "Staging.stage"

    @Builder.TaskGroup(id = "checks")
    static class Checks {
      @Builder.Task
      public void nulls(long staged) { }            // "Staging.checks.nulls"
    }
  }

  @Builder.Deps
  static class Wiring implements EtlPipelineDeps {
    void depends() {
      var rows = extract();
      var staged = staging().stage(rows);
      staging().checks().nulls(staged);
      extract().before(staging());        // the whole group runs after extract
    }
  }
}

The generated view nests the same way, so a group is both the namespace of what it holds and a point
in the flow:

public interface EtlPipelineDeps extends Deps {
  default TaskRef<Long> extract() { ... }
  default Staging staging() { return new Staging() {}; }

  interface Staging extends Group {
    @Override default String groupId() { return "Staging"; }
    default TaskRef<Long> stage(Arg<? extends Long> rows) { ... }
    default Checks checks() { return new Checks() {}; }

    interface Checks extends Group {
      @Override default String groupId() { return "Staging.checks"; }
      default TaskRef<Void> nulls(Arg<? extends Long> staged) { ... }
    }
  }
}

On the interface surface, taskGroup declares a group on the Dag or inside another group:

var staging = dag.taskGroup("staging");
var stage = staging.task("stage", Stage.class);                       // task "staging.stage"
staging.taskGroup("checks").task("nulls", Nulls.class).after(stage);  // "staging.checks.nulls"
extract.before(staging);

How

  • Everything declared in a group carries the group's ID as a prefix, so stage in Staging is the
    task Staging.stage.
  • A group class is static, non-private, and needs a no-argument constructor, because the generated
    task bodies instantiate it. Its ID defaults to the class name, as a Dag's does.
  • Task method names scope to their own group, so two groups can each declare run(). The generated
    task classes are named after the class tree, such as Staging_Checks_Nulls.
  • Support before, after and Flow.of.

Serialization of task groups, and the task-group capability flag, land in #71190, which is where a
native Dag is serialized at all.

prefix_group_id = false and the display options (tooltip, colours, display name) are left for a
follow-up.


Was generative AI tooling used to co-author this PR?

Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Deps.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Deps.kt
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Bundle.kt Outdated
Comment thread java-sdk/processor/src/main/kotlin/org/apache/airflow/sdk/BuilderProcessor.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/internal/Refs.kt Outdated
Comment thread java-sdk/processor/src/main/kotlin/org/apache/airflow/sdk/BuilderProcessor.kt Outdated
@jason810496
jason810496 marked this pull request as draft October 7, 2026 07:24
@jason810496
jason810496 force-pushed the feature/java-sdk-task-group branch from 9412192 to 50b0618 Compare October 7, 2026 07:40
@jason810496
jason810496 force-pushed the feature/java-sdk-task-group branch from 50b0618 to a22621f Compare October 7, 2026 07:44
@jason810496
jason810496 marked this pull request as ready for review October 7, 2026 08:30

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Deps.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Deps.kt
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/DagDef.kt Outdated
Comment thread java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/internal/Refs.kt Outdated
Comment thread java-sdk/processor/src/main/kotlin/org/apache/airflow/sdk/BuilderProcessor.kt Outdated
Comment thread java-sdk/processor/src/main/kotlin/org/apache/airflow/sdk/BuilderProcessor.kt Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst

@henry3260 henry3260 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to help unblock the technical side here, not a formal review.

The reserved names of a group's wiring view are now read from Deps.TaskGroup, so
endpoints, before and after are rejected as task method names inside a group, not
only groupId and nodes. A task named before would otherwise win overload
resolution over the inherited before(Flow...) and silently register a task.

Add tests for that clash, for duplicate group IDs in one scope, and for the order
group edges resolve in: an empty group prefers an earlier edge over its parent,
and edges resolve in the order they were drawn.

Correct the Task groups docs and TaskGroupRef KDoc: only what a group holds is
read once, while edges still resolve in drawing order. Restore the missing
transform, load and audit calls in the group wiring example so it parses.

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Comment thread java-sdk/processor/src/main/kotlin/org/apache/airflow/sdk/BuilderProcessor.kt Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst Outdated
Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst
@jason810496
jason810496 merged commit 8043c70 into apache:main Oct 8, 2026
100 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants