TypeScript SDK: native TypeScript Dag declaration Design

We should wait until AIP-85 Dag Importers pass.

Background

The TypeScript SDK intentionally ships in "Python-stub-Dag mode": TypeScript handlers are registered for Dag/task IDs declared in Python. Higher-level TypeScript Dag authoring was deferred until coordinator-mode execution is reviewed.

What needs to happen

Allow declaring Dags natively in TypeScript.
Revisit bundle ergonomics alongside the authoring interface.

Acceptance criteria

A Dag can be authored end to end in TypeScript with no Python stub file.

Context

Add TypeScript task SDK public interface #67908 (PR description)
Add TypeScript coordinator runtime #69079 (PR description)

@guan404ming

Hi @guan404ming, @shivaam,

I think we can start discussing the user interface for the native TypeScript Dag definition here, if you'd like to join. Otherwise, I'll start the design on my side. (No need to start implementing at this point, just a discussion about what exactly the interface will look like.)

The high-level idea of how the native Dag works for each Lang-SDK is:

We first define a DSL for the Dag and Task (we should be able to set, for example, schedule="@daily" at the Dag level, retries=3 at the Task level, etc.).
Then the Lang-SDK runtime should be able to return an airflow-core/src/airflow/serialization/schema.json-compatible Dag JSON back to the coordinator via the same comm protocol.

Some points we need to be aware of:

From a language perspective, whether the syntax itself can feasibly return a serialized Dag JSON.
Whether the syntax is "TaskFlow"-like enough. By TaskFlow, I mean users can define dependencies through "chaining," without needing something like a "taskId as a string/array" reference on the other task.
How arguments get resolved in the TS-SDK runtime across tasks in the same language, and for the same task across different languages.

For reference, here are the Go-SDK counterparts:

Support TaskFlow Dag definitions natively in the Go-SDK #70158 -- Go-SDK native Dag syntax (no need to dive into the implementation; the syntax in the PR description is enough).
Support TaskFlow call syntax on stub tasks for the Lang SDK #69757 -- the foundation for resolving arguments across native Dags and across different languages: we serialize the argument type annotation as a JSON schema and store it in the serialized Dag.

Either way, let's focus only on the user interface design for now.
Please let me know WDYT, thanks.

Here is one possible TypeScript shape for discussion. The goal is to preserve Python TaskFlow's dependency semantics while keeping the API natural for TypeScript; none of these names are final.

The Dag could be declared first, with tasks defined on it:

const pipeline = new Dag({
  dagId: "example_pipeline",
  schedule: "@daily",
  catchup: false,
});

const extract = pipeline.task({
  taskId: "extract",
  retries: 3,
  run: extractHandler,
});

const transform = pipeline.task({
  taskId: "transform",
  run: transformHandler,
});

const load = pipeline.task({
  taskId: "load",
  run: loadHandler,
});

pipeline.task(...) could use the existing registerTask() mechanism internally while also recording the information needed to serialize the Dag.

Passing task outputs would create data dependencies:

const extracted = extract();
const transformed = transform({ extracted });
const loaded = load({ transformed });

This constructs:

extract -> transform -> load

For ordering without passing data, we could provide both before() and
after():

extract.before(validate, transform); 
publish.after(report, approval);

TypeScript cannot reproduce Python's >> and << because JavaScript uses those operators for numeric bit shifting. I also find before() and after() easier to read than setUpstream() and setDownstream(). The Python-style
methods could potentially be added later as aliases. This is just a proposal right now.

For dependency helpers, I think we can preserve the useful behavior of Python's chain().

A linear chain:

chain(first, second, third, fourth);

would create:

first -> second -> third -> fourth

We could also support Python's pairwise behavior for equally sized arrays:

chain(
  first,
  [second, third],
  [fourth, fifth],
  last,
);

This would create:

first -> second -> fourth -> last
first -> third  -> fifth  -> last

The initial API would not need a separate cross-product helper because the same graph can be expressed with before():

first.before(third, fourth);
second.before(third, fourth);

If this becomes repetitive in real Dags, crossDownstream() could be added later as a convenience:

crossDownstream(
  [first, second],
  [third, fourth],
);

I'd also like to remove registerTask and introduce registerDags(d1, d2) as the entrypoint.

Before removing registerTask, I think we should clarify one compatibility question: does native TypeScript Dag authoring replace the existing Python @task.stub Dag with TypeScript handlers, or should the SDK continue supporting both modes?

If both remain supported, we could keep registerTask() as a compatibility API and use registerDags() only for complete TypeScript-authored Dags.

Alternatively, we could provide a new API:

registerHandlers("sales_pipeline", {
  extract: extractFn,
  transform: transformFn,
});

I would avoid making registerDags() accept a special pseudo-Dag for a Python-defined Dag. TypeScript does not own that Dag’s schedule or dependency graph, so treating its handlers as a Dag could make the source of truth unclear.

Thanks, both proposals make sense to me. For naming, I agree with keeping task() lowercase because it is a factory method rather than a class or constructor.

I also like separating generated Airflow options under spec, but I would keep taskId explicit and top-level. I find this more self-documenting than a positional "extract", which could represent an ID, display name, or task type.

const extract = pipeline.task({
  taskId: "extract",
  run: extractFn,
  spec: {
    retries: 3,
  },
});

Do you have any thoughts on the equivalent for the TS world, where a user could explicitly reference the source argument name?

For the Python @task.stub → TypeScript-handler direction, I think the name already carried by each arg_binding can become the property name in a single input object.

For example:

@task.stub(queue="typescript")
def transform(region_code: str, threshold: float): ...

transform(
    region_code=extract(),
    threshold=0.75,
)

Assuming extract() pushes "us-east-1", the TypeScript runtime would resolve the XCom and literal bindings, then invoke the handler as:

async ({
  input: {
    region_code: region,
    threshold,
  },
  ctx,
  client,
}) => {
  // region === "us-east-1"
  // threshold === 0.75
}

Here, region_code is the source argument name and region is only a local TypeScript alias. Keeping task arguments under input also avoids collisions with SDK-provided fields such as ctx and client (ctx.signal remains available for cancellation).

The runtime could validate each resolved value against its value_schema when provided before constructing input.

I'd also like to remove registerTask and introduce registerDags(d1, d2) as the entrypoint.

Before removing registerTask, I think we should clarify one compatibility question: does native TypeScript Dag authoring replace the existing Python @task.stub Dag with TypeScript handlers, or should the SDK continue supporting both modes?

The existing mixed-language Dag should remain functional - the native TypeScript Dag is a new, additional feature. On top of that, the native TypeScript Dag will respect Dag-level and Task-level arguments, as well as task dependencies, defined on the native Dag side.

If both remain supported, we could keep registerTask() as a compatibility API and use registerDags() only for complete TypeScript-authored Dags.

The TS SDK isn't released yet, so we're free to introduce any kind of breaking change at this point. I still lean toward removing the registerTask interface. The reason being that I'd like to support just one Dag authoring interface. This is also exactly why I challenged the registerTask design back in #67908 (comment). It's fine to have registerTask in the early stage, but I'd like to collapse the interface into the dag_instance = new Dag(), t1 = dag_instance.task() style. Whether the TypeScript Dag serves a "mixed" or "native" purpose, both can be authored in the exact style we're discussing here.

I would avoid making registerDags() accept a special pseudo-Dag for a Python-defined Dag.

This won't happen.

TypeScript does not own that Dag's schedule or dependency graph, so treating its handlers as a Dag could make the source of truth unclear.

The "native Dag" here will own the Dag's schedule and dependency graph. It just focuses on the basics of an SDK — users can define Dag- and Task-level arguments, define each task callable, render properly on the Airflow UI, and trigger it as a normal Dag — except the whole Dag is defined in TypeScript.

Okay, I think it makes sense to keep a single authoring interface and remove registerTask().

I want to clarify how the existing Python-Dag/TypeScript-handler mode would work. In that mode, Python defines the complete Dag, while TypeScript only provides the implementation for one or more tasks.

If the TypeScript user creates a new Dag() with the same dagId and adds those handlers, it would only be a partial representation. Python would still own the schedule, dependencies, and remaining tasks.

Normally, registerDags() would tell the coordinator to serialize the TypeScript Dag as a complete Dag. Doing that here could create a second, incomplete definition with the same dagId.

How should the coordinator know that this TypeScript object is only registering handlers and should not be serialized? Would Dag need an explicit handler-only/external mode, or would this association happen through another mechanism?

Something like:

new Dag({
  dagId: "sales_pipeline",
  mode: "handlerOnly",
});

▎ How should the coordinator know that this TypeScript object is only registering handlers and should not be serialized?

I'd prefer to avoid a mode string on Dag and make the distinction type-level instead:

// native: TS owns schedule + dependencies, gets serialized
const pipeline = new Dag({ dagId: "sales_pipeline", schedule: "@daily" });
pipeline.task("extract", { run: extractFn, spec: { retries: 3 } });

// handler-only: a reference to a Python-owned Dag, never serialized
const external = Dag.external("sales_pipeline");
external.task("transform", { run: transformFn });

The type returned by Dag.external() simply doesn't have schedule/spec/dependency fields, so "TypeScript does not own this Dag" is enforced at compile time rather than ignored at runtime.
The coordinator needs no flag check: Dag instances serialize; external refs only register (dagId, taskId) → handler.
This keeps the single authoring interface — new Dag() remains the only way to author a Dag, and .task() feels identical on both. Handler-only is binding, not authoring.
Compared to a registerHandlers("id", { extract: fn }) map, the .task() form still has room for a per-task args schema to validate the resolved input object.

Yeah, I think both are valid options. Or another direction, having new ExternalDag as interface. Though we will need to check if there's better term than the "external". Or mode: mixed_lang_dag and mode: native_dag.

Thanks both. I agree that a type-level distinction such as Dag.external(...) or ExternalDag is cleaner while preserving the same .task() interface. We can refine the exact name separately.

TypeScript SDK: replace registerTask with Dag and registerDags

Design is finished in #72047.

Navigation Menu

Repository navigation

TypeScript SDK: native TypeScript Dag declaration Design #69288

Background

What needs to happen

Acceptance criteria

Context

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TypeScript SDK: native TypeScript Dag declaration Design #69288

Description

Background

What needs to happen

Acceptance criteria

Context

Activity

jason810496 commented on Jul 24, 2026

shivaam commented on Jul 26, 2026

shivaam commented on Jul 26, 2026

3 remaining items

shivaam commented on Jul 28, 2026

shivaam commented on Jul 28, 2026

shivaam commented on Jul 28, 2026

jason810496 commented on Jul 28, 2026

shivaam commented on Jul 29, 2026

guan404ming commented on Aug 3, 2026

jason810496 commented on Aug 3, 2026

shivaam commented on Aug 4, 2026

jason810496 commented on Sep 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions