Skip to content

TS SDK: add the task state store to the task client - #73618

Merged
jason810496 merged 4 commits into
apache:mainfrom
josix:ts-sdk-task-state-store
Oct 9, 2026
Merged

jason810496 merged 4 commits into
apache:mainfrom
josix:ts-sdk-task-state-store

Conversation

@josix

@josix josix commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

TypeScript task handlers had no access to the per-task-instance state store that Python tasks reach through context["task_state_store"], so a retried TypeScript task could not reconnect to an external job an earlier attempt had submitted.

The supervisor and the generated wire types already carry the four state-store messages. This change exposes them through the TypeScript client, covers them in the example bundle and the ts_sdk e2e mode, and marks task-state-store as supported in the compatibility matrix.

getClient().taskStateStore returns a store scoped to the running task instance, with the same four operations as the Python accessor and the Java and Go stores:

import { getClient, NEVER_EXPIRE } from "apache-airflow-ts-sdk";

export async function runSparkJob() {
  const store = getClient().taskStateStore;

  let jobId = await store.get<string>("job_id");
  if (jobId == null) {
    jobId = await submitSparkJob();
    await store.set("job_id", jobId, { retentionMs: NEVER_EXPIRE });
  }

  const result = await waitForSparkJob(jobId);
  await store.delete("job_id");
  return result;
}
  • get<T>(key) returns the value, or null when the key is missing.
  • set(key, value, { retentionMs? }) stores a value. Omitting retentionMs follows the deployment's [state_store] default_retention_days, and NEVER_EXPIRE stores the key with no expiry.
  • delete(key) removes one key, and resolves when the key does not exist.
  • clear() removes every key for the current task instance.

Related: #73420 (Go SDK), #73464 (Java SDK).


Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code Fable 5.1)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@josix josix changed the title TS SDK: add the task state store to the task client (WIP) TS SDK: add the task state store to the task client Sep 23, 2026

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks! LGTM overall.

Let's also add a TODO before the serialization to respect the state_store/max_value_storage_bytes that will be passed from the coordinator side.

Comment thread ts-sdk/src/sdk/client.ts Outdated
Comment thread ts-sdk/src/sdk/client-types.ts Outdated
Comment thread ts-sdk/src/coordinator/client.ts Outdated
josix added 3 commits October 9, 2026 06:56
TypeScript task handlers had no way to reach the per-task-instance state
store that Python exposes as context["task_state_store"], so a retried
TypeScript task could not reattach to an external job it had already
submitted. The supervisor and the generated wire types already carried
the four state-store messages; only the client surface was missing.

The store is reached through client.taskStateStore, with get, set,
delete and clear, so it reads like the Python accessor and the Java and
Go stores rather than four more flat client methods. It is confined to
the caller's own task instance by the Execution API, so the task
instance id is bound when the coordinator client is built rather than
exposed on TaskContext. Absence handling becomes operation-specific
instead of one shared not-found set, because a fourth lookup makes the
shared set answer null for codes an operation can never receive.

Expiry is resolved on the runtime, as Python does: omitting retentionMs
follows AIRFLOW__STATE_STORE__DEFAULT_RETENTION_DAYS, which the
coordinator passes from [state_store] default_retention_days, and the
write fails when that is absent or malformed rather than retaining the
key for a period nobody configured. NEVER_EXPIRE is a symbol, so a
retention that reaches Infinity by accident is rejected instead of
silently storing a key forever.
The example bundle is what the ts_sdk e2e mode runs, so the new client
methods need handlers there before the capability can be claimed. The
e2e assertions read the store back through the public REST API rather
than the task's own XCom, so they prove the values reached the database
and that NEVER_EXPIRE arrives as a null expiry. Clearing lives in its
own task because clear removes every key of the task instance that
calls it, which would destroy the rows the round-trip test asserts on.
With the client methods in place and exercised end to end, the
compatibility matrix can stop listing task-state-store as unsupported.
The guide spells out the two places TypeScript deliberately differs from
Python so deployment managers are not surprised: the worker-side
state_store_backend is not used, and a config value of 0 means never
expire while retentionMs: 0 expires the key immediately.
@jason810496
jason810496 force-pushed the ts-sdk-task-state-store branch from 4e3f4c7 to 481ec48 Compare October 9, 2026 06:57
@jason810496 jason810496 changed the title (WIP) TS SDK: add the task state store to the task client TS SDK: add the task state store to the task client Oct 9, 2026
@jason810496
jason810496 marked this pull request as ready for review October 9, 2026 07:03

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just fixed the review comments myself.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants