Skip to content

Warn about oversized task state values from language SDK tasks - #74945

Merged
jason810496 merged 2 commits into
apache:mainfrom
ChickenBenny:go-sdk-state-store-size-warning
Oct 21, 2026
Merged

jason810496 merged 2 commits into
apache:mainfrom
ChickenBenny:go-sdk-state-store-size-warning

Conversation

@ChickenBenny

@ChickenBenny ChickenBenny commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

closes: #74944


A Python task logs a warning in its task log when it writes a task state value larger than [state_store] max_value_storage_bytes, but a task run by a language SDK writes the same value silently. Neither the Go, TypeScript nor Java runtime knows the limit, and they cannot compute the size the limit is measured in, which is len(json.dumps(value)) as Python computes it.

Following review, the check now runs in the supervisor instead of in each SDK. When the supervisor for a language SDK task handles SetTaskStateStore, it measures the value with json.dumps and warns in the task log, then still writes the value, as the Python setter does. Python tasks keep warning in their own setter, so they are not warned twice.

Because the check lives in the coordinator, it covers Go, TypeScript and Java at once, and the TODOs left for it in #73420, #73618 and #73464 are removed.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5.5) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:go-sdk area:task-sdk labels Oct 18, 2026

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix.

Instead of fixing on each SDK side, I think it would be better to fix at the coordinator level so that we don't need to estimate the size of Python JSON dump in each SDK.

Thanks.

@ChickenBenny

ChickenBenny commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks, that is simpler. I moved the check to the supervisor in a840c15: the supervisor for language SDK tasks now warns with len(json.dumps(value)) when it handles SetTaskStateStore, an
d the Go-side size estimation and the max_value_storage_bytes env var are gone. It covers TypeScript and Java too, so I removed the same TODO from all three SDKs.

@ChickenBenny
ChickenBenny force-pushed the go-sdk-state-store-size-warning branch from 794edd7 to 3199497 Compare October 20, 2026 15:50
@ChickenBenny ChickenBenny reopened this Oct 20, 2026
@ChickenBenny ChickenBenny changed the title Go SDK: warn when a task state value exceeds max_value_storage_bytes Warn about oversized task state values from language SDK tasks Oct 20, 2026

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update, LGTM now.

@jason810496 jason810496 added backport-to-v3-4-test Backport to v3-4-test and removed area:go-sdk labels Oct 21, 2026
@jason810496 jason810496 added this to the Airflow 3.4.0 milestone Oct 21, 2026
@jason810496

Copy link
Copy Markdown
Member

The PR is good to merge once the CI passes.

@ChickenBenny

Copy link
Copy Markdown
Contributor Author

The docs build failure is unrelated to this PR. #74623 broke it on main and #75243 fixed it. I'll rebase to pick up the fix.

@ChickenBenny
ChickenBenny force-pushed the go-sdk-state-store-size-warning branch from 054efa4 to fb7490b Compare October 21, 2026 06:50
@jason810496
jason810496 merged commit eccdab5 into apache:main Oct 21, 2026
99 checks passed
@boring-cyborg

boring-cyborg Bot commented Oct 21, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
✅ v3-4-test PR Link

jason810496 pushed a commit that referenced this pull request Oct 22, 2026
…tasks (#74945) (#75299)

* Warn about oversized task state values from language SDK tasks

* Write the oversized task state warning to the task log
(cherry picked from commit eccdab5)

Co-authored-by: ChickenBenny <zxc123benny14159@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk backport-to-v3-4-test Backport to v3-4-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Language SDK task state writes skip the max_value_storage_bytes warning

2 participants