Repository navigation
Warn about oversized task state values from language SDK tasks - #74945
Conversation
jason810496
left a comment
There was a problem hiding this comment.
Thanks for the fix.
Instead of fixing on each SDK side, I think it would be better to fix at the coordinator level so that we don't need to estimate the size of Python JSON dump in each SDK.
Thanks.
|
Thanks, that is simpler. I moved the check to the supervisor in a840c15: the supervisor for language SDK tasks now warns with |
794edd7 to
3199497
Compare
jason810496
left a comment
There was a problem hiding this comment.
Thanks for the update, LGTM now.
|
The PR is good to merge once the CI passes. |
054efa4 to
fb7490b
Compare
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Backport successfully created: v3-4-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
closes: #74944
A Python task logs a warning in its task log when it writes a task state value larger than
[state_store] max_value_storage_bytes, but a task run by a language SDK writes the same value silently. Neither the Go, TypeScript nor Java runtime knows the limit, and they cannot compute the size the limit is measured in, which islen(json.dumps(value))as Python computes it.Following review, the check now runs in the supervisor instead of in each SDK. When the supervisor for a language SDK task handles
SetTaskStateStore, it measures the value withjson.dumpsand warns in the task log, then still writes the value, as the Python setter does. Python tasks keep warning in their own setter, so they are not warned twice.Because the check lives in the coordinator, it covers Go, TypeScript and Java at once, and the TODOs left for it in #73420, #73618 and #73464 are removed.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.