Skip to content

[v3-4-test] Run a language SDK runtime in its own process group (#75707) - #75731

Merged
vatsrahul1001 merged 1 commit into
apache:v3-4-testfrom
jason810496:backport-75707-v3-4-test
Oct 27, 2026
Merged

vatsrahul1001 merged 1 commit into
apache:v3-4-testfrom
jason810496:backport-75707-v3-4-test

Conversation

@jason810496

Copy link
Copy Markdown
Member

Why

Stopping a task that a language SDK runs signalled the runtime's pid alone, so a CLI the task had started was reparented to PID 1 and kept running. That child also holds the runtime open, so the runtime itself survived until the SIGKILL escalation killed_task_cleanup_time (60 s by default) later. The issue is reported against 3.4.0b, so the fix is worth having on the release branch.

How

A clean cherry-pick of 750f849 apart from one test-file conflict. TestPopenActivitySubprocessExecutionTimeout is context in the original hunk but does not exist here, since #74869 was reverted from v3-4-test in #75469. Dropping that context is the only change; the resulting diff is otherwise identical to the one on main.

v3-4-test already has new_process_group in the supervisor from #65738, so nothing else needed adapting.


Was generative AI tooling used to co-author this PR?

…he#75707)

Stopping a task that a language SDK runs signalled the runtime's pid alone, so
anything the task had started was reparented to PID 1 and kept running. While
such a child is alive it also holds the runtime open, so the runtime itself
survived until the SIGKILL escalation `killed_task_cleanup_time` later.

The runtime now starts with `process_group=0` and the supervisor is told the
child has its own group, so the existing `os.killpg` path reaches the whole
tree, as it has for Python tasks since apache#65505. A startup that fails signals
the group too, and a runtime that exits before connecting is already reaped,
so its group is reached by the leader's pid.

This covers every way the supervisor stops a task -- a cleared task, `execution_timeout`,
repeated heartbeat failures and task-success overtime -- for the TypeScript, Java and
executable runtimes alike.

(cherry picked from commit 750f849)

closes: apache#75533

Co-Authored-By: Claude <noreply@anthropic.com>
@boring-cyborg boring-cyborg Bot added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk labels Oct 27, 2026
@jason810496 jason810496 added this to the Airflow 3.4.0 milestone Oct 27, 2026
@vatsrahul1001
vatsrahul1001 merged commit e2edfff into apache:v3-4-test Oct 27, 2026
87 checks passed
@jason810496
jason810496 deleted the backport-75707-v3-4-test branch October 27, 2026 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants