Repository navigation
Improve BaseCoordinator interface to support pluggable communication channels #66838
Description
Summary
The current BaseCoordinator interface is tightly coupled to subprocess-based execution and socket communication (TCP or Unix Domain). As the coordinator pattern expands to support additional SDK implementations, the base interface should be abstract enough to accommodate alternative transport mechanisms without requiring subclasses to override low-level internals.
Problem
BaseCoordinatorexposes implementation details specific to subprocess + socket IPC- Coordinators that could benefit from other communication channels (e.g. shared memory, in-process execution) have no clean abstraction to build against
- The current design makes it harder to reason about what is a required contract vs. what is a Java-specific detail
Proposed Changes
- Raise the abstraction level of
BaseCoordinator's public interface so that communication channel and process launching are not assumed - The interface should be defined in terms of high-level operations (e.g. accepting a
DagFileParsingRequestand returning aDagFileParsingResponse) rather than prescribing how those operations are carried out underneath - Concrete implementations such as
JavaCoordinatorremain free to use subprocesses and sockets internally
Acceptance Criteria
BaseCoordinatorpublic interface makes no assumptions about transport (sockets, shared memory, etc.) or process model (subprocess vs. in-process)- Existing
JavaCoordinatorand Go SDK coordinator behavior is preserved - A new coordinator implementation using a non-socket channel could be built against
BaseCoordinatorwithout workarounds
Drafted-by: Claude Code (Opus 4.7); reviewed by @jason810496 before posting
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
anmolxlight commentedon May 14, 2026
Opened a targeted PR against the coordinator feature branch: https://github.com/astronomer/airflow/pull/1579
I intentionally did not open this directly against
apache/airflow:mainbecause the coordinator layer currently lives in #65958; targetingmainwould include that whole draft branch. This PR is a one-commit delta on top ofastronomer:task-sdk/feature/coordinator-interfaceand can update #65958 cleanly.uranusjr commentedon May 20, 2026
I added a very coarse interface to allow JavaCoordinator to rewire the entire execution. We probably still should do some refactoring to make JavaActivitySubprocess more generic to work with Go, but this is not needed for now.
jason810496 commentedon May 20, 2026
Yes, this issue will track the common coordinator interface. (e.g. Socket/ IPC/ InProcess Coordinator, some better naming).
From he TODO comment here: https://github.com/apache/airflow/pull/67161/changes#diff-f476aa1363e4eb3c52afc7d320b35ba9ff0f36e51c4f411f1937333e0144e6a5R49-R54
On hold this issue until #67161 (initial PR to introduce Executable Coordinator) get merged.