Repository navigation

Add Coordinator Layer and Java Coordinator - #65958

Merged

uranusjr merged 75 commits into

apache:mainfrom

astronomer:task-sdk/feature/coordinator-interface

May 27, 2026

Merged

Add Coordinator Layer and Java Coordinator#65958
uranusjr merged 75 commits into
apache:mainfrom
astronomer:task-sdk/feature/coordinator-interface

Commits on May 27, 2026
- Add Task Coordinators and Dag File Processor
  
  jason810496
  authored and
  uranusjr
  committed
  5b24379
- Add initial Java provider for Apache Airflow
  
  jason810496
  authored and
  uranusjr
  committed
  3e1f0bc
- Add common selector loop utilities for socket I/O handling for subprocesses
  
  jason810496
  authored and
  uranusjr
  committed
  babc8f1
- Implement Java DAG file processor with TCP communication bridge
  
  jason810496
  authored and
  uranusjr
  committed
  851c99b
- Make JavaDagFileProcessor.can_handle aware of jar file content
  
  jason810496
  authored and
  uranusjr
  committed
  a57e3f9
- Fix java process startup issue
  
  jason810496
  authored and
  uranusjr
  committed
  64f5f3d
- Fix sockets bidning
  
  jason810496
  authored and
  uranusjr
  committed
  67e9bb4
- Refactor Java DAG file processor to use selector-based I/O multiplexing for improved performance and reliability
  
  jason810496
  authored and
  uranusjr
  committed
  909ccc3
- Add BaseLocaleCoordinator for non-Python DAG file processing and task execution
  
  jason810496
  authored and
  uranusjr
  committed
  bc2b4c9
- Implement JavaLocaleCoordinator
  
  jason810496
  authored and
  uranusjr
  committed
  19d1b60
- Add Java task coordinator and entrypoint for locale-specific execution
  
  jason810496
  authored and
  uranusjr
  committed
  0039418
- Refactor Java provider to with generic process coordinators and update related components
  
  jason810496
  authored and
  uranusjr
  committed
  79c1f5a
- Fix Coordinator by getting the correct dag bundle and dag path
  
  jason810496
  authored and
  uranusjr
  committed
  8e31257
- Make @task.stub(language=java) works
  
  jason810496
  authored and
  uranusjr
  committed
  deff331
- Make coordinator respect Jar bundle based on TI workload type
  
  jason810496
  authored and
  uranusjr
  committed
  eb7ba13
- Add get_code_from_file interface for BaseLocaleCoordinator
  
  jason810496
  authored and
  uranusjr
  committed
  8aad6b0
- Fix the 'Pure Java Dag' disappear in metadata DB issue
  
  jason810496
  authored and
  uranusjr
  committed
  487365c
- Refactor process coordinators to runtime coordinators
  
  jason810496
  authored and
  uranusjr
  committed
  b07bb82
- Rename stub operator language field as sdk
  
  jason810496
  authored and
  uranusjr
  committed
  f491fe9
- Rename languages.java provider to sdk.java
  
  jason810496
  authored and
  uranusjr
  committed
  80c56d1
- Add unit tests for socket handling and selector loop functionality
  
  jason810496
  authored and
  uranusjr
  committed
  0843cd3
- Move TaskInstanceDTO to share to make task_runner retrieve TI.queue
  
  jason810496
  authored and
  uranusjr
  committed
  e1df493
- Add [workers/queue_to_runtime_mapping]
  
  jason810496
  authored and
  uranusjr
  committed
  9f0acdf
- Remove the sdk field from stub operator and respect [workers/queue_to_runtime_mapping] config
  
  jason810496
  authored and
  uranusjr
  committed
  de8e5a5
- Rename `[workers] queue_to_runtime_mapping` to `[sdk] queue_to_sdk`
  
  jason810496
  authored and
  uranusjr
  committed
  8705327
- Simplify coordinator-related names (#1569)
  uranusjr
  committed
  b8adf3a
- CI: Add mypy and unit tests for shared/workloads
  
  jason810496
  authored and
  uranusjr
  committed
  9055d00
- CI: Fix DB migration and breeze images
  
  jason810496
  authored and
  uranusjr
  committed
  6f66928
- CI: Fix failing items
  
  jason810496
  authored and
  uranusjr
  committed
  d8d5202
- CI: Fix failing items
  
  jason810496
  authored and
  uranusjr
  committed
  9dd86d3
- CI: Add compat for create_runtime_ti pytest fixture
  
  jason810496
  authored and
  uranusjr
  committed
  cfd5873
- CI: Fix Java provider test to include configuration options
  
  jason810496
  authored and
  uranusjr
  committed
  a2a8244
- CI: Fix self-review nits
  
  jason810496
  authored and
  uranusjr
  committed
  4cd1611
- Revert MappedOperator change
  
  jason810496
  authored and
  uranusjr
  committed
  45a0a73
- CI: Fix failing items
  
  jason810496
  authored and
  uranusjr
  committed
  5e8664e
- CI: Fix Task SDK test_task_runner failures using TaskInstanceDTO
  
  jason810496
  authored and
  uranusjr
  committed
  31742c8
- CI: Skip non-JAR paths in JavaCoordinator.can_handle_dag_file
  
  jason810496
  authored and
  uranusjr
  committed
  b20641e
- CI: Drop literal Example: line from queue_to_sdk config description
  
  jason810496
  authored and
  uranusjr
  committed
  9fade1b
- CI: Skip sdk-java provider in compat tests for older Airflow
  
  jason810496
  authored and
  uranusjr
  committed
  2cf30ea
- CI: Fix MyPy Liskov violation in JavaCoordinator.task_execution_cmd
  
  jason810496
  authored and
  uranusjr
  committed
  f98bc1d
- CI: Fix sdk-java docs build warnings
  
  jason810496
  authored and
  uranusjr
  committed
  372bc9c
- CI: Update SDK Java configuration and documentation references
  
  jason810496
  authored and
  uranusjr
  committed
  570ac1b
- CI: Update map_index handling and add fixture to restore process context in tests
  
  jason810496
  authored and
  uranusjr
  committed
  91b2291
- CI: Refactor map_index handling and update time travel decorators for timezone awareness in tests
  
  jason810496
  authored and
  uranusjr
  committed
  9d05bed
- CI: Replace TaskInstance with TaskInstanceDTO and add additional parameters in test_supervisor
  
  jason810496
  authored and
  uranusjr
  committed
  9a32fbf
- CI: Update test cases to use -1 for map_index and add additional parameters in TestCommsDecoder
  
  jason810496
  authored and
  uranusjr
  committed
  894ac55
- Remove shared workloads dependency and refactor TaskInstanceDTO usage
  
  jason810496
  authored and
  uranusjr
  committed
  93725b9
- Move sdk.java out of provider as apache-airflow-coordinators-java distribution
  
  jason810496
  authored and
  uranusjr
  committed
  664baa2
- Enhance documentation for BaseCoordinator lifecycle methods and IPC mechanisms
  
  jason810496
  authored and
  uranusjr
  committed
  14c7dc2
- CI: Fix docs spellcheck and code-block indent for sdk coordinators example
  
  jason810496
  authored and
  uranusjr
  committed
  ce4f9f4
- Remove pure dag parsing from coordinator (#1578)
  uranusjr
  committed
  d9ba5af
- Move coordinator implementation back into SDK (#1580)
  uranusjr
  committed
  8de8ff8
- CI: Fix uv.lock by removing coordinator distribution (#1585)
  
  jason810496
  authored and
  uranusjr
  committed
  43581c9
- Fix import ignore test
  uranusjr
  committed
  92af8ad
- Rewrite Java coordinator
  uranusjr
  committed
  d124f84
- Accept [sdk] coordinators as dict and lazy init
  uranusjr
  committed
  f8b3882
- Redo JAR lookup
  uranusjr
  committed
  24932d0
- Drop selector_loop module (#1587)
  
  jason810496
  authored and
  uranusjr
  committed
  32cedb5
- Add schema migration to supervisor-child comm
  uranusjr
  committed
  e4ccc01
- Redo 16332b2e3be427e614a1ddc7c7bb610f4d284b22
  uranusjr
  committed
  7b1593f
- CI: Fix uv lock
  
  jason810496
  authored and
  uranusjr
  committed
  08e1ad8
- Add log to identify queue-coordinator lookup
  uranusjr
  committed
  522985f
- Track Popen object instead of psutil.Process
  uranusjr
  committed
  7d7fa95
- Handle tilde (user home) in coordinator kwargs
  uranusjr
  committed
  142ad46
- Improve reporting on invalid coordinator configs
  uranusjr
  committed
  fb79d63
- Handle JAR inspection errors
  uranusjr
  committed
  ef5a836
- Improve JAR metadata inspection
  uranusjr
  committed
  d2f8993
- Cleanup for errors during Java subprocess init
  uranusjr
  committed
  54335bf
- Drain child stdout/err while accepting conns
  uranusjr
  committed
  78b0d7a
- Close stdout/err write sockets in supervisor
  uranusjr
  committed
  b11ee9f
- Find JAR metadata separately
  uranusjr
  committed
  ae8c492
- Correctly indent config example block
  uranusjr
  committed
  44c8a33
- fixup! Move ExecutorCallback execution into a supervised process (#62645)
  uranusjr
  committed
  7d57673
- Expose startup timeout in coordinator kwargs
  uranusjr
  committed
  6ab32d0
- Fix Supervisor Schema output
  uranusjr
  committed
  110c34c

Commits

Commits on May 27, 2026