Skip to content

Discover Dag definitions through the Task SDK importers - #74020

Merged
uranusjr merged 12 commits into
apache:mainfrom
dilnazanlid:aip-85-definition
Oct 2, 2026
Merged

uranusjr merged 12 commits into
apache:mainfrom
dilnazanlid:aip-85-definition

Conversation

@dilnazanlid

@dilnazanlid dilnazanlid commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

TheDag processor manager still found files with list_py_file_paths, which only returns .py and .zip files. Files handled by a custom importer configured in dag_importer_configs were never queued for parsing. Discovery now uses the importer registry, so anything a configured importer can handle is picked up.

Some importers return definitions nested inside a file, such as members of an archive. The manager still queues and parses the enclosing file once. The member locations are kept on DagFileInfo.definition_locs, and deactivate_deleted_dags and clear_orphaned_import_errors match against them. Before, a separate zip-only pass reopened every archive to collect member paths. The deactivate_deleted_dags(bundle_name, present) signature is unchanged.

Other fixes for nested definitions:

  • If listing a bundle fails, the error is logged and the bundle is left as it was: its known files and Dags are kept, and its version is not advanced, so the next refresh lists it again. Before, the exception escaped the parsing loop and restarted the Dag processor.
  • Stale-Dag detection now matches a Dag nested at any depth in a file that is still present. Before, it only checked one parent level.
  • DagBag.get_dag / process_file now reprocess a nested definition from its own location. Before, the path had to be mapped back to a .zip (correct_maybe_zipped).
  • list_py_file_paths and find_dag_file_paths are removed from airflow.utils.file. They have no remaining callers in this repo.

User-visible changes in discovery, since files are now routed to importers by extension:

  • A sourceless .pyc file (outside __pycache__, with no .py next to it) is now parsed by the Python importer.
  • A corrupt .zip file now shows a zip_read_error import error; it used to be skipped without one.
  • A zip archive without the .zip extension (e.g. .egg) is no longer discovered. Rename it to .zip, or map its extension to the zip importer in dag_importer_configs.

This PR covers #74029's discovery change. One difference for importer authors: here an importer is discovered through its list_dag_definitions, not only through supported_extensions + might_contain_dag.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Antigravity following the guidelines

Comment thread airflow-core/src/airflow/dag_processing/manager.py Outdated
Comment thread airflow-core/src/airflow/dag_processing/manager.py
Comment thread airflow-core/src/airflow/dag_processing/manager.py
Comment thread airflow-core/src/airflow/dag_processing/manager.py
Comment thread airflow-core/src/airflow/dag_processing/manager.py Outdated
Comment thread airflow-core/src/airflow/dag_processing/dagbag.py
Comment thread airflow-core/src/airflow/utils/file.py Outdated
Comment thread airflow-core/src/airflow/dag_processing/manager.py Outdated
Comment thread airflow-core/src/airflow/dag_processing/manager.py
Comment thread airflow-core/src/airflow/dag_processing/manager.py
Comment thread airflow-core/src/airflow/dag_processing/manager.py

@uranusjr uranusjr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor point, otherwise lgtm.

@uranusjr
uranusjr merged commit 4803c0b into apache:main Oct 2, 2026
157 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

4 participants