Skip to content

Discover Dags in sub-folders of packaged (zip) Dag bundles - #71703

Closed
piotrlinski wants to merge 9 commits into
apache:mainfrom
piotrlinski:zip-dag-subfolder-discovery
Closed

piotrlinski wants to merge 9 commits into
apache:mainfrom
piotrlinski:zip-dag-subfolder-discovery

Conversation

@piotrlinski

@piotrlinski piotrlinski commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

Only .py/.pyc files at the root of a zip archive were imported; a Dag file placed in a sub-folder was silently skipped, giving no clue why it never appeared. Zip archives are a supported way to ship a Dag together with its vendored packages, and organising Dags into sub-packages inside the archive is a natural layout that previously did not work.

A member in a sub-folder is now imported when that folder — and every folder above it — is an importable package (contains an __init__.py), which is what zipimport requires to reach it. Parent packages are imported before their sub-modules so autoregistered Dags are attributed to the right module, and stale modules the archive owns are evicted from sys.modules first so members resolve against this archive. A Dag file in a non-package folder cannot be imported from a zip, so instead of being dropped silently it is skipped and a warning names the unreachable file and how to fix it.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 4.8) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Only .py/.pyc files at the root of a zip archive were imported; a Dag file
placed in a sub-folder was silently skipped, giving no clue why it never
appeared. Zip archives are a supported way to ship a Dag together with its
vendored packages, and organising Dags into sub-packages inside the archive
is a natural layout that previously did not work.

A member in a sub-folder is now imported when that folder — and every folder
above it — is an importable package (contains an __init__.py), which is what
zipimport requires to reach it. Parent packages are imported before their
sub-modules so autoregistered Dags are attributed to the right module, and
stale modules the archive owns are evicted from sys.modules first so members
resolve against this archive. A Dag file in a non-package folder cannot be
imported from a zip, so instead of being dropped silently it is skipped and a
warning names the unreachable file and how to fix it.
@kaxil

kaxil commented Oct 1, 2026

Copy link
Copy Markdown
Member

Closing this. #73853 removed airflow-core/src/airflow/dag_processing/importers/python_importer.py, the file this changes; zip parsing now goes through the Task SDK importers. It would also have changed which modules every packaged Dag imports, with no linked issue behind it.

@kaxil kaxil closed this Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants