Skip to content

Parse Dag files through the Task SDK importers - #73853

Merged
uranusjr merged 2 commits into
apache:mainfrom
dilnazanlid:aip-85-parse
Oct 1, 2026
Merged

uranusjr merged 2 commits into
apache:mainfrom
dilnazanlid:aip-85-parse

Conversation

@dilnazanlid

@dilnazanlid dilnazanlid commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor
  • Project Board: AIP-85 Dag Importer (Project #672)
  • 3rd PR for rewiring the Dag processing process with sdk/importers: replace the dag_processing.importers by sdk/importers and move to Dag definitions.

DagBag now discovers and imports Dag definitions through the bundle-scoped Task SDK
importer registry instead of the core airflow.dag_processing.importers
package, which is removed.

  • Importers can be configured globally with [dag_processor] dag_importer_configs, or
    per bundle through an importers key in dag_bundle_config_list.
  • DagBag sets fileloc, relative_fileloc and bundle_name on every imported Dag, so
    importers that build Dags themselves (non-Python formats) are bound to their definition.
  • The source code and language reported by the importer are stored in DagCode. The
    source is read once per file in the Dag processor instead of once per Dag in the manager.
  • Importer-namespaced warnings (e.g. yaml:deprecated_field) become Dag warnings and are
    cleared when no longer reported. Python warnings keep going to captured_warnings.
  • Import errors are cleared per parsed definition, e.g. per zip member.
  • DagFileParsingResult gains parsed_definitions and dag_source_codes, with a
    supervisor schema version change.

Behaviour changes in internal APIs:

  • airflow.dag_processing.importers is removed. It had no configuration or docs, and its
    import_file API differs from the SDK one.
  • DagBag.file_last_changed values are freshness tokens (str) instead of datetime,
    and zip entries are keyed per member.
  • DagBag.has_logged is removed. Files skipped by safe mode are now dropped during
    discovery and logged at debug level.

The Dag processor manager still discovers only .py and zip files. Scheduled parsing of
other file types will be done in the follow-up PR that moves manager discovery to the registry.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Antigravity following the guidelines

@uranusjr

Copy link
Copy Markdown
Member

I think this makes sense. A few nits that I’ll touch later.

Maybe we should mark this as experimental since #73457 is undecided. It looks like we can manage without breaking the interface, but I’d still add the experimental marker just in case for 3.4.

@dilnazanlid

Copy link
Copy Markdown
Contributor Author

Marked [dag_processor] dag_importer_configs and the per-bundle importers key as experimental in config.yml, and added the same |experimental| note to AbstractDagImporter, so the importer interface can still change for 3.5 depending on how #73457 lands. Default importers for .py/.zip parsing is unaffected.

@dilnazanlid
dilnazanlid force-pushed the aip-85-parse branch 2 times, most recently from 340baf4 to 85c69d8 Compare September 30, 2026 15:00
DagBag now discovers and imports Dag definitions through the bundle-scoped
Task SDK importer registry, replacing the core importers package. Importers
can be configured globally with [dag_processor] dag_importer_configs or per
bundle via an importers key in dag_bundle_config_list.

The source code and language reported by the importer are stored in DagCode,
importer-namespaced warnings become Dag warnings, and the parsed definitions
are sent to the manager so import errors are cleared per definition.
DagFileParsingResult gains parsed_definitions and dag_source_codes.
@uranusjr
uranusjr merged commit adaa16f into apache:main Oct 1, 2026
158 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

4 participants