[ColtenOuO]

Summary

A manually triggered Dag run's logical_date can coincide with a later, unrelated scheduled run's logical_date -- they represent different data intervals despite sharing that value (a manual trigger's data interval is derived from when it fires, not the period it's labeled with). The dag_run table's (dag_id, logical_date) uniqueness constraint didn't distinguish between run types, so whichever run landed first silently and permanently claimed that logical_date: the scheduler's dedup check would find the manual run, assume the scheduled run for that period already existed, and skip creating it -- dropping that scheduling period for good, with only an easily-missed WARNING log as a trace.

This isn't only an in-memory dedup-check bug. Verified directly against the metadata database: even bypassing the scheduler's pre-check, inserting the correctly-computed scheduled run fails with the same unique-constraint violation as long as a manual run already occupies that (dag_id, logical_date) pair. The fix has to widen what the database considers a duplicate, not just how the scheduler checks for one.

Change

• Added a migration widening the dag_run unique constraint from (dag_id, logical_date) to (dag_id, logical_date, run_type).w only matches SCHEDULED runs, so a manual (or backfill/asset-triggered) run sharing a logical_date is no longer mistaken for the scheduled run itself.
• SchedulerJobRunner._create_dag_runs's dedup lookup now only matches SCHEDULED runs, so a manual (or backfill/asset-triggered) run sharing a logical_date is no longer mistaken for the scheduled run itself.
• Backfill._create_backfill_dag_run_non_partitioned and _handle_clear_run now lock and update the specific DagRun row by id instead of by (dag_id, logical_date), since that pair can now match more than one row.
• Added test_scheduler_create_dag_runs_manual_run_does_not_block_scheduled_run (test_scheduler_job.py) and test_handle_clear_run_only_updates_the_targeted_run (test_backfill.py) as regression tests; both confirmed to fail without their corresponding fix.

Downgrade handling

After this constraint has been in effect, data can legitimately exist that violates the narrower, pre-migration constraint (a manual and a scheduled run sharing a logical_date), so downgrading straight to the old constraint would fail outright. The downgrade first finds every (dag_id, logical_date) group with more than one row, keeps the lowest-id row of each group in dag_run, and moves the rest into an _airflow_moved__-prefixed table before re-adding the narrower constraint -- data is preserved for review, not discarded. This mirrors the existing 0032_..._rename_execution_date_to_logical_date_and_nullable migration, which handles the equivalent case for NULL logical_date rows the same way.

closes: #41208

---

Was generative AI tooling used to co-author this PR?

• Yes -- Claude Code (Sonnet 5) for writing test

[potiuk]

Hello @ColtenOuO - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 19 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

• [AIP-94] Add airflowctl Tasks List Command #70472 - [AIP-94] Add airflowctl Tasks List Command
• Let HookToolset retry a failed tool call instead of failing the task #71853 - Let HookToolset retry a failed tool call instead of failing the task
• Fix scheduler crash when the API server response is a redirect #71962 - Fix scheduler crash when the API server response is a redirect
• Fix manually triggered runs blocking a scheduled run at the same logical_date #72003 - Fix manually triggered runs blocking a scheduled run at the same logical_date
• Fix mismatched XCom values when a dynamically mapped task has a gap #72007 - Fix mismatched XCom values when a dynamically mapped task has a gap
• Refactor schema version checks to share target-branch and worktree logic #72070 - Refactor schema version checks to share target-branch and worktree logic
• Document custom OTel metrics endpoint path support #72073 - Document custom OTel metrics endpoint path support
• Batch XCom lookups when resolving a mapped task's expand() arguments #72107 - Batch XCom lookups when resolving a mapped task's expand() arguments
• Remove redundant full-Dag TaskInstance query from the execution API task-states endpoint #72270 - Remove redundant full-Dag TaskInstance query from the execution API task-states endpoint
• Fix asset dependency graph dropping outlets for tasks that both consume and produce #72272 - Fix asset dependency graph dropping outlets for tasks that both consume and produce
• Make supervisor split-line stdout test actually cover the split in Task SDK #72585 - Make supervisor split-line stdout test actually cover the split in Task SDK
• Fix GET /assets/{asset_id} returning 500 when asset events share a timestamp #72599 - Fix GET /assets/{asset_id} returning 500 when asset events share a timestamp
• Show detailed API error messages in UI toasters #72831 - Show detailed API error messages in UI toasters

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

• Fix output type loss in AgentOperator's human-in-the-loop review #70132 - Fix output type loss in AgentOperator's human-in-the-loop review
• Decode base64 for bytes-typed HookToolset parameters #70433 - Decode base64 for bytes-typed HookToolset parameters
• Allow providers to ship testing Dags separately from example Dags #70541 - Allow providers to ship testing Dags separately from example Dags
• Reject unknown update_mask fields instead of silently ignoring them #71003 - Reject unknown update_mask fields instead of silently ignoring them
• Speed up clearing downstream tasks on large Dags #71203 - Speed up clearing downstream tasks on large Dags
• Add deadline alert coverage to Dag endpoint tests #72296 - Add deadline alert coverage to Dag endpoint tests

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting