-
Notifications
You must be signed in to change notification settings - Fork 17.5k
Fix create_delta_data_intervals being ignored for timedelta schedules #69869
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sharetheknowledge
wants to merge
11
commits into
apache:main
Choose a base branch
from
sharetheknowledge:fix/create-delta-data-intervals-config
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
4a756fc
Fix create_delta_data_intervals being ignored for timedelta schedules
sharetheknowledge 2d46db8
Merge branch 'main' into fix/create-delta-data-intervals-config
sharetheknowledge e05a092
Add 69869.significant.rst newsfragment for create_delta_data_interval…
sharetheknowledge 18ffcd6
Add create_delta_data_intervals bullet to upgrading_to_airflow3.rst
sharetheknowledge 9667424
Add create_delta_data_intervals to Switching section in timetable.rst
sharetheknowledge 1937beb
Add skip-one-period warning to create_delta_data_intervals in config.yml
sharetheknowledge 410126a
Hoist conf_vars import to module level in test_dag.py
sharetheknowledge 8999d2c
Refactor test to parametrize over (schedule, delta, cron, expected)
sharetheknowledge 61721fe
Fix import ordering in test_dag.py to satisfy ruff isort
sharetheknowledge d52d6a0
Update 'two ways' to 'three ways' in timetable.rst
sharetheknowledge 53feb8e
Use hyphen instead of em dash in newsfragment
sharetheknowledge File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| Fix ``create_delta_data_intervals`` being ignored for timedelta/relativedelta schedules | ||
|
|
||
| ``[scheduler] create_delta_data_intervals`` now correctly controls timetable selection | ||
| for DAGs that pass a ``timedelta`` or ``relativedelta`` to ``schedule=``. Before this | ||
| fix, ``create_cron_data_intervals`` silently governed both cron-string DAGs **and** | ||
| timedelta/relativedelta DAGs, making ``create_delta_data_intervals`` a no-op. | ||
|
|
||
| **Who is affected:** | ||
|
|
||
| Users who set ``create_cron_data_intervals = True`` in their Airflow 3 config to | ||
| preserve Airflow 2 data-interval semantics for cron DAGs (as recommended by | ||
| ``upgrading_to_airflow3.rst``) are also unintentionally running their | ||
| ``timedelta``/``relativedelta`` DAGs on ``DeltaDataIntervalTimetable``. After this | ||
| fix, those DAGs will fall through to the correct default (``DeltaTriggerTimetable``) | ||
| unless ``create_delta_data_intervals`` is also set to ``True``. | ||
|
|
||
| This also affects DAGs migrated from Airflow 2 via ``conversion_v1_to_v2``, which | ||
| rebuilds timetables through the same code path. | ||
|
|
||
| **Behaviour changes:** | ||
|
|
||
| - ``create_delta_data_intervals`` now takes effect as documented. | ||
| - ``create_cron_data_intervals`` now affects **only** cron-string DAGs. | ||
| - The two config keys are independent, as ``config.yml`` always documented. | ||
|
|
||
| **Migration:** | ||
|
|
||
| If you rely on ``DeltaDataIntervalTimetable`` for ``timedelta``/``relativedelta`` | ||
| DAGs (i.e. you need contiguous data intervals and ``ds``/``ts`` anchored at | ||
| ``data_interval_start``), set this **before** the next scheduler restart:: | ||
|
|
||
| [scheduler] | ||
| create_delta_data_intervals = True | ||
|
|
||
| You can safely set this flag on current ``main`` (before this fix ships) because | ||
| ``create_delta_data_intervals`` was previously a no-op - setting it to ``True`` | ||
| will have no effect until the fix is in place, so there is no risk of an | ||
| unexpected timetable switch during the transition. | ||
|
|
||
| Switching a DAG from ``DeltaTriggerTimetable`` to ``DeltaDataIntervalTimetable`` | ||
| (i.e. flipping ``create_delta_data_intervals`` from ``False`` to ``True`` when | ||
| existing DAG runs are present) skips one scheduled run to avoid colliding with | ||
| the previous run's ``logical_date``. Plan for this one-period gap or convert | ||
| affected DAGs to an explicit ``DeltaDataIntervalTimetable(...)`` instance in | ||
| ``schedule=`` before the flag changes. | ||
|
|
||
| * Types of change | ||
|
|
||
| * [ ] DAG changes | ||
| * [x] Config changes | ||
| * [ ] API changes | ||
| * [ ] CLI changes | ||
| * [x] Behaviour changes | ||
| * [ ] Plugin changes | ||
| * [ ] Dependency changes | ||
| * [ ] Code interface changes | ||
|
|
||
| * Migration rules needed | ||
|
|
||
| * Users who set ``create_cron_data_intervals = True`` to preserve Airflow 2 | ||
| data-interval behavior for cron DAGs should also set | ||
| ``create_delta_data_intervals = True`` if they have ``timedelta``/ | ||
| ``relativedelta`` DAGs that depended on ``DeltaDataIntervalTimetable``. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.