sql,adapter: Add RETAIN HISTORY PIN AT for pinned history retention - #38748
Draft
frankmcsherry wants to merge 1 commit into
Draft
sql,adapter: Add RETAIN HISTORY PIN AT for pinned history retention#38748frankmcsherry wants to merge 1 commit into
frankmcsherry wants to merge 1 commit into
Conversation
Add a `PIN AT <time>` form of the `RETAIN HISTORY` option on tables, sources, materialized views, and indexes. A pinned collection keeps its since at or below the given time indefinitely, while otherwise following the default compaction lag: the new `CompactionWindow::PinAt` variant lowers to `ReadPolicy::Multiple([lag, ValidFrom(pin)])`. The pin is a catalog fact: it round-trips through the item's SQL, is reported as strategy `PIN AT` in `mz_history_retention_strategies`, and is exposed to the planner through `CatalogItem::compaction_window` so dependents that read history (CHANGES, correlated AS OF) can check it statically. CREATE MATERIALIZED VIEW and ALTER ... RETAIN HISTORY refuse a pin below the initial as-of or the current read frontier, since the history before that point does not exist. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, PR 1 of 3 (
pin->changes->correlated-asof). Spike quality; opened for discussion.Motivation
RETAIN HISTORY FOR <duration>retains history relative to the write frontier, so a reader that needs a collection's history from a fixed time onward, indefinitely, has nothing to lean on: the since eventually passes any time it might name. Two such readers are in the following PRs,CHANGES(x AS OF t)and correlatedAS OF SYSTEM TIME. Both need "the since ofxnever advances pastt", and both need it to be a durable fact they can check statically, not an in-memory read hold.What this adds
RETAIN HISTORY [=] PIN AT <time>on tables, sources, materialized views and indexes, also viaALTER ... SET (RETAIN HISTORY PIN AT <time>).<time>is a number of milliseconds or a timestamp string, parsed withmz_repr::Timestamp::from_str.CompactionWindow::PinAt(t), lowered toReadPolicy::Multiple([lag_writes_by(default), ValidFrom(t)]): the since follows the default lag until it reachest, then holds.PIN ATinmz_internal.mz_history_retention_strategies, and is exposed to the planner via a newCatalogItem::compaction_window()so dependents can checkpin <= tat planning time.CREATE MATERIALIZED VIEWrefuses a pin below the view's initial as-of, andALTERrefuses a pin below the collection's current read frontier, since that history no longer exists. (ALTERalso refuses to move a pin past a dependent's CHANGES literal, but that check lands in PR 2 where the dependents exist.)Observed on a local instance: after
CREATE MATERIALIZED VIEW mv WITH (RETAIN HISTORY = PIN AT 1788537747100) ...,mz_internal.mz_frontiersreportsmv's read frontier at exactly1788537747100while its input table compacts to about one second behind its upper.Not done here
PIN ATonCREATE TABLE/CREATE SOURCEis not validated against the initial since; MVs andALTERare.enable_logical_compaction_windowgate applies. A dedicated flag (off in production, on in CI) should be added before this leaves draft.Tests
Parser round-trip cases for the new option forms in
src/sql-parser/tests/testdata/create.Release note: none while draft.
🤖 Generated with Claude Code