Skip to content

sql,adapter: Add RETAIN HISTORY PIN AT for pinned history retention - #38748

Draft
frankmcsherry wants to merge 1 commit into
MaterializeInc:mainfrom
frankmcsherry:pin
Draft

sql,adapter: Add RETAIN HISTORY PIN AT for pinned history retention#38748
frankmcsherry wants to merge 1 commit into
MaterializeInc:mainfrom
frankmcsherry:pin

Conversation

@frankmcsherry

Copy link
Copy Markdown
Contributor

Draft, PR 1 of 3 (pin -> changes -> correlated-asof). Spike quality; opened for discussion.

Motivation

RETAIN HISTORY FOR <duration> retains history relative to the write frontier, so a reader that needs a collection's history from a fixed time onward, indefinitely, has nothing to lean on: the since eventually passes any time it might name. Two such readers are in the following PRs, CHANGES(x AS OF t) and correlated AS OF SYSTEM TIME. Both need "the since of x never advances past t", and both need it to be a durable fact they can check statically, not an in-memory read hold.

What this adds

  • RETAIN HISTORY [=] PIN AT <time> on tables, sources, materialized views and indexes, also via ALTER ... SET (RETAIN HISTORY PIN AT <time>). <time> is a number of milliseconds or a timestamp string, parsed with mz_repr::Timestamp::from_str.
  • CompactionWindow::PinAt(t), lowered to ReadPolicy::Multiple([lag_writes_by(default), ValidFrom(t)]): the since follows the default lag until it reaches t, then holds.
  • The pin is a catalog fact. It round-trips through the item's SQL (so it survives restarts), is reported as strategy PIN AT in mz_internal.mz_history_retention_strategies, and is exposed to the planner via a new CatalogItem::compaction_window() so dependents can check pin <= t at planning time.
  • Guards: CREATE MATERIALIZED VIEW refuses a pin below the view's initial as-of, and ALTER refuses a pin below the collection's current read frontier, since that history no longer exists. (ALTER also refuses to move a pin past a dependent's CHANGES literal, but that check lands in PR 2 where the dependents exist.)

Observed on a local instance: after CREATE MATERIALIZED VIEW mv WITH (RETAIN HISTORY = PIN AT 1788537747100) ..., mz_internal.mz_frontiers reports mv's read frontier at exactly 1788537747100 while its input table compacts to about one second behind its upper.

Not done here

  • PIN AT on CREATE TABLE / CREATE SOURCE is not validated against the initial since; MVs and ALTER are.
  • No new feature flag; the existing enable_logical_compaction_window gate applies. A dedicated flag (off in production, on in CI) should be added before this leaves draft.
  • A pinned collection's state costs history-proportional reads (persist consolidates all retained updates on every snapshot read). Measured in PR 2: hydrating an index over a pinned 100k-row view with 1.1M retained updates took 11.5s against 3.5s for an unpinned twin. This is inherent to pinning a shard's since; the mitigation is to pin a second, derived view rather than the one everyone reads.

Tests

Parser round-trip cases for the new option forms in src/sql-parser/tests/testdata/create.

Release note: none while draft.

🤖 Generated with Claude Code

Add a `PIN AT <time>` form of the `RETAIN HISTORY` option on tables,
sources, materialized views, and indexes. A pinned collection keeps its
since at or below the given time indefinitely, while otherwise following
the default compaction lag: the new `CompactionWindow::PinAt` variant
lowers to `ReadPolicy::Multiple([lag, ValidFrom(pin)])`.

The pin is a catalog fact: it round-trips through the item's SQL, is
reported as strategy `PIN AT` in `mz_history_retention_strategies`, and
is exposed to the planner through `CatalogItem::compaction_window` so
dependents that read history (CHANGES, correlated AS OF) can check it
statically. CREATE MATERIALIZED VIEW and ALTER ... RETAIN HISTORY refuse
a pin below the initial as-of or the current read frontier, since the
history before that point does not exist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant