Skip to content

feat(datafusion): support timestamp time travel for external engines - #1121

Merged
JingsongLi merged 1 commit into
apache:mainfrom
plusplusjiajia:feat/engine-timestamp-travel
Oct 14, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
plusplusjiajia:feat/engine-timestamp-travel

Conversation

@plusplusjiajia

Copy link
Copy Markdown
Member

Purpose

Support TIMESTAMP AS OF '<yyyy-MM-dd HH:mm:ss>' for registered external table engines through SQLContext and DataFusion SessionContext:

SELECT * FROM lake.db.events TIMESTAMP AS OF '2026-10-01 00:00:00';

The timestamp is parsed as for Paimon tables (local time zone, epoch milliseconds) and passed to the engine, which picks the snapshot.

Follow-up to #1113.

Brief change log

  • Add TableEngineResolver::resolve_table_at_timestamp. The default implementation rejects timestamp time travel, so existing resolvers keep failing closed.
  • Route TIMESTAMP AS OF on external data tables through the same read-only wrapper and session option checks as VERSION AS OF.
  • Keep the existing boundaries: SQLContext resolves without loading the current provider; raw SessionContext preloads it and propagates its errors; branches, metadata tables and FOR SYSTEM_TIME AS OF remain rejected.
  • Update the DataFusion README.

Tests

  • Timestamps at and between engine snapshot commit times reach the resolver as exact epoch milliseconds and return that snapshot's data and schema.
  • SQLContext reads a timestamp snapshot when the current provider cannot be resolved; raw SessionContext propagates the current resolution error.
  • Missing snapshots, backend failures and missing tables are reported without falling back to current data; resolvers without timestamp support and unregistered engines are rejected.
  • Timestamp providers reject writes; session scan options and query-auth are checked before the engine is called.
  • Paimon tables keep Paimon's own timestamp resolution, including when joined with an external timestamp snapshot.

@plusplusjiajia
plusplusjiajia marked this pull request as ready for review October 13, 2026 10:07
@JingsongLi
JingsongLi merged commit 2104a02 into apache:main Oct 14, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants