Skip to content

[python] Support window joins in temporal alignment - #9760

Merged
JingsongLi merged 13 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-window-aggregate
Sep 13, 2026
Merged

[python] Support window joins in temporal alignment#9760
JingsongLi merged 13 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-window-aggregate

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

Join higher-frequency samples, such as IMU, to temporal anchor rows and aggregate each matched window.

Changes

  • add top-level and chainable join_window
  • support group-local preceding / following ranges with configurable endpoint closure
  • support named outputs and multiple aggregations per source column
  • support mean, min, max, first, last, and count; the latter three also accept non-numeric values
  • prune right payload reads to aggregation columns
  • reuse snapshot pinning, authorization masking, batched row-ID fetches, and the bounded row-group cache

API choice

rolling().agg() aggregates windows within one input. This operation uses each left timestamp to select rows from a right input, then aggregates the matches, so it follows established window-join terminology. The verb-first join_window also matches the public join_asof API.

Named (source, operation) pairs allow one source column to produce multiple outputs; same-name operation strings remain shorthand.

Tests

150 temporal and multimodal table tests passed on Python 3.11 / PyArrow 19, including regressions for sub-unit timestamp bounds, NumPy floating-point bounds, and preserving mask identity after pruning colliding nested aliases. The masking regressions cover mean after STRING-to-DOUBLE masking, first/last after INT-to-STRING masking, unmatched windows, and rejection of mean/min/max on masked strings. Both integer-boundary regression tests also passed on Python 3.6 / NumPy 1.19.5 / PyArrow 6.0.1. An additional 2,400 boundary cases passed on each of NumPy 1.19.5 and 2.2.6. Python 3.6 syntax and flake8 checks passed.

The masked-alias regressions cannot complete on PyArrow 6 because the existing masking reader requires RecordBatch.set_column, which is unavailable in that version.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add window aggregation to temporal alignment [python] Add temporal window join Sep 12, 2026
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add temporal window join [python] Support window joins in temporal alignment Sep 12, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 13, 2026 01:29

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found three reproducible issues in window-bound handling and masked projection pruning. Verified on Python 3.13.5, PyArrow 19.0.1, and NumPy 2.4.6. The existing temporal and multimodal table tests passed (145 passed, 2 skipped because Ray is unavailable); the additional cases below expose the failures.

Comment thread paimon-python/pypaimon/multimodal/temporal.py
Comment thread paimon-python/pypaimon/multimodal/temporal.py
Comment thread paimon-python/pypaimon/multimodal/temporal.py
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit d5d42d4 into apache:master Sep 13, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants