Skip to content

perf: Optimize date_part for timestamp values without timezones - #11187

Merged
Jefffrey merged 4 commits into
apache:mainfrom
neilconway:neilc/perf-timestamp-extract-hour-min
Oct 3, 2026
Merged

Jefffrey merged 4 commits into
apache:mainfrom
neilconway:neilc/perf-timestamp-extract-hour-min

Conversation

@neilconway

@neilconway neilconway commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

  • N/A

Rationale for this change

For timestamp inputs, date_part generally converts the timestamp to a Chrono value and then uses Chrono to extract the requested component. For timestamps without a timezone, hour through nanosecond components can be calculated directly from the stored integer, without determining the calendar date. This is roughly 5-20x faster than going through Chrono on a microbenchmark (results below).

This is partly motivated by improving the performance of date_part and extract in DataFusion. For example, ClickBench Q18 improves by about 2% using this optimization.

Benchmarks: (ARM64, elapsed times in microseconds)

  • timestamp_s/Hour/no_nulls: 33.366 → 5.827, −82.5%
  • timestamp_s/Minute/no_nulls: 36.901 → 7.108, −80.7%
  • timestamp_s/Second/no_nulls: 35.321 → 2.530, −92.8%
  • timestamp_s/Hour/mixed_nulls: 30.885 → 5.821, −81.2%
  • timestamp_s/Year/control: 32.893 → 32.913, +0.1%
  • timestamp_s/Minute/timezone_control: 59.305 → 59.081, −0.4%
  • timestamp_s/Second/timezone_control: 55.782 → 56.263, +0.9%
  • timestamp_ms/Hour/no_nulls: 45.351 → 5.839, −87.1%
  • timestamp_ms/Minute/no_nulls: 49.214 → 6.235, −87.3%
  • timestamp_ms/Second/no_nulls: 47.252 → 6.226, −86.8%
  • timestamp_ms/Millisecond/no_nulls: 44.510 → 2.317, −94.8%
  • timestamp_ms/Microsecond/no_nulls: 44.503 → 2.472, −94.4%
  • timestamp_ms/Nanosecond/no_nulls: 43.116 → 2.375, −94.5%
  • timestamp_ms/Hour/mixed_nulls: 39.190 → 5.859, −85.1%
  • timestamp_ms/Year/control: 44.099 → 44.754, +1.5%
  • timestamp_ms/Minute/timezone_control: 73.426 → 73.751, +0.4%
  • timestamp_ms/Nanosecond/timezone_control: 65.517 → 65.771, +0.4%
  • timestamp_us/Hour/no_nulls: 41.028 → 6.520, −84.1%
  • timestamp_us/Minute/no_nulls: 44.774 → 6.777, −84.9%
  • timestamp_us/Second/no_nulls: 42.947 → 6.238, −85.5%
  • timestamp_us/Millisecond/no_nulls: 40.022 → 2.629, −93.4%
  • timestamp_us/Microsecond/no_nulls: 39.975 → 2.304, −94.2%
  • timestamp_us/Nanosecond/no_nulls: 39.116 → 2.365, −94.0%
  • timestamp_us/Hour/mixed_nulls: 39.438 → 6.486, −83.6%
  • timestamp_us/Year/control: 39.921 → 40.406, +1.2%
  • timestamp_us/Minute/timezone_control: 68.042 → 68.354, +0.5%
  • timestamp_us/Nanosecond/timezone_control: 61.106 → 61.602, +0.8%
  • timestamp_ns/Hour/no_nulls: 40.454 → 3.540, −91.2%
  • timestamp_ns/Minute/no_nulls: 44.499 → 3.388, −92.4%
  • timestamp_ns/Second/no_nulls: 42.425 → 6.241, −85.3%
  • timestamp_ns/Millisecond/no_nulls: 39.546 → 2.628, −93.4%
  • timestamp_ns/Microsecond/no_nulls: 39.388 → 2.627, −93.3%
  • timestamp_ns/Nanosecond/no_nulls: 38.727 → 2.304, −94.1%
  • timestamp_ns/Hour/mixed_nulls: 38.805 → 3.542, −90.9%
  • timestamp_ns/Year/control: 39.567 → 39.710, +0.4%
  • timestamp_ns/Minute/timezone_control: 69.019 → 68.191, −1.2%
  • timestamp_ns/Nanosecond/timezone_control: 61.682 → 61.172, −0.8%

What changes are included in this PR?

  • Optimize date_part as described above
  • Add unit tests
  • Add benchmark

Are these changes tested?

Existing tests pass. Two new tests were added: one checks the consistency of this code path with the results produced by Chrono; the second checks the results of this code path for extreme values (outside of Chrono's supported calendar range).

Are there any user-facing changes?

There is one user-visible behavior change. Chrono has a limited calendar range (roughly +/- 262,000 years); calling date_part on timestamps outside that range returned null. This implementation is defined for the entire range of timestamp values. The returned value is correct, but this behavior is slightly inconsistent with date_part for units that are still implemented via Chrono, and for timestamp values with time zones.

AI usage

Developed with Codex Astra 6, reviewed with Claude Code Fable 5.1. I then reviewed and revised the resulting code.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @neilconway, this is a nice speedup and the tests against Chrono are thorough.

Comment thread arrow-arith/benches/temporal.rs
///
/// Null inputs produce null outputs. A timestamp outside the supported calendar
/// range also produces null, except that for timestamps without a timezone the
/// time parts (`Hour` through `Nanosecond`) are defined for every value.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for calling this out. I'm not sure which behavior is better here, and I'd be interested in what you and the other maintainers think. With this PR, one out-of-range row in a TimestampSecond array returns Hour = 15 and Year = NULL. The same value with a timezone returns NULL for both, and casting it to Time64 returns an error (as_time_res_with_timezone, used by the Timestamp to Time64 cast).

If we'd like to keep the current semantics, one option is to check the range before taking the fast path. For TimestampNanosecond every i64 is in Chrono's range, so it needs no check. For the other units, one branch-free pass over values() against the DateTime::<Utc>::MIN_UTC and MAX_UTC bounds would work, with a fallback to the existing path when a value is out of range. This is the check I tried, at the top of timestamp_time_part:

if !matches!(
    part,
    DatePart::Hour
        | DatePart::Minute
        | DatePart::Second
        | DatePart::Millisecond
        | DatePart::Microsecond
        | DatePart::Nanosecond
) {
    return None;
}
if T::UNIT != TimeUnit::Nanosecond {
    let (min, max) = (DateTime::<Utc>::MIN_UTC, DateTime::<Utc>::MAX_UTC);
    let (lo, hi) = match T::UNIT {
        TimeUnit::Second => (min.timestamp(), max.timestamp()),
        TimeUnit::Millisecond => (min.timestamp_millis(), max.timestamp_millis()),
        _ => (min.timestamp_micros(), max.timestamp_micros()),
    };
    if !array.values().iter().fold(true, |ok, &v| ok & (lo <= v) & (v <= hi)) {
        return None;
    }
}

The early matches! keeps Year and the other calendar parts from paying for the scan. With this in place, test_timestamp_time_parts_extremes would expect NULL for the out-of-range values.

When I tried this with your bench on an M5 Max, it added about 0.7 us per 8192 values. For example, ms Hour went from 5.37 to 6.08 us and ms Millisecond from 2.12 to 2.87 us, against about 43 us on the Chrono path. That seems like a reasonable cost to me, but you may see a better way to do it. If we keep the new behavior instead, the api-change label would make sure it shows up in the changelog.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My two cents: the current behavior implemented by the PR is the better option. The historical behavior of date_part is an unfortunate implementation detail from using Chrono; defining date_part for a wider range is values is not inherently bad (it's good, actually!), and it is also considerably faster.

The main downside is inconsistency between extracting different time components for timestamps with extreme values. That's a bit odd but it doesn't seem like a showstopper to me. The current behavior for out-of-range timestamps is arguably a defect to begin with -- e.g., in the future we could perhaps improve date_part so that it is defined for all timestamps and all time components.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good as long as it's documented for users.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. The current PR has documentation that seems sufficient to me, but lmk if you have suggestions for other places to document this.

@neilconway

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @mbutrovich !

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @neilconway for adding the timezone cases and refreshing the numbers. The bench now matches the description.

Comment thread arrow-arith/benches/temporal.rs Outdated

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @neilconway, LGTM!

@neilconway

Copy link
Copy Markdown
Contributor Author

@alamb Can you take a look, if you get a chance? This has already been approved by Matt.

@Jefffrey
Jefffrey enabled auto-merge October 3, 2026 15:55
@Jefffrey

Jefffrey commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

thanks @neilconway & @mbutrovich

@Jefffrey
Jefffrey added this pull request to the merge queue Oct 3, 2026
Merged via the queue into apache:main with commit 646328f Oct 3, 2026
31 checks passed
efegokdemir pushed a commit to efegokdemir/arrow-rs that referenced this pull request Oct 3, 2026
…che#11199)

# Which issue does this PR close?

- N/A

# Rationale for this change

Casting from timestamp to another datetime type currently goes through
Chrono. When casting timestamp values without a timezone to `Date32`,
`Time32`, or `Time64`, we can avoid going through Chrono and do the cast
directly from the stored integer timestamp value. This is similar to the
approach taken in apache#11187; it improves performance by about 10-20x.

Along the way, adjust casts from timestamp to `Date32`, `Time32` and
`Time64` to correctly respect "safe" mode; previously, such casts
returned an error for out-of-range values.

Benchmarks: (Arm64)

  - timestamp_s/date32/no_nulls: 88.338 → 3.155 µs, −96.4%
  - timestamp_s/date32/mixed_nulls: 73.260 → 3.155 µs, −95.7%
  - timestamp_s/date32/timezone_control: 145.930 → 145.460 µs, −0.3%
  - timestamp_s/time32_s/no_nulls: 66.287 → 2.293 µs, −96.5%
  - timestamp_s/time32_s/mixed_nulls: 54.928 → 2.297 µs, −95.8%
  - timestamp_s/time32_s/timezone_control: 76.130 → 73.408 µs, −3.6%
  - timestamp_s/time32_ms/no_nulls: 68.243 → 2.367 µs, −96.5%
  - timestamp_s/time32_ms/mixed_nulls: 56.229 → 2.369 µs, −95.8%
  - timestamp_s/time32_ms/timezone_control: 79.243 → 77.735 µs, −1.9%
  - timestamp_s/time64_us/no_nulls: 68.461 → 4.648 µs, −93.2%
  - timestamp_s/time64_us/mixed_nulls: 58.283 → 4.657 µs, −92.0%
  - timestamp_s/time64_us/timezone_control: 79.850 → 79.258 µs, −0.7%
  - timestamp_s/time64_ns/no_nulls: 67.501 → 4.510 µs, −93.3%
  - timestamp_s/time64_ns/mixed_nulls: 57.150 → 4.570 µs, −92.0%
  - timestamp_s/time64_ns/timezone_control: 78.252 → 77.134 µs, −1.4%
  - timestamp_ms/date32/no_nulls: 102.210 → 3.176 µs, −96.9%
  - timestamp_ms/date32/mixed_nulls: 84.143 → 3.150 µs, −96.3%
  - timestamp_ms/date32/timezone_control: 165.530 → 165.770 µs, +0.1%
  - timestamp_ms/time32_s/no_nulls: 78.368 → 2.614 µs, −96.7%
  - timestamp_ms/time32_s/mixed_nulls: 63.744 → 2.612 µs, −95.9%
  - timestamp_ms/time32_s/timezone_control: 88.934 → 88.578 µs, −0.4%
  - timestamp_ms/time32_ms/no_nulls: 80.194 → 2.295 µs, −97.1%
  - timestamp_ms/time32_ms/mixed_nulls: 65.715 → 2.289 µs, −96.5%
  - timestamp_ms/time32_ms/timezone_control: 91.717 → 91.280 µs, −0.5%
  - timestamp_ms/time64_us/no_nulls: 80.959 → 4.274 µs, −94.7%
  - timestamp_ms/time64_us/mixed_nulls: 67.123 → 4.290 µs, −93.6%
  - timestamp_ms/time64_us/timezone_control: 93.714 → 93.021 µs, −0.7%
  - timestamp_ms/time64_ns/no_nulls: 79.801 → 4.688 µs, −94.1%
  - timestamp_ms/time64_ns/mixed_nulls: 66.226 → 4.687 µs, −92.9%
  - timestamp_ms/time64_ns/timezone_control: 91.657 → 90.890 µs, −0.8%
  - timestamp_us/date32/no_nulls: 100.840 → 2.410 µs, −97.6%
  - timestamp_us/date32/mixed_nulls: 82.909 → 2.415 µs, −97.1%
  - timestamp_us/date32/timezone_control: 160.590 → 160.570 µs, −0.0%
  - timestamp_us/time32_s/no_nulls: 78.063 → 4.334 µs, −94.4%
  - timestamp_us/time32_s/mixed_nulls: 63.884 → 4.385 µs, −93.1%
  - timestamp_us/time32_s/timezone_control: 86.076 → 86.178 µs, +0.1%
  - timestamp_us/time32_ms/no_nulls: 78.962 → 4.263 µs, −94.6%
  - timestamp_us/time32_ms/mixed_nulls: 65.085 → 4.306 µs, −93.4%
  - timestamp_us/time32_ms/timezone_control: 87.605 → 89.651 µs, +2.3%
  - timestamp_us/time64_us/no_nulls: 80.197 → 2.359 µs, −97.1%
  - timestamp_us/time64_us/mixed_nulls: 66.402 → 2.370 µs, −96.4%
  - timestamp_us/time64_us/timezone_control: 88.763 → 90.953 µs, +2.5%
  - timestamp_us/time64_ns/no_nulls: 79.202 → 4.476 µs, −94.3%
  - timestamp_us/time64_ns/mixed_nulls: 66.193 → 4.479 µs, −93.2%
  - timestamp_us/time64_ns/timezone_control: 87.896 → 89.171 µs, +1.5%
  - timestamp_ns/date32/no_nulls: 100.430 → 2.394 µs, −97.6%
  - timestamp_ns/date32/mixed_nulls: 82.835 → 2.397 µs, −97.1%
  - timestamp_ns/date32/timezone_control: 160.090 → 160.820 µs, +0.5%
  - timestamp_ns/time32_s/no_nulls: 77.068 → 4.346 µs, −94.4%
  - timestamp_ns/time32_s/mixed_nulls: 63.343 → 4.560 µs, −92.8%
  - timestamp_ns/time32_s/timezone_control: 84.787 → 85.908 µs, +1.3%
  - timestamp_ns/time32_ms/no_nulls: 78.946 → 4.114 µs, −94.8%
  - timestamp_ns/time32_ms/mixed_nulls: 64.964 → 4.258 µs, −93.4%
  - timestamp_ns/time32_ms/timezone_control: 87.032 → 89.097 µs, +2.4%
  - timestamp_ns/time64_us/no_nulls: 79.828 → 5.607 µs, −93.0%
  - timestamp_ns/time64_us/mixed_nulls: 65.953 → 5.651 µs, −91.4%
  - timestamp_ns/time64_us/timezone_control: 88.268 → 90.504 µs, +2.5%
  - timestamp_ns/time64_ns/no_nulls: 78.514 → 2.372 µs, −97.0%
  - timestamp_ns/time64_ns/mixed_nulls: 65.090 → 2.371 µs, −96.4%
  - timestamp_ns/time64_ns/timezone_control: 87.227 → 88.661 µs, +1.6%

# What changes are included in this PR?

* Optimize casting timestamp without timezone to time-of-day /
days-since-epoch
* Add tests ensuring that optimized code path preserves the same
behavior as the Chrono code path
* Add benchmarks

# Are these changes tested?

Yes; existing tests pass, new tests added.

# Are there any user-facing changes?

Previously, attempting to cast a timestamp value that was outside
Chrono's supported calendar range produced an error. There are two
behavior changes here:

* When taking the optimized code path (the timestamp does not have a
timezone and the target of the cast is `Date32`, `Time32`, or `Time64`),
the cast will succeed and produce a correct result value
* Otherwise, "safe" mode will be respected.

---------

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate arrow-arith performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants