fix: [spark] honor ANSI mode in next_day, and fix a panic and a missing argument shape

[andygrove]

Which issue does this PR close?

• Closes #.

No single issue tracks these fixes. They were surfaced by an audit of
next_day against the Spark source and a live Spark 4.2.0. The divergences
this PR does not fix are filed separately and referenced from the test file:

• [Bug] next_day rejects STRING and TIMESTAMP start dates that Spark accepts #23889 — next_day rejects
  STRING and TIMESTAMP start dates that Spark accepts
• [Bug] next_day cross-version divergence: ANSI default flip and error class change between Spark 3.5 and 4.x #23890 — cross-version divergence
  in the ANSI default and error class between Spark 3.5 and 4.x
• [Bug] next_day returns NULL for far-future start dates where Spark returns a value #23891 — next_day returns NULL
  for far-future start dates where Spark returns a value

Rationale for this change

Three problems, only one of which is version-related.

The ANSI flag was ignored. Spark's NextDay has taken
failOnError: Boolean = SQLConf.get.ansiEnabled since at least 3.5, raising on
an unparseable day-of-week name and returning NULL otherwise. This is not new
in Spark 4: what changed in 4.0 is the default value of ansiEnabled and the
exception type, not the behavior the flag selects. datafusion-spark ignored
the flag entirely and always returned NULL, with TODO comments in the source
acknowledging the gap.

A panic reachable from SQL. next_date_for_day_of_week used
NaiveDate + Duration. Date32Type::to_naive_date_opt succeeds up to
chrono::NaiveDate::MAX (epoch day 95026236), and the subsequent addition
then panics on overflow. Every epoch day in 95026230..=95026236 panicked, for
any weekday:

SELECT next_day(arrow_cast(95026236, 'Date32'), 'Mon'::string);
-- thread panicked: `NaiveDate + TimeDelta` overflowed

This is reachable from plain SQL and from any Date32 column containing such a
value.

A whole argument shape was unsupported. SELECT next_day(<date literal>, <string column>) fell into an exec_err! catch-all and raised a hard error on
a query Spark evaluates normally.

The last two are version-independent bugs. The cross-version divergence proper,
the ANSI default flip and the error class change between 3.5 and 4.x, is not
fixed here and is tracked by
#23890.

What changes are included in this PR?

• next_date_for_day_of_week uses checked_add_signed, trading the panic for
  NULL. NULL still diverges from Spark, which computes on the epoch day as
  Int with no calendar limit, so the residual gap is filed as [Bug] next_day returns NULL for far-future start dates where Spark returns a value #23891.
• The match over ColumnarValue shapes is now exhaustive over all four
  combinations, so the (Scalar, Array) case works and the compiler enforces
  coverage. The _ => exec_err! catch-all is gone.
• ANSI mode is honored via datafusion.execution.enable_ansi_mode, matching
  how math/abs.rs and math/negative.rs read it. Spark's per-row null
  intolerance is preserved: a NULL start date short-circuits to NULL before the
  day-of-week name is validated, so next_day(NULL, 'xx') is NULL in both
  modes. On the (Array, Scalar) path this is expressed as
  date_array.null_count() < date_array.len(), which is the closed form of
  exactly when the row-wise path raises.
• The error message gains Spark 4.2.0's trailing period, matching the
  ILLEGAL_DAY_OF_WEEK template rather than Spark 3.5.8's older
  IllegalArgumentException text.
• spark_next_day is split into parse_day_of_week and
  next_date_for_day_of_week. The old code round-tripped "MO" through
  "MONDAY" into str::parse::<Weekday>() with an unreachable Err arm; the
  replacement is a single table matching DateTimeUtils.getDayOfWeekFromString
  case for case.

Are these changes tested?

Yes. next_day.slt grows from 90 to 336 lines. Every expected value was
observed by running the query against a local pyspark==4.2.0 rather than
derived from reading the Spark source.

Coverage is organized as a product rather than a checklist, because the two
bugs above hid in the gaps between axes: all four ColumnarValue shapes, each
crossed with all-NULL, some-NULL and no-NULL inputs, each crossed with both
ANSI modes. The (Array, Scalar) null-intolerance case in particular passes on
the row-wise path and previously failed on the vectorized one, so testing only
one shape would have missed it.

Also covered: boundary dates through +262142-12-31 including the entire
previously-panicking range, day-name parsing (all three lengths, case
insensitivity, whitespace padding, non-ASCII), and the argument type and arity
errors.

Divergences that are not fixed here are checked in as commented-out queries
carrying the Spark 4.2.0 result and a link to the tracking issue. Uncommenting
one is the contract for verifying a future fix.

Verified with:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --test sqllogictests -- spark/datetime/next_day
cargo test -p datafusion-spark --lib next_day

[codecov-commenter]

Codecov Report

❌ Patch coverage is 89.83051% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.94%. Comparing base (140c7c5) to head (9f94893).

Files with missing lines	Patch %	Lines
datafusion/spark/src/function/datetime/next_day.rs	89.83%	9 Missing and 3 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23892   +/-   ##
=======================================
  Coverage   81.93%   81.94%           
=======================================
  Files        1136     1136           
  Lines      429152   429199   +47     
  Branches   429152   429199   +47     
=======================================
+ Hits       351633   351694   +61     
+ Misses      56475    56461   -14     
  Partials    21044    21044           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[andygrove]

Rebased onto main. The conflict was substantive rather than textual, so noting the resolution here.

#24194 landed a fix for the same panic while this branch was open, and it closes #23891 — the issue this branch also addressed. The two fixes are not equivalent:

• This branch used checked_add_signed, which avoids the panic but returns NULL once the next occurrence passes chrono::NaiveDate::MAX.
• fix: prevent next_day panic on far-future start dates #24194 computes the result on the epoch day (days.checked_add(delta)), which returns the value Spark's Int arithmetic produces.

main's behavior is the correct one, so next_date_for_day_of_week now uses the epoch-day arithmetic and the weaker fix is dropped. Consequences:

• The unit test next_day_returns_null_past_the_last_representable_date asserted NULL past NaiveDate::MAX. It is renamed to ..._returns_values_past_... and now asserts the epoch day. A start day that is not itself a representable date still returns NULL, since the weekday is derived via NaiveDate.
• Three next_day.slt cases asserted NULL for the same reason. Those epoch days have no Date32 rendering, so they are cast to Int32 and assert the epoch day, matching how fix: prevent next_day panic on far-future start dates #24194 wrote its own slt cases.
• fix: prevent next_day panic on far-future start dates #24194's next_day_handles_far_future_start_dates called spark_next_day, which this branch splits into parse_day_of_week and next_date_for_day_of_week. It is ported to the new function and kept, along with its [Bug] next_day returns NULL for far-future start dates where Spark returns a value #23891 reference.

What is left specific to this branch is the ANSI-mode handling and the missing (date literal, string column) argument shape.

Verified on the rebased branch: datafusion-spark 281 unit tests pass, and all 244 spark/ slt files pass. cargo fmt and cargo clippy -p datafusion-spark --all-targets -D warnings are clean.