Make the datafusion-spark function library available in datafusion-cli behind an opt-in flag

[andygrove]

Is your feature request related to a problem or challenge?

datafusion-cli does not register the datafusion-spark function library, so
there is no interactive way to run a Spark-compatible function. It fails as
though the function does not exist:

$ ./target/debug/datafusion-cli -c "SELECT next_day(arrow_cast(95026236,'Date32'),'Mon');"
Error during planning: Invalid function 'next_day'. Did you mean 'today'?

The only way to execute one today is to add a query to a file under
datafusion/sqllogictest/test_files/spark/ and run the sqllogictest target.
That works, but it is a poor fit for exploration: every probe is an edit plus a
test run, throwaway queries have to be deleted afterwards, and the error above
sends people looking for a missing implementation rather than a missing
registration.

datafusion-cli/Cargo.toml has no datafusion-spark dependency at all today.

Describe the solution you'd like

Make the Spark function library available in datafusion-cli behind an
explicit opt-in. A CLI flag such as --spark, a SET option, or both.

Describe alternatives you've considered

Registering unconditionally is not viable. datafusion_spark::register_all
overrides any existing function with the same name, and
SessionStateBuilderSpark::with_spark_features documents the same behavior:

Note: This overwrites any previously registered items with the same name.

Names present in both libraries include abs, which behaves differently:
datafusion-spark's abs wraps on integral overflow when
datafusion.execution.enable_ansi_mode is false, while core abs raises. A
default-on registration would silently change results for every existing CLI
user with no indication in the query that a different function ran. The opt-in
has to be explicit for that reason, not merely for tidiness.

An EXPLAIN-visible or banner indication of which library is active would help
here, since the shadowing is otherwise invisible.

Additional context

Raised in review of #23893, where the
CLI's behavior had to be documented as a pitfall for the
audit-datafusion-spark-expression skill.

[goutamadwant]

take

[goutamadwant]

@alamb @andygrove I opened #24179 to add an opt-in --spark CLI flag. The flag registers the Spark-compatible function library for the session while leaving the default CLI registry unchanged. Let me know if this looks good. thanks!