[SPARK-59358][SDP] SCD2 Ignore-null support; Coalesce ignored-null values

[AnishMahto]

This is a stacked PR. See incremental diff here: AnishMahto/spark@SPARK-59640-scd2-version-map-authorship-operations...SPARK-59358-coalesce-ignored-null-columns.

See previous PR in stack here: #58914

What changes were proposed in this pull request?

Once the version map is constructed per incoming row in the microbatch, we need to start actually reconciling columns in the incoming microbatch that have been deduced as unauthored - they should instead inherit the last known authored value.

While reconciling the affected existing rows from the aux/target table, it's also possible out of order received events in the current microbatch affect what authored value existing ignored-null columns inherit; these need to be updated too.

The core algorithm iterates over the full set of affected rows by the incoming microbatch in window order, coalescing unauthored columns values with the last known authored value within the window.

It's worth noting that only columns explicitly in the ignore-null selection in the current reconciliation are coalesced, even if existing rows have initialized version maps from a different ignore-null selection in a previous reconciliation. Version maps themselves are immutable, and the chosen semantic is:

1. Version maps represent per-column authorship. They are the source of truth for what SDP considered authored in the event, per upsert event. This is deduced using whatever the ignore-null selection was when the event was ingested.
2. The current ignore-null selection represents the full set of columns whose unauthored values will be coalesced during this microbatch's reconciliation, for all rows pulled into reconciliation.

Why are the changes needed?

Core algorithm for coalescing/reconciling ignore-null columns for rows in or affected by the incoming microbatch.

Does this PR introduce any user-facing change?

No. Ignore null reconciliation is strictly only invoked if an ignore-null selection is specified. The ignore-null API is not yet released, so all existing AutoCDC workloads necessarily do not specify an ignore-null selection.

How was this patch tested?

Scd2CoalesceIgnoredNullSuite.

Was this patch authored or co-authored using generative AI tooling?

GPT Sol 5.6.