Fix MAP columns dropping complex values with EnableArrow=1 (#1505)

[rileythomp]

Summary

With EnableArrow=1 (and complex datatype support disabled, which is the default), selecting a MAP column whose values are complex types returned malformed data — the map value was dropped while the key survived:

SELECT MAP(0, ARRAY(34277,0))
-- expected: {0:[34277,0]}
-- actual:   {0:}

With EnableArrow=0 the server returns the value pre-formatted as a string, so only the Arrow path was affected.

Root cause

When complex datatype support is disabled, ArrowStreamResult.getObjectWithComplexTypeHandling reads the map as raw Arrow JSON (entry-array form, e.g. [{"key":0,"value":[34277,0]}]) and formats it via ComplexDataTypeParser.formatMapString. That hand-rolled formatter called JsonNode.asText() on the value node, which returns "" for any container (array/object) node — so array/struct/nested-map values were rendered empty.

Fix

Delegate map formatting to the existing recursive parser: parseToMap(node, mapMetadata).toString(). parseToMap builds a DatabricksMap whose nested values are recursively converted to DatabricksArray / DatabricksMap / DatabricksStruct, and their toString() methods already emit the canonical {key:value} / [a,b] format. This:

• Produces {0:[34277,0]} for the reported case.
• Reproduces byte-for-byte identical output for scalar maps (verified against all existing formatMapString tests).
• Covers all map-rooted columns with arbitrary nesting (arrays, structs, nested maps, deep nesting).

Also hardened DatabricksMap.convertValue and DatabricksArray.convertElements against null nested values (previously threw an NPE on value.getClass()), so e.g. a null array value in a map now renders as null.

Scope / follow-up

Scoped to MAP-rooted columns (the reported bug). Top-level ARRAY/STRUCT columns whose elements/fields are themselves maps still pass through raw Arrow output — a separate latent gap that can be addressed in a follow-up if desired.

Test plan

• New unit tests in ComplexDataTypeParserTest covering the exact repro plus map-of-arrays, map-of-struct, nested-map, and null-complex-value cases, and the existing scalar-map regressions.
• Strengthened the disabled-support assertions in the testMapOfArrays e2e test so dropped array values would be caught.
• mvn test -pl jdbc-core passes (3493 run; the single unrelated failure is a pre-existing DST/timezone test, Slf4jFormatterTest, expecting EST for a June date). spotless:check is clean.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes incorrect string formatting for MAP columns with complex nested values when EnableArrow=1 and complex datatype support is disabled (default), addressing issue #1505 where nested map values could be dropped (e.g. {0:} instead of {0:[34277,0]}).

Changes:

• Updated ComplexDataTypeParser.formatMapString to use the recursive typed parsing path for MAPs so nested values (ARRAY/STRUCT/MAP) are preserved.
• Hardened DatabricksMap and DatabricksArray conversions against null nested values.
• Added unit and e2e regressions to catch dropped nested values on the Arrow path; updated changelog entry.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/test/java/com/databricks/jdbc/integration/e2e/ComplexTypeQueryTests.java	Strengthens e2e assertions to ensure map array values are not dropped (regression coverage for #1505).
src/test/java/com/databricks/jdbc/api/impl/ComplexDataTypeParserTest.java	Adds unit tests covering map-of-arrays, map-of-struct, nested-map, and null nested values.
src/main/java/com/databricks/jdbc/api/impl/DatabricksMap.java	Adds null guard in value conversion to avoid NPE for null nested values.
src/main/java/com/databricks/jdbc/api/impl/DatabricksArray.java	Adds null guard per element to avoid NPE and preserve nulls in nested arrays.
src/main/java/com/databricks/jdbc/api/impl/ComplexDataTypeParser.java	Routes MAP formatting through recursive parsing to correctly render nested complex values.
NEXT_CHANGELOG.md	Documents the MAP formatting fix and null-handling improvement.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[msrathore-db]

Can you please move the PR to ready for review if you want this to be reviewed