chore(deps): bump databricks-zerobus-ingest-sdk to 2.4.0 and arrow to 59.1.0

[flaviofcruz]

Summary

Upgrade databricks-zerobus-ingest-sdk to 2.4.0 and with it arrow dependencies to 59.1.0.

Vector configuration

No changes.

How did you test this PR?

Manually run vector with configuration:

  sinks:
    my_zerobus_sink:
      type: databricks_zerobus
      ingestion_endpoint: https://ingest.dev.databricks.com
      unity_catalog_endpoint: https://your-workspace.cloud.databricks.com
      table_name: main.default.vector_logs
      auth:
        strategy: oauth
        client_id: ${DATABRICKS_CLIENT_ID}
        client_secret: ${DATABRICKS_CLIENT_SECRET}

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

[yorickvanzweeden]

Hey @flaviofcruz ,

(Not a maintainer of Vector here)

This PR upgrades the Zerobus SDK and Arrow dependencies, but it does not implement the logical-to-physical encoding required for Databricks VARIANT columns.

For example, Vector may receive:

 "gauge": {
   "value": 42.5,
   "attributes": "{\"instance\":\"10.0.0.1:8000\"}"
 }

However, Unity Catalog exposes a VARIANT column through Arrow as:

 struct<
   metadata: large_binary,
   value: large_binary
 >

The existing serializer converts the Vector event to JSON and passes it directly to Arrow’s schema-driven decoder. Arrow therefore sees a JSON string where the schema requires a struct, resulting in an error such as:

 gauge.attributes: expected { got "..."

Passing a native JSON object would not be sufficient either. metadata and value are not ordinary object fields: they contain the binary Parquet Variant representation. Raw JSON cannot simply be placed into value.

That conversion is what #25890 adds. It:

1. Recognizes the Arrow representation of a VARIANT.
2. Encodes JSON with VariantBuilder.
3. Supplies the resulting metadata and value buffers in Arrow’s binary representation.
4. Recursively handles VARIANTs nested inside structs, lists, and maps.
5. Preserves the required nullability of the internal fields.

The SDK only transports an already-valid Arrow RecordBatch, while Arrow sees only a generic struct and cannot infer Databricks VARIANT semantics. Therefore, upgrading those dependencies alone is insufficient: Vector still needs an explicit adapter from ordinary JSON values to the physical VARIANT representation.

Please let me know if you see any mistakes in my logic!

Best, Yorick