Skip to content

[FLINK-40705][python] Reset Arrow map child writers between batches - #29247

Merged
HuangXingBo merged 2 commits into
apache:masterfrom
auroflow:auroflow/codex/fix-arrow-map-writer-reset
Sep 21, 2026
Merged

HuangXingBo merged 2 commits into
apache:masterfrom
auroflow:auroflow/codex/fix-arrow-map-writer-reset

Conversation

@auroflow

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Fix FLINK-40705: pandas UDFs consuming MAP columns can fail across Arrow batches because map child writers retain their previous write positions. Resetting these writers with the parent keeps map offsets and child positions aligned.

Brief change log

  • Override MapWriter.reset() to reset both key and value writers recursively.
  • Extend ArrowReaderWriterTest to verify MAP values across consecutive batches.

Verifying this change

The regression test checks Arrow IPC round trips with batch sizes 1 and 2, covering direct and nested maps, empty maps, null maps, null values, and a partial final batch.

Both parameterized cases failed before the fix and passed afterward during the original implementation. Tests have not been rerun after the latest rebase.

./mvnw -pl flink-python \
  -Dtest=ArrowReaderWriterTest#testMapsAcrossBatches test

Does this pull request potentially affect one of the following parts:

  • Dependencies: no
  • The public API: no
  • The serializers: yes — Arrow writer lifecycle; no serialization format changes.
  • The runtime per-record code paths: yes — resets child writers at batch boundaries.
  • Deployment or recovery components: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Codex (GPT-6)

Reset key and value writers together with map vectors so consecutive batches reuse aligned offsets. Cover direct and nested maps, nulls, empty maps, and partial batches through Arrow IPC round trips.

Generated-by: Codex (GPT-6)
@auroflow
auroflow marked this pull request as ready for review September 20, 2026 09:32
@flinkbot

flinkbot commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

}
}

@Override

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the finish method of MapWriter was forgotten to be overridden as well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Added finish override to finish both child writers.

@HuangXingBo
HuangXingBo merged commit 3c78d5f into apache:master Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants