feat(spill): add SpillWriter, SpillReader and SpillChannelManager for…#219
Merged
zjw1111 merged 2 commits intoalibaba:mainfrom Apr 21, 2026
Merged
feat(spill): add SpillWriter, SpillReader and SpillChannelManager for…#219zjw1111 merged 2 commits intoalibaba:mainfrom
zjw1111 merged 2 commits intoalibaba:mainfrom
Conversation
dalingmeng
force-pushed
the
main
branch
5 times, most recently
from
081de6b to
c738a46
Compare
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds spill-file infrastructure to support a spillable write buffer for PK table workflows (issue #149), including channel creation/enumeration, spill file read/write, and corresponding unit tests.
Changes:
- Introduces
FileIOChanneland extendsIOManagerto create unique spill channels and enumerators. - Adds
SpillWriter,SpillReader, andSpillChannelManagerfor writing/reading Arrow IPC spill files and tracking their lifecycles. - Adds unit tests and wires them into the test CMake target.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/paimon/core/mergetree/spill_writer_test.cpp | New tests validating spill write + readback behavior across multiple writers/batches |
| src/paimon/core/mergetree/spill_writer.h | Adds SpillWriter to write Arrow IPC spill files with compression and channel tracking |
| src/paimon/core/mergetree/spill_reader_test.cpp | New tests validating multi-batch read behavior and decoded KeyValue fields |
| src/paimon/core/mergetree/spill_reader.h | Adds SpillReader implementing KeyValueRecordReader over Arrow IPC spill files |
| src/paimon/core/mergetree/spill_channel_manager_test.cpp | New tests for tracking, deleting, and resetting spill channels/files |
| src/paimon/core/mergetree/spill_channel_manager.h | Adds SpillChannelManager to register and delete spill files/channels |
| src/paimon/core/disk/io_manager_test.cpp | Adds tests for new channel creation APIs and enumerator naming/sequencing |
| src/paimon/core/disk/io_manager.cpp | Implements new IOManager channel APIs with RNG + mutex for thread safety |
| src/paimon/CMakeLists.txt | Registers new spill tests in the test build |
| include/paimon/disk/io_manager.h | Extends public IOManager API with channel and enumerator creation |
| include/paimon/disk/file_io_channel.h | Adds FileIOChannel::{ID, Enumerator} for unique spill file naming |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
lxy-9602
reviewed
Apr 13, 2026
dalingmeng
force-pushed
the
main
branch
9 times, most recently
from
286a6ec to
b3c6350
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
dalingmeng
force-pushed
the
main
branch
6 times, most recently
from
be2ae71 to
111fe4a
Compare
zjw1111
reviewed
Apr 16, 2026
lucasfang
reviewed
Apr 20, 2026
… spill-to-disk support
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add FileIOChannel, SpillWriter, SpillReader, SpillChannelManager classes to support spillable write buffer for PK Table
Linked issue: #149
Tests
TEST_F(SpillWriterTest, TestWriteBatch)
TEST_F(SpillReaderTest, TestReadMultipleBatches)
TEST_F(SpillChannelManagerTest.*)
TEST(IOManagerTest, CreateChannelShouldReturnValidAndUniquePaths)
TEST(IOManagerTest, CreateChannelEnumeratorShouldReturnSequentialAndUniquePaths)
API and Format
Documentation
Generative AI tooling
Generated-by: Aone Copilot(Claude-4.6-Opus)