feat(blob): support blob.split-by-file-size to weigh blob files in scan splitting

[SteNicholas]

Purpose

Linked issue: close #416

Align with CoreOptions.BLOB_SPLIT_BY_FILE_SIZE in apache/paimon. This PR adds the blob.split-by-file-size option (no default value) to control whether blob file size is considered as a factor when performing scan splitting. When unset, CoreOptions::BlobSplitByFileSize() derives the value from the negation of blob-as-descriptor: content reads keep counting blob file sizes, while descriptor reads, which materialize only small descriptors instead of blob payloads, weigh a blob file only by the open file cost. DataEvolutionSplitGenerator gains a count_blob_size flag wired from the option in the data-evolution scan path, so huge blob files no longer dominate split weights and shatter the scan into many tiny splits when their contents are not read.

Tests

• UT CoreOptionsTest.TestDefaultValue: the option defaults to true when neither it nor blob-as-descriptor is set.
• UT CoreOptionsTest.TestFromMap: with blob-as-descriptor=true and the option unset, CoreOptions::BlobSplitByFileSize() derives false.
• UT CoreOptionsTest.TestFallback: an explicit blob.split-by-file-size=true wins over the default derived from blob-as-descriptor=true.
• UT SplitGeneratorTest.TestDataEvolutionBlobSplitByFileSize: the same data/blob file layout packs into two splits when blob file sizes count in the weight, and into a single split when blob files weigh only the open file cost.

API and Format

Adds the public option constant Options::BLOB_SPLIT_BY_FILE_SIZE in include/paimon/defs.h and the getter CoreOptions::BlobSplitByFileSize(). The DataEvolutionSplitGenerator constructor gains a count_blob_size parameter (internal, under src/). No storage format or protocol change.

Documentation

The option is documented in the include/paimon/defs.h doc comment, following the existing blob options.

Generative AI tooling

Generated-by: Claude Code (Claude Fable 5)

🤖 Generated with Claude Code

[CLAassistant]

All committers have signed the CLA.

[Copilot]

Pull request overview

Adds a new read option to align paimon-cpp scan splitting behavior with Java Paimon for blob tables, so that descriptor-based reads can avoid over-weighting large .blob payload files during split planning.

Changes:

• Introduces public option Options::BLOB_SPLIT_BY_FILE_SIZE and CoreOptions::BlobSplitByFileSize() with fallback derived from blob-as-descriptor.
• Wires the option into the data-evolution scan path by extending DataEvolutionSplitGenerator with a count_blob_size flag and adjusting its weight function to treat .blob files specially.
• Adds unit tests covering CoreOptions fallback/precedence and the resulting split packing behavior.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/paimon/core/table/source/table_scan.cpp	Passes CoreOptions::BlobSplitByFileSize() into the data-evolution split generator construction.
src/paimon/core/table/source/split_generator_test.cpp	Adds a focused test ensuring blob-weighting changes split packing as expected.
src/paimon/core/table/source/data_evolution_split_generator.h	Extends the constructor to accept count_blob_size and stores it.
src/paimon/core/table/source/data_evolution_split_generator.cpp	Updates split weight computation to optionally discount .blob file sizes.
src/paimon/core/core_options.h	Exposes CoreOptions::BlobSplitByFileSize() getter.
src/paimon/core/core_options.cpp	Parses blob-as-descriptor and blob.split-by-file-size, and implements derived fallback behavior.
src/paimon/core/core_options_test.cpp	Adds assertions for default/derived/override behavior of BlobSplitByFileSize().
src/paimon/common/defs.cpp	Defines the new option key string constant.
include/paimon/defs.h	Documents the new public option constant.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.