The C++ Parquet writer does not apply write.parquet.page-size-bytes or write.parquet.dict-size-bytes. Both properties are already declared in TableProperties, but passing them through WriterProperties::FromMap has no effect on the underlying Parquet writer configuration.
On the current main, the ParquetWriter::Impl::Open has already configured the compression, compression level, and the row-count limit, however it builds Arrow's writer properties without setting the page or dictionary size. This prevents callers from using the standard Iceberg properties to tune these sizes.
For instance, providing the following properties should configure a 64 KiB data-page target and a 256 KiB dictionary-page limit on output parquet writer:
write.parquet.page-size-bytes=65536
write.parquet.dict-size-bytes=262144
Proposed change:
- Expose these two settings through
WriterProperties and apply them when constructing Arrow's Parquet writer properties.
- Use the Iceberg defaults already declared in
table_properties.h: 1 MiB for data pages and 2 MiB for dictionary pages.
- Reject invalid or non-positive values before creating the output file.
- Add tests for defaults, overrides, invalid values, and propagation into the underlying writer, plus write/read coverage with custom settings.
Background:
Note:
Byte-based row-group sizing (write.parquet.row-group-size-bytes) can be handled in a separate follow-up because it requires examining the buffering and flush policy.
The C++ Parquet writer does not apply
write.parquet.page-size-bytesorwrite.parquet.dict-size-bytes. Both properties are already declared inTableProperties, but passing them throughWriterProperties::FromMaphas no effect on the underlying Parquet writer configuration.On the current main, the
ParquetWriter::Impl::Openhas already configured thecompression,compression level, and therow-count limit, however it builds Arrow's writer properties without setting thepageordictionarysize. This prevents callers from using the standard Iceberg properties to tune these sizes.For instance, providing the following properties should configure a 64 KiB data-page target and a 256 KiB dictionary-page limit on output parquet writer:
Proposed change:
WriterPropertiesand apply them when constructing Arrow's Parquet writer properties.table_properties.h: 1 MiB for data pages and 2 MiB for dictionary pages.Background:
Parquet.WriteBuilder.Context.dataContext.ParquetWriterBuilder::from_table_properties.Note:
Byte-based row-group sizing (
write.parquet.row-group-size-bytes) can be handled in a separate follow-up because it requires examining the buffering and flush policy.