GH-48701: [C++][Parquet] Add ALPpd encoding - #48345

Open

prtkgaur wants to merge 154 commits into

apache:mainfrom

prtkgaur:gh540-alp-pseudoDecimal-encoding

Commits on Sep 14, 2026

Add alp code

sfc-gh-pgaur
and
dhirhank
committed
b3d73be
Integrate ALP with arrow
sfc-gh-pgaur
committed
4bd8aff
Add alp benchmark
sfc-gh-pgaur
committed
a6fbb47
Add datasets for alp benchmarking
sfc-gh-pgaur
committed
32e3c65
Update cmake file
sfc-gh-pgaur
committed
e619fe6
Move hpp files to h
sfc-gh-pgaur
committed
d5a69f4
Update flow digram and layout digram to use ASCII and not unicode characters
sfc-gh-pgaur
committed
ffc5d54
Rename cpp files to cc
sfc-gh-pgaur
committed
4c459f0
Update documentation to align with arrow's doxygen style
sfc-gh-pgaur
committed
6671686
Adapt methods and variable names to arrow style
sfc-gh-pgaur
committed
2640f0e
Update the tests to adhere to arrow style code
sfc-gh-pgaur
committed
af19332
Update callers
sfc-gh-pgaur
committed
8e85d36
Fuse FOR and decode loop
sfc-gh-pgaur
committed
e5aefbd
Reduce memory allocation in the decompress call
sfc-gh-pgaur
committed
69cbe5a
Attempt at making decoding faster with SIMD
sfc-gh-pgaur
committed
51300ff
Revert "Attempt at making decoding faster with SIMD"
sfc-gh-pgaur
committed
b2a2cea
Move cpp files to cc
sfc-gh-pgaur
committed
84b169f
Move data file to parquet-testing submodule
sfc-gh-pgaur
committed
5a0d4b5
Update path to the data file
sfc-gh-pgaur
committed
d96ba0e
Adapt files names to arrow convention
sfc-gh-pgaur
committed
ec1e479
File rename
sfc-gh-pgaur
committed
4840cce
Obtain compressed size and number of elements from page header
sfc-gh-pgaur
committed
82fb469
Fix namespace depth
sfc-gh-pgaur
committed
2054874
Better pack the compression block header
sfc-gh-pgaur
committed
d4e7fa7
Rename class
sfc-gh-pgaur
committed
424de34
Rearrage field for vector metadata for better packing
sfc-gh-pgaur
committed
34972f9
Add spec files
sfc-gh-pgaur
committed
d4d10e1
Add more tests
sfc-gh-pgaur
committed
9566e1e
Move number of elements metadata info to uber header
sfc-gh-pgaur
committed
cd19193
Keep version as 1
sfc-gh-pgaur
committed
113d39f
Tests pass
sfc-gh-pgaur
committed
778fbf0
Get rid of numElements and derive it for each vector
sfc-gh-pgaur
committed
ba87438
Update the md files
sfc-gh-pgaur
committed
4f49b1c
Fix ubsan error
sfc-gh-pgaur
committed
b12de9d
Use safe copy
sfc-gh-pgaur
committed
cafe698
Add more unit tests
sfc-gh-pgaur
committed
9bc7992
Templatize AlpEncodedVectorInfo
sfc-gh-pgaur
committed
cb1079c
Fix
sfc-gh-pgaur
committed
055926f
[SpecReview] Num values in page is at max int32
sfc-gh-pgaur
committed
9d1e666
[SpecReview] Rename layout to IntegerEncoding
sfc-gh-pgaur
committed
5cb8130
[SpecReview] Rename enum
sfc-gh-pgaur
committed
4e97e4c
[SpecReview] Make AlpHeader exact 8 bytes
sfc-gh-pgaur
committed
beb5514
[SpecReview] Move all vector data towards the start
sfc-gh-pgaur
committed
901ef06
[SpecReview][Metadata] Introduce wrapper to read metadata
sfc-gh-pgaur
committed
3e81c08
[SpecReview] Separate out ALP and FOR metadata
sfc-gh-pgaur
committed
2cab849
[SpecReview] Remove unnecessary field
sfc-gh-pgaur
committed
de77baa
[FutureFlexibility] Add the FOR specific code behind switch
sfc-gh-pgaur
committed
179f60b
Add unit test
sfc-gh-pgaur
committed
048697f
ALP: Implement offset-based layout for O(1) random access
sfc-gh-pgaur
committed
72d2bae
ALP: Update documentation for offset-based layout
sfc-gh-pgaur
committed
c2adb77
ALP: Add CreateSamplingPreset and EncodeWithPreset APIs
sfc-gh-pgaur
committed
fae42cd
ALP: Update all comments and docs for offset-based layout
sfc-gh-pgaur
committed
3ea75ab
ALP: Add cross-implementation reference blob generator
sfc-gh-pgaur
committed
810ffc7
ALP: Add generate_alp_parquet utility for creating ALP test parquet files
sfc-gh-pgaur
committed
1358469
ALP: Remove version field from header (8B → 7B)
sfc-gh-pgaur
committed
c475464
Update parquet-testing submodule with ALP test data files
sfc-gh-pgaur
committed
54179c1
ALP: Add C++ correctness tests for ALP-encoded parquet files
sfc-gh-pgaur
committed
88a0654
ALP: Add cross-language tests reading Java-generated ALP parquet files
sfc-gh-pgaur
committed
5c66645
ALP: Add float32 cross-language tests and generator support
sfc-gh-pgaur
committed
8320621
Address code review comments
sfc-gh-pgaur
committed
454cb39
Address code review comments: endianness assert and decode bounds checks
sfc-gh-pgaur
committed
a7ebd18
Address code review comments: renames, API cleanup, and simplifications
sfc-gh-pgaur
committed
fd378af
Fix build errors from code review changes
sfc-gh-pgaur
committed
75da39d
Split EstimateCompressedSize loop into batched passes for vectorization
sfc-gh-pgaur
committed
2ca2f0e
Address review nits: uint64_t→int64_t, spell out ALP, use uint8_t for byte spans
sfc-gh-pgaur
committed
99ea525
Move decoder creation outside benchmark loop
sfc-gh-pgaur
committed
aa09dc3
Rename alp_wrapper.h/.cc to alp_codec.h/.cc
sfc-gh-pgaur
committed
059a4b9
Convert ARROW_CHECK to Status returns on decode/load paths
sfc-gh-pgaur
committed
e1d80a9
Add precondition docs to Store methods
sfc-gh-pgaur
committed
b43b1a7
Use signed integer types consistently per Arrow style guide
sfc-gh-pgaur
committed
e5ac660
Move out parameters to last position and remove unused enforce_mode
sfc-gh-pgaur
committed
3b7a7e9
Remove ALP spec files; spec belongs in parquet-format repo
sfc-gh-pgaur
committed
b60c059
Rename decomp/comp params to input/output; remove ARROW_CHECK(false) crash paths
sfc-gh-pgaur
committed
920388e
Copy header fields individually instead of bulk memcpy
sfc-gh-pgaur
committed
b606c55
Replace StaticVector with std::vector for flexible vector size
sfc-gh-pgaur
committed
1cdf26e
Make ALP vector_size configurable on the encode path
sfc-gh-pgaur
committed
a3721e1
Add parquet-level round-trip test for ALP at non-default vector sizes
sfc-gh-pgaur
committed
2d7267b
Address reviewer feedback: use int64_t, reorder out-params last, clean up naming
sfc-gh-pgaur
committed
e86f584
Add sign branching to FastRound for correct rounding of negative values
sfc-gh-pgaur
committed
d735dae
Address reviewer feedback: SafeLoad/SafeStore, struct→class conversion
sfc-gh-pgaur
committed
3870f9c
Convert AlpEncodedVector and AlpEncodedVectorView to private members with getters
sfc-gh-pgaur
committed
83768a7
Remove unused AlpMetadataCache class and redundant GetNumElements()
sfc-gh-pgaur
committed
a38529d
Remove private inheritance from AlpCompression and AlpInlines
sfc-gh-pgaur
committed
198e45f
Replace trailing return type with explicit Result<AlpHeader> in LoadHeader
sfc-gh-pgaur
committed
2fa39cb
Use SafeLoadAs for all header fields in LoadHeader
sfc-gh-pgaur
committed
a02f1c3
Validate header fields in LoadHeader
sfc-gh-pgaur
committed
c65f56d
Change AlpCodec public API from byte size to element count
sfc-gh-pgaur
committed
21c16ae
Use uint8_t* consistently in AlpCodec API instead of char*
sfc-gh-pgaur
committed
1259946
Replace trailing return types with explicit return types
sfc-gh-pgaur
committed
7f93133
Cite ALP paper §3.1.2 for combination tie-break rule
sfc-gh-pgaur
committed
537bb0a
Pin AlpMode underlying type to uint8_t for serialization safety
sfc-gh-pgaur
committed
f6754f9
Address reviewer nits: doc tweaks, sizeof(member_), incremental-decode TODO
sfc-gh-pgaur
committed
2fd3389
Convert AlpSampler uint64_t to int64_t
sfc-gh-pgaur
committed
fef545d
Remove GH-48701 reference from incremental-decode TODO
sfc-gh-pgaur
committed
9a4c491
Document actual crash/abort behavior in AlpCodec encode-path docs
sfc-gh-pgaur
committed
3e00c10
Use bit_util::CeilDiv for vector-count computations in AlpCodec
sfc-gh-pgaur
committed
d851810
Replace memcmp with IsBitwiseEqual helper in alp_test.cc
sfc-gh-pgaur
committed
fb18dcd
Type-parameterize alp_test integration tests and fix TruncatedData
sfc-gh-pgaur
committed
c2da81a
Type-parameterize AlpEncodingAdHoc tests in encoding_test.cc
sfc-gh-pgaur
committed
b8add9b
Widen RandomData test range to exercise ALP exception fallback path
sfc-gh-pgaur
committed
50c3ae6
Remove uncompiled generate_alp_parquet and generate_reference_blobs
sfc-gh-pgaur
committed
e1019ca
Add incremental-encode TODO and snake_case rename in AlpEncoder
sfc-gh-pgaur
committed
f9dc7e7
Rename EncodeAlp parameter combinations to preset
sfc-gh-pgaur
committed
44533b3
Use bit_util::IsPowerOf2 and BytesForBits instead of raw bit manipulation
sfc-gh-pgaur
committed
7ea926c
Remove remaining 'non-templated' references in ALP comments
sfc-gh-pgaur
committed
d1feff0
Remove GetBitPackedSize wrapper, inline bit_util::BytesForBits at callers
sfc-gh-pgaur
committed
9d4eb7b
Code review changes
sfc-gh-pgaur
committed
3accab0
Convert AlpCodec public API from ARROW_CHECK aborts to Status returns
sfc-gh-pgaur
committed
05c8960
Use arrow::csv::TableReader for ALP benchmark dataset loading
sfc-gh-pgaur
committed
bbb632a
Reorder AlpEncoded(For)VectorInfo so kStoredSize uses sizeof(member_)
sfc-gh-pgaur
committed
460d644
Make ALP num_exceptions and exception positions unsigned
sfc-gh-pgaur
committed
3f1fd43
Enforce the spec's [3, 15] log_vector_size range in ALP
sfc-gh-pgaur
committed
d6c741c
Use array matchers in ALP tests; note PLAIN fallback
sfc-gh-pgaur
committed
715d5f8
Document the [3, 15] vector_size range in AlpCodec docs
sfc-gh-pgaur
committed
c281092
Put EncodeAlp output parameters last
sfc-gh-pgaur
committed
4e31f0f
Test that ALP rejects malformed page headers
sfc-gh-pgaur
committed
e60da88
Use std::countr_zero instead of __builtin_ctz in ALP
sfc-gh-pgaur
committed
0dc78da
Apply clang-format 18 to the ALP sources
sfc-gh-pgaur
committed
fd235db
Validate ALP vector metadata before it steers the decode
sfc-gh-pgaur
committed
47e5b15
Correct inaccurate ALP comments; drop unused UnsafeResize
sfc-gh-pgaur
committed
f98fdcb
Type-parameterize ALP compression tests; tighten round-trip asserts
sfc-gh-pgaur
committed
13c6e8a
Drop the ALP benchmark that needs out-of-tree datasets
sfc-gh-pgaur
committed
c14e579
Test ALP against the parquet-testing conformance file
sfc-gh-pgaur
committed
e808be7
Put ALP behind an opt-in writer flag
sfc-gh-pgaur
committed
535f67a
Tighten the ALP tests and document the encoding
sfc-gh-pgaur
committed
24b04d0
Validate the ALP element count and the offset chain
sfc-gh-pgaur
committed
ba44d71
Take the ALP value count from the page's own header
sfc-gh-pgaur
committed
824deb3
Encode an all-null ALP page as a bare header
sfc-gh-pgaur
committed
a7b974f
Build the ALP sources once, and under meson too
sfc-gh-pgaur
committed
0a5d442
Serialize ALP little-endian instead of refusing to build
sfc-gh-pgaur
committed
03684e0
Move AlpMode out of the public header
sfc-gh-pgaur
committed
f91c96f
Test the ALP element-count and offset-chain validation
sfc-gh-pgaur
committed
90f30a8
Cover batched ALP decoding, and trim the encoder TODOs
sfc-gh-pgaur
committed
e3015fc
Decode ALP pages one vector at a time
sfc-gh-pgaur
committed
2df3aa7
Move the ALP end-to-end tests to their own file
sfc-gh-pgaur
committed
22302ed
Name the ALP headers _internal.h
sfc-gh-pgaur
committed
0a8a7cc
Fold the ad-hoc ALP vector-size test into the typed suite
sfc-gh-pgaur
committed
8f3da39
Stop allocating scratch on every ALP vector decode
sfc-gh-pgaur
committed
b46d37e
Drop dead ALP code, link the TODOs, unpause the benchmarks
sfc-gh-pgaur
committed
a46d939
Export AlpCodec and concatenate the ALP namespaces
sfc-gh-pgaur
committed
38bad7a
Drop the AlpSampler members that were compile-time constants
sfc-gh-pgaur
committed
f3e8876
Guard the loop pragmas behind the compiler that honours them
sfc-gh-pgaur
committed
5de0a86
Export the nested VectorReader as well as its enclosing class
sfc-gh-pgaur
committed
7cbddd8
Reject an ALP page whose values outlive its definition levels
sfc-gh-pgaur
committed
1f3114f
Size the sample buffer without going through a double
sfc-gh-pgaur
committed
3bf1b71
Say in prose what the target type may be
sfc-gh-pgaur
committed
94fae20
Export the decode member templates from the shared library
sfc-gh-pgaur
committed
dd8353d
Cast the element count the ALP test passes
sfc-gh-pgaur
committed
9aa9bc1
Export the internal ALP classes the test links against
sfc-gh-pgaur
committed
a42f4d9
Widen the shift the ALP test takes a modulus with
sfc-gh-pgaur
committed
b5c25ba
Give the ALP roundtrip helper a name of its own
sfc-gh-pgaur
committed
3a1a45a
Read the stored size through its accessor
sfc-gh-pgaur
committed
befda26
Skip the byte copies that have nothing to copy
sfc-gh-pgaur
committed
869c109
Keep the encoding bounds one ulp clear of the integer limit
sfc-gh-pgaur
committed
7462b91