Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
154 commits
Select commit Hold shift + click to select a range
b3d73be
Add alp code
sfc-gh-pgaur Dec 4, 2025
4bd8aff
Integrate ALP with arrow
sfc-gh-pgaur Dec 4, 2025
a6fbb47
Add alp benchmark
sfc-gh-pgaur Dec 4, 2025
32e3c65
Add datasets for alp benchmarking
sfc-gh-pgaur Dec 4, 2025
e619fe6
Update cmake file
sfc-gh-pgaur Dec 4, 2025
d5a69f4
Move hpp files to h
sfc-gh-pgaur Dec 6, 2025
ffc5d54
Update flow digram and layout digram to use ASCII and not unicode cha…
sfc-gh-pgaur Dec 7, 2025
4c459f0
Rename cpp files to cc
sfc-gh-pgaur Dec 7, 2025
6671686
Update documentation to align with arrow's doxygen style
sfc-gh-pgaur Dec 7, 2025
2640f0e
Adapt methods and variable names to arrow style
sfc-gh-pgaur Dec 7, 2025
af19332
Update the tests to adhere to arrow style code
sfc-gh-pgaur Dec 7, 2025
8e85d36
Update callers
sfc-gh-pgaur Dec 7, 2025
e5aefbd
Fuse FOR and decode loop
sfc-gh-pgaur Dec 7, 2025
69cbe5a
Reduce memory allocation in the decompress call
sfc-gh-pgaur Dec 7, 2025
51300ff
Attempt at making decoding faster with SIMD
sfc-gh-pgaur Dec 8, 2025
b2a2cea
Revert "Attempt at making decoding faster with SIMD"
sfc-gh-pgaur Dec 8, 2025
84b169f
Move cpp files to cc
sfc-gh-pgaur Dec 8, 2025
5a0d4b5
Move data file to parquet-testing submodule
sfc-gh-pgaur Dec 8, 2025
d96ba0e
Update path to the data file
sfc-gh-pgaur Dec 9, 2025
ec1e479
Adapt files names to arrow convention
sfc-gh-pgaur Dec 15, 2025
4840cce
File rename
sfc-gh-pgaur Dec 15, 2025
82fb469
Obtain compressed size and number of elements from page header
sfc-gh-pgaur Dec 15, 2025
2054874
Fix namespace depth
sfc-gh-pgaur Dec 16, 2025
d4e7fa7
Better pack the compression block header
sfc-gh-pgaur Dec 16, 2025
424de34
Rename class
sfc-gh-pgaur Dec 16, 2025
34972f9
Rearrage field for vector metadata for better packing
sfc-gh-pgaur Dec 16, 2025
d4d10e1
Add spec files
sfc-gh-pgaur Dec 31, 2025
9566e1e
Add more tests
sfc-gh-pgaur Jan 2, 2026
cd19193
Move number of elements metadata info to uber header
sfc-gh-pgaur Jan 12, 2026
113d39f
Keep version as 1
sfc-gh-pgaur Jan 12, 2026
778fbf0
Tests pass
sfc-gh-pgaur Jan 12, 2026
ba87438
Get rid of numElements and derive it for each vector
sfc-gh-pgaur Jan 12, 2026
4f49b1c
Update the md files
sfc-gh-pgaur Jan 12, 2026
b12de9d
Fix ubsan error
sfc-gh-pgaur Jan 13, 2026
cafe698
Use safe copy
sfc-gh-pgaur Jan 13, 2026
9bc7992
Add more unit tests
sfc-gh-pgaur Jan 13, 2026
cb1079c
Templatize AlpEncodedVectorInfo
sfc-gh-pgaur Jan 14, 2026
055926f
Fix
sfc-gh-pgaur Jan 14, 2026
9d1e666
[SpecReview] Num values in page is at max int32
sfc-gh-pgaur Jan 14, 2026
5cb8130
[SpecReview] Rename layout to IntegerEncoding
sfc-gh-pgaur Jan 14, 2026
4e97e4c
[SpecReview] Rename enum
sfc-gh-pgaur Jan 14, 2026
beb5514
[SpecReview] Make AlpHeader exact 8 bytes
sfc-gh-pgaur Jan 14, 2026
901ef06
[SpecReview] Move all vector data towards the start
sfc-gh-pgaur Jan 15, 2026
3e81c08
[SpecReview][Metadata] Introduce wrapper to read metadata
sfc-gh-pgaur Jan 15, 2026
2cab849
[SpecReview] Separate out ALP and FOR metadata
sfc-gh-pgaur Jan 15, 2026
de77baa
[SpecReview] Remove unnecessary field
sfc-gh-pgaur Jan 16, 2026
179f60b
[FutureFlexibility] Add the FOR specific code behind switch
sfc-gh-pgaur Jan 16, 2026
048697f
Add unit test
sfc-gh-pgaur Jan 17, 2026
72d2bae
ALP: Implement offset-based layout for O(1) random access
sfc-gh-pgaur Jan 26, 2026
c2adb77
ALP: Update documentation for offset-based layout
sfc-gh-pgaur Jan 26, 2026
fae42cd
ALP: Add CreateSamplingPreset and EncodeWithPreset APIs
sfc-gh-pgaur Jan 27, 2026
3ea75ab
ALP: Update all comments and docs for offset-based layout
sfc-gh-pgaur Jan 27, 2026
810ffc7
ALP: Add cross-implementation reference blob generator
sfc-gh-pgaur Mar 7, 2026
1358469
ALP: Add generate_alp_parquet utility for creating ALP test parquet f…
sfc-gh-pgaur Mar 7, 2026
c475464
ALP: Remove version field from header (8B → 7B)
sfc-gh-pgaur Mar 7, 2026
54179c1
Update parquet-testing submodule with ALP test data files
sfc-gh-pgaur Mar 7, 2026
88a0654
ALP: Add C++ correctness tests for ALP-encoded parquet files
sfc-gh-pgaur Mar 7, 2026
5c66645
ALP: Add cross-language tests reading Java-generated ALP parquet files
sfc-gh-pgaur Mar 7, 2026
8320621
ALP: Add float32 cross-language tests and generator support
sfc-gh-pgaur Mar 8, 2026
454cb39
Address code review comments
sfc-gh-pgaur Apr 22, 2026
a7ebd18
Address code review comments: endianness assert and decode bounds checks
sfc-gh-pgaur Apr 27, 2026
fd378af
Address code review comments: renames, API cleanup, and simplifications
sfc-gh-pgaur Apr 27, 2026
75da39d
Fix build errors from code review changes
sfc-gh-pgaur Apr 27, 2026
2ca2f0e
Split EstimateCompressedSize loop into batched passes for vectorization
sfc-gh-pgaur Apr 27, 2026
99ea525
Address review nits: uint64_t→int64_t, spell out ALP, use uint8_t for…
sfc-gh-pgaur May 6, 2026
aa09dc3
Move decoder creation outside benchmark loop
sfc-gh-pgaur May 6, 2026
059a4b9
Rename alp_wrapper.h/.cc to alp_codec.h/.cc
sfc-gh-pgaur May 6, 2026
e1d80a9
Convert ARROW_CHECK to Status returns on decode/load paths
sfc-gh-pgaur May 6, 2026
b43b1a7
Add precondition docs to Store methods
sfc-gh-pgaur May 6, 2026
e5ac660
Use signed integer types consistently per Arrow style guide
sfc-gh-pgaur May 6, 2026
3b7a7e9
Move out parameters to last position and remove unused enforce_mode
sfc-gh-pgaur May 7, 2026
b60c059
Remove ALP spec files; spec belongs in parquet-format repo
sfc-gh-pgaur May 13, 2026
920388e
Rename decomp/comp params to input/output; remove ARROW_CHECK(false) …
sfc-gh-pgaur May 13, 2026
b606c55
Copy header fields individually instead of bulk memcpy
sfc-gh-pgaur May 13, 2026
1cdf26e
Replace StaticVector with std::vector for flexible vector size
sfc-gh-pgaur May 13, 2026
a3721e1
Make ALP vector_size configurable on the encode path
sfc-gh-pgaur May 13, 2026
2d7267b
Add parquet-level round-trip test for ALP at non-default vector sizes
sfc-gh-pgaur May 13, 2026
e86f584
Address reviewer feedback: use int64_t, reorder out-params last, clea…
sfc-gh-pgaur May 14, 2026
d735dae
Add sign branching to FastRound for correct rounding of negative values
sfc-gh-pgaur Jun 3, 2026
3870f9c
Address reviewer feedback: SafeLoad/SafeStore, struct→class conversion
sfc-gh-pgaur Jun 8, 2026
83768a7
Convert AlpEncodedVector and AlpEncodedVectorView to private members …
sfc-gh-pgaur Jun 8, 2026
a38529d
Remove unused AlpMetadataCache class and redundant GetNumElements()
sfc-gh-pgaur Jun 8, 2026
198e45f
Remove private inheritance from AlpCompression and AlpInlines
sfc-gh-pgaur Jun 8, 2026
2fa39cb
Replace trailing return type with explicit Result<AlpHeader> in LoadH…
sfc-gh-pgaur Jun 8, 2026
a02f1c3
Use SafeLoadAs for all header fields in LoadHeader
sfc-gh-pgaur Jun 8, 2026
c65f56d
Validate header fields in LoadHeader
sfc-gh-pgaur Jun 8, 2026
21c16ae
Change AlpCodec public API from byte size to element count
sfc-gh-pgaur Jun 8, 2026
1259946
Use uint8_t* consistently in AlpCodec API instead of char*
sfc-gh-pgaur Jun 8, 2026
7f93133
Replace trailing return types with explicit return types
sfc-gh-pgaur Jun 25, 2026
537bb0a
Cite ALP paper §3.1.2 for combination tie-break rule
sfc-gh-pgaur Jun 25, 2026
f6754f9
Pin AlpMode underlying type to uint8_t for serialization safety
sfc-gh-pgaur Jun 25, 2026
2fd3389
Address reviewer nits: doc tweaks, sizeof(member_), incremental-decod…
sfc-gh-pgaur Jun 25, 2026
fef545d
Convert AlpSampler uint64_t to int64_t
sfc-gh-pgaur Jun 25, 2026
9a4c491
Remove GH-48701 reference from incremental-decode TODO
sfc-gh-pgaur Jun 25, 2026
3e00c10
Document actual crash/abort behavior in AlpCodec encode-path docs
sfc-gh-pgaur Jun 25, 2026
d851810
Use bit_util::CeilDiv for vector-count computations in AlpCodec
sfc-gh-pgaur Jun 25, 2026
fb18dcd
Replace memcmp with IsBitwiseEqual helper in alp_test.cc
sfc-gh-pgaur Jun 25, 2026
c2da81a
Type-parameterize alp_test integration tests and fix TruncatedData
sfc-gh-pgaur Jun 26, 2026
b8add9b
Type-parameterize AlpEncodingAdHoc tests in encoding_test.cc
sfc-gh-pgaur Jun 26, 2026
50c3ae6
Widen RandomData test range to exercise ALP exception fallback path
sfc-gh-pgaur Jun 26, 2026
e1019ca
Remove uncompiled generate_alp_parquet and generate_reference_blobs
sfc-gh-pgaur Jun 26, 2026
f9dc7e7
Add incremental-encode TODO and snake_case rename in AlpEncoder
sfc-gh-pgaur Jun 26, 2026
44533b3
Rename EncodeAlp parameter combinations to preset
sfc-gh-pgaur Jun 26, 2026
7ea926c
Use bit_util::IsPowerOf2 and BytesForBits instead of raw bit manipula…
sfc-gh-pgaur Jun 26, 2026
d1feff0
Remove remaining 'non-templated' references in ALP comments
sfc-gh-pgaur Jun 26, 2026
9d4eb7b
Remove GetBitPackedSize wrapper, inline bit_util::BytesForBits at cal…
sfc-gh-pgaur Jun 26, 2026
3accab0
Code review changes
sfc-gh-pgaur Jun 28, 2026
05c8960
Convert AlpCodec public API from ARROW_CHECK aborts to Status returns
sfc-gh-pgaur Jun 30, 2026
bbb632a
Use arrow::csv::TableReader for ALP benchmark dataset loading
sfc-gh-pgaur Jun 30, 2026
460d644
Reorder AlpEncoded(For)VectorInfo so kStoredSize uses sizeof(member_)
sfc-gh-pgaur Jun 30, 2026
3f1fd43
Make ALP num_exceptions and exception positions unsigned
sfc-gh-pgaur Aug 12, 2026
d6c741c
Enforce the spec's [3, 15] log_vector_size range in ALP
sfc-gh-pgaur Aug 12, 2026
715d5f8
Use array matchers in ALP tests; note PLAIN fallback
sfc-gh-pgaur Aug 12, 2026
c281092
Document the [3, 15] vector_size range in AlpCodec docs
sfc-gh-pgaur Aug 14, 2026
4e31f0f
Put EncodeAlp output parameters last
sfc-gh-pgaur Aug 24, 2026
e60da88
Test that ALP rejects malformed page headers
sfc-gh-pgaur Aug 24, 2026
0dc78da
Use std::countr_zero instead of __builtin_ctz in ALP
sfc-gh-pgaur Aug 24, 2026
fd235db
Apply clang-format 18 to the ALP sources
sfc-gh-pgaur Aug 24, 2026
47e5b15
Validate ALP vector metadata before it steers the decode
sfc-gh-pgaur Aug 26, 2026
f98fdcb
Correct inaccurate ALP comments; drop unused UnsafeResize
sfc-gh-pgaur Aug 26, 2026
13c6e8a
Type-parameterize ALP compression tests; tighten round-trip asserts
sfc-gh-pgaur Aug 27, 2026
c14e579
Drop the ALP benchmark that needs out-of-tree datasets
sfc-gh-pgaur Aug 31, 2026
e808be7
Test ALP against the parquet-testing conformance file
sfc-gh-pgaur Aug 31, 2026
535f67a
Put ALP behind an opt-in writer flag
sfc-gh-pgaur Aug 31, 2026
24b04d0
Tighten the ALP tests and document the encoding
sfc-gh-pgaur Sep 1, 2026
ba44d71
Validate the ALP element count and the offset chain
sfc-gh-pgaur Sep 1, 2026
824deb3
Take the ALP value count from the page's own header
sfc-gh-pgaur Sep 1, 2026
a7b974f
Encode an all-null ALP page as a bare header
sfc-gh-pgaur Sep 1, 2026
0a5d442
Build the ALP sources once, and under meson too
sfc-gh-pgaur Sep 1, 2026
03684e0
Serialize ALP little-endian instead of refusing to build
sfc-gh-pgaur Sep 1, 2026
f91c96f
Move AlpMode out of the public header
sfc-gh-pgaur Sep 1, 2026
90f30a8
Test the ALP element-count and offset-chain validation
sfc-gh-pgaur Sep 1, 2026
e3015fc
Cover batched ALP decoding, and trim the encoder TODOs
sfc-gh-pgaur Sep 1, 2026
2df3aa7
Decode ALP pages one vector at a time
sfc-gh-pgaur Sep 1, 2026
22302ed
Move the ALP end-to-end tests to their own file
sfc-gh-pgaur Sep 1, 2026
0a8a7cc
Name the ALP headers _internal.h
sfc-gh-pgaur Sep 1, 2026
8f3da39
Fold the ad-hoc ALP vector-size test into the typed suite
sfc-gh-pgaur Sep 3, 2026
b46d37e
Stop allocating scratch on every ALP vector decode
sfc-gh-pgaur Sep 3, 2026
a46d939
Drop dead ALP code, link the TODOs, unpause the benchmarks
sfc-gh-pgaur Sep 3, 2026
38bad7a
Export AlpCodec and concatenate the ALP namespaces
sfc-gh-pgaur Sep 3, 2026
f3e8876
Drop the AlpSampler members that were compile-time constants
sfc-gh-pgaur Sep 3, 2026
5de0a86
Guard the loop pragmas behind the compiler that honours them
sfc-gh-pgaur Sep 7, 2026
7cbddd8
Export the nested VectorReader as well as its enclosing class
sfc-gh-pgaur Sep 7, 2026
1f3114f
Reject an ALP page whose values outlive its definition levels
sfc-gh-pgaur Sep 7, 2026
3bf1b71
Size the sample buffer without going through a double
sfc-gh-pgaur Sep 8, 2026
94fae20
Say in prose what the target type may be
sfc-gh-pgaur Sep 8, 2026
dd8353d
Export the decode member templates from the shared library
sfc-gh-pgaur Sep 8, 2026
9aa9bc1
Cast the element count the ALP test passes
sfc-gh-pgaur Sep 8, 2026
a42f4d9
Export the internal ALP classes the test links against
sfc-gh-pgaur Sep 8, 2026
b5c25ba
Widen the shift the ALP test takes a modulus with
sfc-gh-pgaur Sep 8, 2026
3a1a45a
Give the ALP roundtrip helper a name of its own
sfc-gh-pgaur Sep 8, 2026
befda26
Read the stored size through its accessor
sfc-gh-pgaur Sep 8, 2026
869c109
Skip the byte copies that have nothing to copy
sfc-gh-pgaur Sep 9, 2026
7462b91
Keep the encoding bounds one ulp clear of the integer limit
sfc-gh-pgaur Sep 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cpp/src/arrow/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -562,6 +562,9 @@ endforeach()

set(ARROW_UTIL_SRCS
util/align_util.cc
util/alp/alp.cc
util/alp/alp_codec.cc
util/alp/alp_sampler.cc
util/async_util.cc
util/atfork_internal.cc
util/basic_decimal.cc
Expand Down
3 changes: 3 additions & 0 deletions cpp/src/arrow/meson.build
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,9 @@ arrow_components = {

arrow_util_srcs = [
'util/align_util.cc',
'util/alp/alp.cc',
'util/alp/alp_codec.cc',
'util/alp/alp_sampler.cc',
'util/async_util.cc',
'util/atfork_internal.cc',
'util/basic_decimal.cc',
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/util/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,8 @@ add_arrow_test(bit-utility-test
rle_encoding_test.cc
test_common.cc)

add_arrow_test(alp-test SOURCES alp/alp_test.cc)

add_arrow_test(crc32-test
SOURCES
crc32_test.cc
Expand Down
Loading