Conversation
Pin arrow/parquet crates to arrow-rs rev 901e69f913bb8a1738009218898959e8f4cd7ec8 via [patch.crates-io] until arrow 60.0.0 is released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tore 0.14, etc) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # datafusion/core/tests/user_defined/user_defined_aggregates.rs # datafusion/core/tests/user_defined/user_defined_scalar_functions.rs # datafusion/core/tests/user_defined/user_defined_window_functions.rs # datafusion/ffi/src/udaf/mod.rs # datafusion/optimizer/src/optimizer.rs
…anges - data_buffers() now returns &Arc<[Buffer]> and GenericByteViewArray::new_unchecked takes Arc<[Buffer]> - ParquetMetaData::column_index()/offset_index() replaced by page_index() returning the new PageIndex struct - PageIndex cannot be constructed outside the parquet crate (apache/arrow-rs#10824): patch serialized index bytes in statistics_order_tests and drop the synthetic page index from single_column_metadata until upstream exposes a constructor - New ColumnOrder/SortOrder variants: INT96 now reports SortOrder::INT96_TIMESTAMP (still treated as untrusted for min/max), and ColumnOrder union fields 2/3 are now recognized
…adapt to API changes
# Conflicts: # Cargo.lock # datafusion/common/src/dfschema.rs # datafusion/physical-expr/src/planner.rs
- Use PageIndexBuilder (apache/arrow-rs#10842) to restore the page index tests in statistics_order_tests.rs - Update RunEndEncoded Display expectation (apache/arrow-rs#10840) - Update decimal formatting expectation (apache/arrow-rs#10869) - Update memory size expectations Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
github-actions
Bot
added
sql
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25335 +/- ##
==========================================
- Coverage 81.93% 81.92% -0.01%
==========================================
Files 1136 1136
Lines 429152 429132 -20
Branches 429152 429132 -20
==========================================
- Hits 351633 351587 -46
- Misses 56475 56498 +23
- Partials 21044 21047 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
alamb
changed the title
60.0.0 and object_store 0.14.2
Contributor
|
Looks like smooth upgrade, thanks @alamb |
25 tasks
Resolved conflicts in Cargo.lock and datafusion/physical-expr/src/scalar_subquery.rs (arrow 60 field metadata conversion combined with main's destructuring). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
alamb
commented
Sep 16, 2026
| "flight-sql-experimental", | ||
| arrow-buffer = { version = "60.0.0", default-features = false } | ||
| arrow-data = { version = "60.0.0", default-features = false } | ||
| arrow-flight = { version = "60.0.0", features = [ |
Contributor
Author
There was a problem hiding this comment.
| memchr = "2.8.1" | ||
| num-traits = { version = "0.2" } | ||
| object_store = { version = "0.13.2", default-features = false } | ||
| object_store = { version = "0.14.2", default-features = false } |
Contributor
Author
There was a problem hiding this comment.
| | alltypes_plain.parquet | 1851 | 8794 | 1 | page_index=false | | ||
| | alltypes_tiny_pages.parquet | 454233 | 268970 | 2 | page_index=true | | ||
| | lz4_raw_compressed_larger.parquet | 380836 | 1331 | 1 | page_index=false | | ||
| | alltypes_plain.parquet | 1851 | 8938 | 1 | page_index=false | |
Contributor
Author
There was a problem hiding this comment.
- due to Implement PARQUET-2249: Introduce IEEE 754 total order arrow-rs#9619 (adds
nan_counttoStatistics, 16 bytes per column chunk) - due to Add
PageIndexBuilderandPageIndexProviderfor Parquet page indexes arrow-rs#10842 (ParquetMetaDataholds anArc<dyn PageIndexProvider>instead of separate column/offset index vecs, 32 bytes smaller) - due to Allow reading and writing more than 32k Parquet row groups arrow-rs#10149 (row group ordinals are
i32rather thani16)
| +---------------------+-----------+-----------------+------+------+ | ||
| | 212 | 0.parquet | 3642 | 0 | 2 | | ||
| | 212 | 1.parquet | 3642 | 1 | 2 | | ||
| | 216 | 0.parquet | 3620 | "0" | 2 | |
Contributor
Author
There was a problem hiding this comment.
- etag changes due to fix: quote ETags per RFC 9110 in InMemory and LocalFileSystem arrow-rs-object-store#770
- the size got bigger due to new ordering info Implement PARQUET-2249: Introduce IEEE 754 total order arrow-rs#9619
| use arrow::{ | ||
| array::AsArray, | ||
| datatypes::{DataType, Field}, | ||
| datatypes::{DataType, Field, Metadata}, |
Contributor
Author
There was a problem hiding this comment.
| Plan with Metrics | ||
| 01)SortExec: TopK(fetch=1), expr=[b@1 DESC], preserve_partitioning=[false], filter=[b@1 IS NULL OR b@1 > bd], metrics=[output_rows=1, output_batches=1, row_replacements=1] | ||
| 02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_single_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 IS NULL OR b@1 > bd ], sort_order_for_reorder=[b@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@0 > 0 OR b_null_count@0 != row_count@2 AND b_max@1 > bd, required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=4, predicate_cache_records=4, scan_efficiency_ratio=21.62% (222/1.03 K)] | ||
| 02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_single_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 IS NULL OR b@1 > bd ], sort_order_for_reorder=[b@1 DESC], reverse_row_groups=true, dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@0 > 0 OR b_null_count@0 != row_count@2 AND b_max@1 > bd, required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=4, predicate_cache_records=4, scan_efficiency_ratio=21.94% (222/1.01 K)] |
Contributor
Author
There was a problem hiding this comment.
| ); | ||
| let err = scalar.eq_array(&run_array, 0).unwrap_err(); | ||
| let expected = "Internal error: could not cast array of type RunEndEncoded(\"run_ends\": non-null Int16, \"values\": Float32) to arrow_array::array::run_array::RunArray<arrow_array::types::Int32Type>"; | ||
| let expected = "Internal error: could not cast array of type RunEndEncoded(non-null Int16, Float32) to arrow_array::array::run_array::RunArray<arrow_array::types::Int32Type>"; |
Contributor
Author
There was a problem hiding this comment.
| if let (Some(first), Some(last)) = (offsets.first(), offsets.last()) { | ||
| let first = first.as_usize(); | ||
| let last = last.as_usize(); | ||
| let first = offsets.first().as_usize(); |
Contributor
Author
There was a problem hiding this comment.
| round(cast(-500 as decimal(10,-2)), -3); | ||
| ---- | ||
| Decimal128(10, -3) 1000 0000 -1000 | ||
| Decimal128(10, -3) 1000 0 -1000 |
Contributor
Author
There was a problem hiding this comment.
| ---- | ||
| Plan with Metrics | ||
| 01)ProjectionExec: expr=[a@0 + 1 as t1.a + Int64(1), power(CAST(a@0 AS Float64), 2) as pow(t1.a,Int64(2))], metrics=[output_rows=100, elapsed_compute=<slt:ignore>, output_bytes=1632.0 B, output_batches=1, expr_0_eval_time=<slt:ignore>, expr_1_eval_time=<slt:ignore>] | ||
| 01)ProjectionExec: expr=[a@0 + 1 as t1.a + Int64(1), power(CAST(a@0 AS Float64), 2) as pow(t1.a,Int64(2))], metrics=[output_rows=100, elapsed_compute=<slt:ignore>, output_bytes=1600.0 B, output_batches=1, expr_0_eval_time=<slt:ignore>, expr_1_eval_time=<slt:ignore>] |
Contributor
Author
There was a problem hiding this comment.
github-actions
Bot
added
the
documentation
alamb
commented
Sep 16, 2026
| converter: StatisticsConverter<'a>, | ||
| column_index: &'a ParquetColumnIndex, | ||
| offset_index: &'a ParquetOffsetIndex, | ||
| page_index: &'a dyn PageIndexProvider, |
Contributor
Author
There was a problem hiding this comment.
This is the nice new API that @etseidl added upstream in apache/arrow-rs#10842 to encapsulate and lay the foundation for supplying the page indexes via API
| Plan with Metrics | ||
| 01)SortExec: TopK(fetch=2), expr=[b@1 ASC NULLS LAST, a@0 DESC], preserve_partitioning=[false], filter=[b@1 < bb OR b@1 = bb AND (a@0 IS NULL OR a@0 > ac)], metrics=[output_rows=2, output_batches=1, row_replacements=2] | ||
| 02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_multi_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 < bb OR b@1 = bb AND (a@0 IS NULL OR a@0 > ac) ], sort_order_for_reorder=[b@1 ASC NULLS LAST, a@0 DESC], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_min@0 < bb OR b_null_count@1 != row_count@2 AND b_min@0 <= bb AND bb <= b_max@3 AND (a_null_count@4 > 0 OR a_null_count@4 != row_count@2 AND a_max@5 > ac), required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=8, predicate_cache_records=8, scan_efficiency_ratio=21.62% (222/1.03 K)] | ||
| 02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/topk_multi_col.parquet]]}, projection=[a, b, c], file_type=parquet, predicate=DynamicFilter [ b@1 < bb OR b@1 = bb AND (a@0 IS NULL OR a@0 > ac) ], sort_order_for_reorder=[b@1 ASC NULLS LAST, a@0 DESC], dynamic_rg_pruning=eligible, pruning_predicate=b_null_count@1 != row_count@2 AND b_min@0 < bb OR b_null_count@1 != row_count@2 AND b_min@0 <= bb AND bb <= b_max@3 AND (a_null_count@4 > 0 OR a_null_count@4 != row_count@2 AND a_max@5 > ac), required_guarantees=[], metrics=[output_rows=4, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, row_groups_pruned_bloom_filter=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=4, pushdown_rows_pruned=0, predicate_cache_inner_records=8, predicate_cache_records=8, scan_efficiency_ratio=21.94% (222/1.01 K)] |
Contributor
Author
There was a problem hiding this comment.
In case it is not clear (I had to double check) the difference here is
- scan_efficiency_ratio=21.62% (222/1.03 K)]
+ scan_efficiency_ratio=21.94% (222/1.01 K)]Specifically since the file size is slightly different, this metric is also reported slightly differently
Contributor
Author
There was a problem hiding this comment.
The same pattern exists in the other tests below as well
alamb
marked this pull request as ready for review
September 16, 2026 19:49
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing arrow-60 (3c2e777) to 140c7c5 (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing arrow-60 (3c2e777) to 140c7c5 (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing arrow-60 (3c2e777) to 140c7c5 (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing arrow-60 (3c2e777) to 140c7c5 (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing arrow-60 (3c2e777) to 140c7c5 (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing arrow-60 (3c2e777) to 140c7c5 (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
60.0.0(August 2026) arrow-rs#9880object_storeto0.14.1#23693Rationale for this change
Update to the latest arrow/parquet release,
60.0.0. See the upstream CHANGELOG for details.What changes are included in this PR?
60.0.0andobject_store0.14.2What is the testing strategy for this PR?
Existing CI.
Are there any user-facing changes?
New dependency versions.
Also, following the upstream change, a few DataFusion APIs now use arrow's new
Metadatatype (thanks @emilk !) instead ofHashMap<String, String>