feat: add a sorted TPC-H benchmark dataset by shinzoxD · Pull Request #646 · datafusion-contrib/datafusion-distributed

shinzoxD · 2026-08-15T19:57:50Z

Summary

Adds a deterministic tpch/sorted_sf1 benchmark dataset variant so physical data ordering is a controlled dimension while keeping the standard TPC-H query workload. Part of #629, closes #632.

Uses the same tpchgen generators and Arrow writer as unsorted TPC-H, recording table-specific Parquet sorting_columns metadata in one pass (prepare-tpch --sorted).

Sort keys

Table	Sort columns
region	`r_regionkey`
nation	`n_nationkey`
customer	`c_custkey`
supplier	`s_suppkey`
part	`p_partkey`
partsupp	`ps_partkey`
orders	`o_orderkey`
lineitem	`l_orderkey`, `l_linenumber`

partsupp records only ps_partkey because ps_suppkey wraps within each part.

Data is written to testdata/tpch/sorted_sf<scale-factor>/.

# Default SF1, 16 files per table
./benchmarks/gen-tpch-sorted.sh

# Tiny smoke-scale dataset
SCALE_FACTOR=0.01 ./benchmarks/gen-tpch-sorted.sh

WORKERS=8 ./benchmarks/run.sh --threads 2 --dataset tpch/sorted_sf1

Queries stay in testdata/tpch/queries. run_local extracts the tpch suite from tpch/sorted_sf1, so queries_for_dataset is unchanged. register_tables is unchanged.

Implementation

benchmarks/src/datasets/tpch.rs: optional sorting_columns on the existing writer
prepare-tpch --sorted instead of a separate async rewrite command
benchmarks/gen-tpch-sorted.sh writes testdata/tpch/sorted_sf<scale-factor>

SF1 was not generated in this PR (heavy, and blocked here by the OpenSSL build). Use SCALE_FACTOR=0.01 to smoke the path, then ./benchmarks/gen-tpch-sorted.sh for SF1.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

shinzoxD · 2026-08-15T21:11:34Z

CI was failing while compiling datafusion-distributed-benchmarks:

error[E0382]: use of moved value: `df`
--> benchmarks/src/datasets/tpch_sorted.rs:163:10
    let mut stream = df.execute_stream().await?;
    drop(df);  // used after move

DataFrame::execute_stream takes self, so the follow-up drop(df) was a use-after-move. Clippy also treated the unused std::sync::Arc import as an error under -D warnings.

Fixed in 077e224: drop the finished stream (and session) so file handles are released before the table directory is replaced, and move the unused import plus test-only sort helpers into the test module.

gabotechs · 2026-08-17T10:31:07Z

+    parts: usize,
+) -> Result<(), Box<dyn std::error::Error>> {
+    tpch::generate_tpch_data(data_dir, sf, parts)?;
+    rewrite_tables_globally_sorted(data_dir).await


This extra read/sort/rewrite pass should not be needed. It should be enough to implement the sorted layout as a small variant of the existing datasets/tpch.rs: reuse the same tpchgen generators and Arrow writer, passing table-specific SortingColumns to WriterProperties when each Parquet file is created. I verified that the pinned tpchgen emits lineitem ordered by (l_orderkey, l_linenumber) and orders/customer/supplier/part by their generated keys. For partsupp, only ps_partkey is naturally ordered because ps_suppkey wraps, so declare only that prefix. The existing prepare-tpch command can take a sorted-variant flag instead of adding this rewrite implementation and a separate asynchronous preparation path.

Implemented in 26040dc. Sorted TPC-H now reuses the existing tpchgen generators and Arrow writer in datasets/tpch.rs, writing table-specific SortingColumn metadata in one pass. partsupp records only ps_partkey. --sorted is a flag on prepare-tpch; the rewrite path and prepare-tpch-sorted command are gone.

gabotechs · 2026-08-17T10:31:07Z

 fn queries_for_dataset(dataset: &str) -> Result<Vec<(String, String)>, DataFusionError> {
    match dataset {
-        "tpch" => tpch::get_queries()
+        "tpch" | "tpch-sorted" => tpch::get_queries()


We can model this as the TPC-H variant tpch/sorted_sf1, rather than a new top-level tpch-sorted dataset. run_local already extracts tpch as the suite before calling queries_for_dataset, and dataset_path resolves the suffix to testdata/tpch/sorted_sf1, so the existing "tpch" branch works unchanged. The generation script should write to that location and this additional match arm can be removed.

Implemented in 26040dc. The variant is now tpch/sorted_sf1, resolved to testdata/tpch/sorted_sf1. queries_for_dataset keeps the existing "tpch" branch; gen-tpch-sorted.sh writes to that path.

gabotechs · 2026-08-17T10:31:07Z

+        .count())
+}
+
+#[cfg(test)]


We can remove this test module, including its test-only helpers and imports. This is benchmark data-preparation code rather than production code, and these tests add roughly 240 lines of maintenance overhead to what should be a small Parquet-metadata variant of the existing TPC-H generator.

Implemented in 26040dc. Removed the test module, helpers, and tpch_sorted.rs.

gabotechs · 2026-08-19T19:14:50Z

+
+set -e
+
+SCALE_FACTOR=${SCALE_FACTOR:-1}


It should be fine to use reuse the existing gen-tpch.sh script with a simple SORTED={true|false} flag populated by an env variable.

Implemented in f31267d. Sorted generation is now SORTED=true ./gen-tpch.sh; gen-tpch-sorted.sh is removed. Default remains unsorted (SORTED=false).

Add a deterministic tpch-sorted variant whose Parquet tables are globally sorted by documented TPC-H primary keys. Generation follows the existing TPC-H scale-factor model and writes testdata/tpch-sorted/sf<scale-factor>. Fixes datafusion-contrib#632

DataFrame::execute_stream takes self, so drop(df) after starting the stream is a use-after-move. Drop the finished stream instead so Windows file locks are released before the table directory is replaced. Move unused Arc and test-only sort helpers into the test module so clippy -D warnings stays clean.

Reuse the existing tpchgen generators and Arrow writer, recording table-specific Parquet sorting_columns in one pass. Model the dataset as the TPC-H variant tpch/sorted_sf1, fold --sorted into prepare-tpch, and drop the rewrite path and its tests.

Fold the sorted variant into gen-tpch.sh so SORTED=true writes tpch/sorted_sf<scale-factor> with --sorted, and default SORTED=false keeps the existing unsorted path.

gabotechs

This one looks good. Thanks @shinzoxD!

## Summary Adds a deterministic `tpch/sorted_sf1` benchmark dataset variant so physical data ordering is a controlled dimension while keeping the standard TPC-H query workload. Part of datafusion-contrib#629, closes datafusion-contrib#632. Uses the same `tpchgen` generators and Arrow writer as unsorted TPC-H, recording table-specific Parquet `sorting_columns` metadata in one pass (`prepare-tpch --sorted`). ## Sort keys | Table | Sort columns | | --- | --- | | region | `r_regionkey` | | nation | `n_nationkey` | | customer | `c_custkey` | | supplier | `s_suppkey` | | part | `p_partkey` | | partsupp | `ps_partkey` | | orders | `o_orderkey` | | lineitem | `l_orderkey`, `l_linenumber` | `partsupp` records only `ps_partkey` because `ps_suppkey` wraps within each part. Data is written to `testdata/tpch/sorted_sf<scale-factor>/`. ```bash # Default SF1, 16 files per table ./benchmarks/gen-tpch-sorted.sh # Tiny smoke-scale dataset SCALE_FACTOR=0.01 ./benchmarks/gen-tpch-sorted.sh WORKERS=8 ./benchmarks/run.sh --threads 2 --dataset tpch/sorted_sf1 ``` Queries stay in `testdata/tpch/queries`. `run_local` extracts the `tpch` suite from `tpch/sorted_sf1`, so `queries_for_dataset` is unchanged. `register_tables` is unchanged. ## Implementation - `benchmarks/src/datasets/tpch.rs`: optional `sorting_columns` on the existing writer - `prepare-tpch --sorted` instead of a separate async rewrite command - `benchmarks/gen-tpch-sorted.sh` writes `testdata/tpch/sorted_sf<scale-factor>` SF1 was not generated in this PR (heavy, and blocked here by the OpenSSL build). Use `SCALE_FACTOR=0.01` to smoke the path, then `./benchmarks/gen-tpch-sorted.sh` for SF1.

Copilot AI lite review requested due to automatic review settings August 15, 2026 19:57

Copilot AI reviewed Aug 15, 2026

gabotechs reviewed Aug 17, 2026

View reviewed changes

shinzoxD force-pushed the feat-tpch-sorted-632 branch from 077e224 to 26040dc Compare August 19, 2026 19:11

gabotechs reviewed Aug 19, 2026

View reviewed changes

shinzoxD added 4 commits August 24, 2026 20:14

fix: generate sorted TPC-H via SORTED= on gen-tpch.sh

f31267d

Fold the sorted variant into gen-tpch.sh so SORTED=true writes tpch/sorted_sf<scale-factor> with --sorted, and default SORTED=false keeps the existing unsorted path.

shinzoxD force-pushed the feat-tpch-sorted-632 branch from 26040dc to f31267d Compare August 24, 2026 14:46

gabotechs approved these changes Aug 24, 2026

View reviewed changes

gabotechs merged commit 9cc7c01 into datafusion-contrib:main Aug 24, 2026
31 checks passed

Navigation Menu

Uh oh!

feat: add a sorted TPC-H benchmark dataset - #646

feat: add a sorted TPC-H benchmark dataset#646
gabotechs merged 4 commits into
datafusion-contrib:mainfrom
shinzoxD:feat-tpch-sorted-632

shinzoxD commented Aug 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

shinzoxD commented Aug 15, 2026

Uh oh!

gabotechs Aug 17, 2026 •

edited

Loading

Uh oh!

shinzoxD Aug 19, 2026

Uh oh!

gabotechs Aug 17, 2026 •

edited

Loading

Uh oh!

shinzoxD Aug 19, 2026

Uh oh!

gabotechs Aug 17, 2026 •

edited

Loading

Uh oh!

shinzoxD Aug 19, 2026

Uh oh!

gabotechs Aug 19, 2026

Uh oh!

shinzoxD Aug 24, 2026

Uh oh!

gabotechs left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		set -e

		SCALE_FACTOR=${SCALE_FACTOR:-1}

Uh oh!

Conversation

shinzoxD commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Sort keys

Implementation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

shinzoxD commented Aug 15, 2026

Uh oh!

gabotechs Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shinzoxD Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

gabotechs Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shinzoxD Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

gabotechs Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shinzoxD Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

gabotechs Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

shinzoxD Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

gabotechs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shinzoxD commented Aug 15, 2026 •

edited

Loading

gabotechs Aug 17, 2026 •

edited

Loading

gabotechs Aug 17, 2026 •

edited

Loading

gabotechs Aug 17, 2026 •

edited

Loading