ci: move benchmark runners to c8id.metal-48xl - #9283
Open
joseph-isaacs wants to merge 1 commit into
Open
Conversation
joseph-isaacs
added
the
action/bench-all
Claude
github-actions
Bot
removed
the
action/bench-all
joseph-isaacs
marked this pull request as ready for review
August 7, 2026 16:47
Member
|
this will change all of our benchmark right? so bench.vortex.dev will have a big shift? |
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Merging this PR will degrade performance by 10.03%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
Contributor
Benchmarks: String Encoding 📖vortex / vortex-file-compressed / ms (0.927x ➖, 2↑ 0↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
Contributor
Author
|
The CodSpeed regression on
Once a successful Generated by Claude Code |
Contributor
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 1.014x ➖ datafusion / vortex-file-compressed / ns (1.014x ➖, 4↑ 5↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: Likely regression (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.103x ❌, 9↑ 57↓)
datafusion / vortex-compact / ns (1.046x ➖, 11↑ 36↓)
datafusion / parquet / ns (1.008x ➖, 18↑ 27↓)
duckdb / vortex-file-compressed / ns (1.791x ❌, 0↑ 97↓)
duckdb / vortex-compact / ns (1.591x ❌, 0↑ 97↓)
duckdb / parquet / ns (1.013x ➖, 6↑ 16↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb S3 📖Verdict: Likely regression (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.807x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.853x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.948x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.788x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.744x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.160x ✅, 9↑ 0↓)
|
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.974x ➖, 5↑ 1↓)
datafusion / vortex-compact / ns (0.922x ➖, 5↑ 0↓)
datafusion / parquet / ns (0.915x ➖, 5↑ 2↓)
duckdb / vortex-file-compressed / ns (0.974x ➖, 4↑ 3↓)
duckdb / vortex-compact / ns (0.950x ➖, 4↑ 3↓)
duckdb / parquet / ns (1.735x ❌, 0↑ 9↓)
File Size Changes (200 files changed, -0.0% overall, 101↑ 99↓)
Totals:
|
Contributor
Benchmarks: Random Access 📖Verdict: Likely regression (high confidence) How to read Verdict and Engines
random-access / vortex-file-compressed / ns (1.874x ❌, 0↑ 14↓)
random-access / parquet / ns (0.947x ➖, 3↑ 0↓)
random-access / lance / ns (2.147x ❌, 0↑ 18↓)
|
Contributor
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.819x ✅, 32↑ 6↓)
datafusion / vortex-compact / ns (0.830x ✅, 33↑ 5↓)
datafusion / parquet / ns (0.852x ✅, 27↑ 5↓)
duckdb / vortex-file-compressed / ns (0.973x ➖, 22↑ 9↓)
duckdb / vortex-compact / ns (0.855x ✅, 29↑ 5↓)
duckdb / parquet / ns (1.304x ❌, 10↑ 23↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-compact / ns (1.058x ➖, 2↑ 6↓)
datafusion / parquet / ns (0.881x ➖, 3↑ 2↓)
duckdb / vortex-compact / ns (1.089x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.084x ➖, 0↑ 1↓)
|
Contributor
Benchmarks: Statistical and Population Genetics 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.987x ➖, 4↑ 2↓)
duckdb / vortex-compact / ns (0.981x ➖, 5↑ 3↓)
duckdb / parquet / ns (1.031x ➖, 3↑ 4↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.797x ✅, 17↑ 0↓)
datafusion / vortex-compact / ns (0.797x ✅, 20↑ 0↓)
datafusion / parquet / ns (0.789x ✅, 18↑ 0↓)
duckdb / vortex-file-compressed / ns (1.061x ➖, 6↑ 9↓)
duckdb / vortex-compact / ns (1.091x ➖, 4↑ 9↓)
duckdb / parquet / ns (0.971x ➖, 4↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: Appian on NVME 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-compact / ns (0.884x ✅, 6↑ 1↓)
datafusion / parquet / ns (0.911x ➖, 3↑ 0↓)
duckdb / vortex-compact / ns (0.929x ➖, 3↑ 1↓)
duckdb / parquet / ns (0.821x ✅, 7↑ 0↓)
File Size Changes (10 files changed, -63.8% overall, 0↑ 10↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.113x ➖, 0↑ 4↓)
datafusion / vortex-compact / ns (1.016x ➖, 3↑ 4↓)
datafusion / parquet / ns (0.947x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.077x ➖, 1↑ 2↓)
duckdb / vortex-compact / ns (1.163x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.013x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: Likely regression (high confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.100x ❌, 3↑ 14↓)
datafusion / vortex-compact / ns (1.080x ➖, 3↑ 11↓)
datafusion / parquet / ns (0.933x ➖, 5↑ 0↓)
duckdb / vortex-file-compressed / ns (1.731x ❌, 1↑ 21↓)
duckdb / vortex-compact / ns (1.679x ❌, 1↑ 20↓)
duckdb / parquet / ns (0.929x ➖, 2↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.896x ✅, 5↑ 0↓)
datafusion / vortex-compact / ns (0.749x ✅, 8↑ 0↓)
datafusion / parquet / ns (0.832x ✅, 8↑ 1↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 5↑ 2↓)
duckdb / vortex-compact / ns (0.889x ✅, 7↑ 2↓)
duckdb / parquet / ns (1.274x ❌, 1↑ 6↓)
No file size changes detected. |
Contributor
Benchmarks: Compression 📖vortex / vortex-file-compressed / ns (1.088x ➖, 7↑ 14↓)
vortex / vortex-file-compressed / bytes (1.001x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (1.109x ❌, 5↑ 18↓)
vortex / parquet / ns (0.932x ➖, 9↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
|
Update the bench-dedicated runner family used by the PR, develop, and nightly benchmark workflows from c6id.metal to c8id.metal-48xl. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EE1ue4mLVxfSXTXUNbB8Y6
joseph-isaacs
force-pushed
the
claude/benchmark-metal-c8id-update-zzy8n4
branch
from
August 7, 2026 18:03
260de0a to
fcfb171
Compare
Contributor
Author
|
Correcting my earlier comment on the first CodSpeed failure: I said a re-run against a proper The re-run does, however, produce much stronger evidence that this is measurement noise. The force-push changed only commit metadata —
No overlap. A real regression reproduces on the same benchmark across identical input; a set that reshuffles entirely does not. The second run also contradicts itself internally: And the mechanism is still absent. The diff is 4 lines across 4 workflow YAML files, no Suggest acknowledging on CodSpeed rather than treating it as a blocker. Worth noting separately that the noise floor here looks wider than the ±10% threshold — two consecutive runs on an identical tree each produced multiple >12% swings. Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The benchmark workflows all pin their dedicated
runs-onrunner to thec6id.metalfamily. This moves them toc8id.metal-48xlso benchmarks execute on the newer generation bare-metal instance.What changes are included in this PR?
Replaced every
c6id.metalrunner reference withc8id.metal-48xl:.github/workflows/pr-bench-runner.yml—family=in thebenchjob'sruns-onexpression..github/workflows/develop-bench.yml—family=in thebenchjob'sruns-onexpression..github/workflows/sql-bench-matrix.yml— default value of themachine_typeworkflow_call input, which feeds thebuildand downstream jobs..github/workflows/nightly-bench.yml—instance_namefor thex86matrix entry passed into the SQL benchmark matrix.No other
c6id.metalreferences remain under.github/. The unrelatedec2_c6id.8xlargestring invortex-bench/src/runner.rsis a runner-id parser test fixture, not a CI runner selection, so it is untouched.What APIs are changed? Are there any user-facing changes?
None. CI configuration only — no public API, feature flag, or generated-file changes.
Checks run
yamllint --strict -c .yamllint.yamlon the four changed workflow files — passes..github/YAML with no Rust or API impact.Generated by Claude Code