From cb2b63d8ae481596a14a6ce5131ec88305da7a43 Mon Sep 17 00:00:00 2001 From: Stefan Steiner Date: Sat, 5 Sep 2026 13:25:55 -0700 Subject: [PATCH 1/3] feat(bootstrap)!: source hyperd from the PyPI tableauhyperapi wheels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `hyperdb-bootstrap` fetched `hyperd` from Tableau's Hyper Java API zip, whose filename embeds an opaque build id (`r07abb490`) that cannot be derived from the version. Discovering it required scraping the public releases page — and that scraper had been broken for three-plus releases without anyone noticing, because its tests ran against a synthetic fixture rather than the live page. The PyPI `tableauhyperapi` wheels carry the same engine at a fully constructible URL, and PyPI publishes a sha256 per file: https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-{version}-py3-none-{wheel_tag}.whl The bytes are unchanged. The `hyperd` extracted from the macosx_13_0_arm64 wheel is bit-identical to the one the crate downloaded before — sha256 aef5c81970bb4d84d06fb9513d5ffd722526fce779632a0c5f63d87b6450e478, 277,836,448 bytes, Mach-O 64-bit executable arm64. Both report `minos 13.0`, so the wheel tag is not a raised support floor. Bumping the pin no longer means downloading four ~80 MB archives and hashing them by hand; the digests are read off the JSON API and still committed, since a hash in git attests independently of the host serving the bytes. Wheel tags are stored per platform in the pin rather than hardcoded in Rust: they are not stable across releases (arm64 wheels start at 0.0.19484, and a macOS floor bump would change macosx_13_0_arm64) and a wrong tag 404s silently, so a change should be a visible pin edit. `build.rs` now requires one for every supported platform. `verify` additionally cross-checks each pinned digest against the one PyPI publishes for that exact wheel filename, so it validates the pinned bytes rather than merely that the CDN serves something. The npm workflow's independent pin moves to the wheel URL and its guard grows a wheel-tag cross-check. The release strategy is settled: this break lands inside the 1.0.0-rc line as 1.0.0-rc.2, pinned by a `Release-As:` footer on the following commit. The `!` below marks the break honestly; the footer keeps it from computing 2.0.0. BREAKING CHANGE: hyperdb-bootstrap no longer has a build id or a releases-page scraper, because a wheel URL is fully constructible from the version alone. Removed public API: - `PinnedRelease::build_id` and `InstalledHyperd::build_id` — use `.version`, now the only release identifier, plus the new `PinnedRelease::wheel_tag_for(Platform)` for the per-platform wheel tag. - `PinnedRelease::version_tag()` — use `.version`. It existed only to join the version and the build id into `0.0.26479.r96880f6a`; there is no build id left to join. - `VersionSource::ScrapeLatest` and the `scrape` module — deleted with no replacement. With a constructible URL and PyPI-published digests there is nothing left to discover. - `Error::Http`, `Error::HttpStatus` and `Error::ScrapeFailed` — all three existed only to serve the scraper. `Error::MissingWheelTag` is new, because `url::build_download_url` is now fallible. - the `--latest` CLI flag — no replacement, it is deleted along with the scraper it drove. Pass `--version X` or `--version-file PATH` instead. - the `--build-id` CLI flag — it simply goes away. Wheel URLs need no build id, so `--version X` on its own is now a complete version source: it inherits the builtin pin's `[wheel_tag]` values and carries no digests, so the download is unverified and logs a WARN. Use `--version-file` with a full pin when you need verified bytes. - the `regex`, `reqwest` and `rustls` dependencies — no in-process HTTP client remains, which also retires the rustls crypto-provider workaround. --- .agents/skills/update-hyperd-release/SKILL.md | 176 +++++--- .claude/skills/update-hyperd-release/SKILL.md | 176 +++++--- .github/scripts/verify-npm-hyperd-pin.py | 41 +- .github/workflows/npm-build-publish.yml | 46 +- .github/workflows/verify-hyperd-pin.yml | 10 +- AGENTS.md | 12 +- Cargo.lock | 6 +- DEVELOPMENT.md | 22 +- Makefile | 9 +- hyperdb-bootstrap/CHANGELOG.md | 125 +++++- hyperdb-bootstrap/Cargo.toml | 10 +- hyperdb-bootstrap/README.md | 126 ++++-- hyperdb-bootstrap/build.rs | 59 ++- hyperdb-bootstrap/hyperd-version.toml | 46 +- .../src/bin/hyperdb-bootstrap.rs | 109 +++-- hyperdb-bootstrap/src/error.rs | 41 +- hyperdb-bootstrap/src/extract.rs | 184 +++++++- hyperdb-bootstrap/src/install.rs | 42 +- hyperdb-bootstrap/src/lib.rs | 23 +- hyperdb-bootstrap/src/release.rs | 79 ++-- hyperdb-bootstrap/src/scrape.rs | 154 ------- hyperdb-bootstrap/src/url.rs | 137 ++++-- hyperdb-bootstrap/src/verify.rs | 409 ++++++++++++++++-- hyperdb-bootstrap/tests/integration.rs | 103 ++++- 24 files changed, 1450 insertions(+), 695 deletions(-) delete mode 100644 hyperdb-bootstrap/src/scrape.rs diff --git a/.agents/skills/update-hyperd-release/SKILL.md b/.agents/skills/update-hyperd-release/SKILL.md index e19b51fd..fd15183e 100644 --- a/.agents/skills/update-hyperd-release/SKILL.md +++ b/.agents/skills/update-hyperd-release/SKILL.md @@ -1,6 +1,6 @@ --- name: update-hyperd-release -description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version, updating hyperd-version.toml (version + build_id + 4 sha256s), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. +description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version on PyPI, updating hyperd-version.toml (version + 4 wheel tags + 4 sha256s, all read straight off the PyPI JSON API), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. --- # Update the pinned `hyperd` release @@ -12,11 +12,13 @@ gotchas learned in practice. ## Key facts (don't relearn these the hard way) -- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 5. - - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, `build_id`, and four per-platform sha256s. This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. - - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin, used for the `hyperd` bundled into the npm packages. -- **We download the Java bundle, NOT the C++ one.** The C++ `macos-arm64` zip ships an **x86_64** `hyperd` (upstream packaging defect) that only runs under Rosetta on Apple Silicon. The Java `macos-arm64` bundle carries a **native arm64** `hyperd`. Same URL template (only the `java`/`cxx` token differs), same internal layout (`lib/hyper/hyperd`). **Verify this invariant every bump** (step 6) — if a future Java bundle regresses to x86_64, the whole reason for using it is gone. -- **URL template:** `https://downloads.tableau.com/tssoftware/tableauhyperapi-java--release-main...zip` — platforms: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. +- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 4. + - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, four `[wheel_tag]` entries, and four per-platform sha256s. **There is no `build_id` any more.** This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. + - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin (`HYPERD_VERSION`, plus a `hyperd-wheel-tag` / `hyperd-sha256` pair per platform), used for the `hyperd` bundled into the npm packages. +- **We download the PyPI `tableauhyperapi` wheels.** Every input to the URL is either the version you're bumping to or a value already in the pin, so nothing has to be discovered by scraping. The wheel carries `hyperd` at `tableauhyperapi/bin/hyper/hyperd` (`hyperd.exe` plus `crashdumper.exe` on Windows). +- **PyPI publishes a sha256 per file.** You *read* the four digests off the JSON API (step 2) instead of downloading four ~80 MB archives and hashing them by hand. This is the single biggest time saving in the whole procedure. The digests still get committed — a hash in git is an attestation that's independent of the host serving the bytes. +- **URL template:** `https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi--py3-none-.whl` — this legacy path is constructible without an API call and 302-redirects to the content-addressed URL. Platform slugs: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. +- **The wheel tags live in the pin because they are not guaranteed stable.** arm64 wheels only exist from `0.0.19484` onward, and a future macOS floor bump would change `macosx_13_0_arm64`. A wrong tag is a **silent 404**, not a loud error — so keeping the tags as pin data makes any such change a visible diff in the pin file. (Empirically all four tags are unchanged from `0.0.19484` through `0.0.26479`.) - **Crate version is workspace-driven + release-please.** `hyperdb-bootstrap` uses `version.workspace = true`; do **not** hand-edit a crate version. The conventional-commit type drives the release — use `fix(bootstrap): ...` for a routine bump (patch release). - **Never invent `hyperd` flags** (AGENTS.md reminder #9) and **never report tests/benches green without real output** (#10). Tests start a real `hyperd` subprocess; a misconfigured server hangs rather than erroring. @@ -27,71 +29,101 @@ Track these as todos. Each step gates the next. ### 1. Create a branch ```bash -git checkout -b chore/bump-hyperd- # e.g. chore/bump-hyperd-0.0.26225 +git checkout -b chore/bump-hyperd- # e.g. chore/bump-hyperd-0.0.26479 ``` -### 2. Find the latest version + build id +### 2. Read the version and all four digests off PyPI + +Two commands. The first gives you the version to bump to; the second gives you +every wheel for that version with its published sha256, ready to paste into the +toml. ```bash -curl -sL "https://tableau.github.io/hyper-db/docs/releases" | rg -o "0\.0\.[0-9]+" | head -1 -curl -sL "https://tableau.github.io/hyper-db/docs/releases" | \ - rg -o "tableauhyperapi-java-[a-z0-9_-]+-release-main\.\.r[a-z0-9]+\.zip" | sort -u +# Latest version on PyPI +curl -s https://pypi.org/pypi/tableauhyperapi/json | jq -r .info.version + +# Every wheel + its published sha256 for that version — paste straight into the toml +curl -s https://pypi.org/pypi/tableauhyperapi//json \ + | jq -r '.urls[] | "\(.filename) \(.digests.sha256)"' ``` -Confirm all four platform zips are listed for that version and share one build id. +The second command prints four lines. Map filename → platform slug so each digest +lands on the right toml line: + +| Wheel filename suffix | Platform slug | `[wheel_tag]` value | +|--------------------------------|------------------|--------------------------| +| `macosx_13_0_arm64.whl` | `macos-arm64` | `macosx_13_0_arm64` | +| `macosx_10_11_x86_64.whl` | `macos-x86_64` | `macosx_10_11_x86_64` | +| `manylinux2014_x86_64.whl` | `linux-x86_64` | `manylinux2014_x86_64` | +| `win_amd64.whl` | `windows-x86_64` | `win_amd64` | + +**Confirm the four printed filenames still carry exactly those four tags.** A +changed tag is the silent-404 vector: the pin would still compile and `verify` +would be the only thing that catches it. If a tag *has* changed, update the +matching `[wheel_tag]` entry in the same edit as the digests. + +Expect exactly four wheels. If PyPI lists more (or fewer) for the version, stop +and work out why before pinning it. + (The [`hyper-api-release-verify-upcoming-packages`](../hyper-api-release-verify-upcoming-packages/SKILL.md) -skill — bundled `verify_release.py` — validates the whole page's downloadability and -zip integrity for a given `--version`.) +skill — bundled `verify_release.py` — validates downloadability and archive +integrity for a given `--version` if you want a second opinion.) -### 3. Compute the four sha256s +### 3. Edit `hyperd-version.toml` -Download each Java zip and hash it. The values go verbatim into the toml. +Update `version` and all four `[sha256]` entries from the step-2 output; update a +`[wheel_tag]` entry only if step 2 showed the tag changed. Record the **old** +version first — you need it for the A/B benchmark (step 7). -```bash -V=; B=; cd "$(mktemp -d)" -for p in macos-arm64 macos-x86_64 linux-x86_64 windows-x86_64; do - curl -sL --fail -o "$p.zip" \ - "https://downloads.tableau.com/tssoftware/tableauhyperapi-java-$p-release-main.$V.$B.zip" & -done; wait -for p in macos-arm64 macos-x86_64 linux-x86_64 windows-x86_64; do - printf '%-16s ' "$p"; shasum -a 256 "$p.zip" | awk '{print $1}' -done -``` +The file should end up looking like this: -### 4. Edit `hyperd-version.toml` +```toml +version = "0.0.26479" -Update `version`, `build_id`, and all four `[sha256]` entries. Record the **old** -version/build_id first — you need it for the A/B benchmark (step 8). +[wheel_tag] +"macos-arm64" = "macosx_13_0_arm64" +"macos-x86_64" = "macosx_10_11_x86_64" +"linux-x86_64" = "manylinux2014_x86_64" +"windows-x86_64" = "win_amd64" -### 5. Mirror the pin into the npm release workflow +[sha256] +"macos-arm64" = "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" +"macos-x86_64" = "960e276028137847a3870695d9c2d5a1392c173b1e119ff1146d24a75deca71a" +"linux-x86_64" = "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc" +"windows-x86_64" = "7a4f96d2a22351e944fea6db5d03ab5272ad4c0577acc987bfcb3739ed639502" +``` + +### 4. Mirror the pin into the npm release workflow **The step that is easy to miss, and it reddens CI every time it is missed.** [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) bundles `hyperd` into the npm packages from its own hardcoded pin, decoupled -from the toml. Update it in the same commit as step 4 — the two files must +from the toml. Update it in the same commit as step 3 — the two files must never be bumped separately. | Key to update | Where in the workflow (line numbers drift — grep) | |---|---| -| `HYPERD_VERSION` | top-level `env:` block, ~line 26 | -| `HYPERD_BUILD_ID` | top-level `env:` block, ~line 27 | -| `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 99 | -| `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 110 | -| `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 115 | -| `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 105 | +| `HYPERD_VERSION` | top-level `env:` block, ~line 30 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 106 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 119 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 125 | +| `hyperd-wheel-tag` + `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 113 | ```bash -grep -nE "HYPERD_VERSION|HYPERD_BUILD_ID|hyperd-slug|hyperd-sha256" \ +grep -nE "HYPERD_VERSION|hyperd-slug|hyperd-wheel-tag|hyperd-sha256" \ .github/workflows/npm-build-publish.yml ``` -- **The matrix hashes are the same Java-zip sha256s you computed in step 3** — - not hashes of the extracted binary or of some other artifact. The workflow - downloads the identical URL - (`tableauhyperapi-java-${SLUG}-release-main.${HYPERD_VERSION}.${HYPERD_BUILD_ID}.zip`), - and the guard compares each `hyperd-sha256` **directly** against - `[sha256].""` in the toml, so the values are byte-for-byte identical. - Copy them across verbatim. +- **There is no `HYPERD_BUILD_ID` to update.** The workflow builds the wheel URL + from `HYPERD_VERSION` plus the per-platform `hyperd-wheel-tag`, so the build + id the Java-zip pin needed has no counterpart here. +- **The matrix values are the ones you read off PyPI in step 2** — the + `hyperd-sha256`s are the published `.whl` digests, not hashes of the extracted + binary or of some other artifact. The workflow downloads the identical URL + (`tableauhyperapi-${HYPERD_VERSION}-py3-none-${WHEEL_TAG}.whl`), and the guard + compares each `hyperd-sha256` **directly** against `[sha256].""` in the + toml, and each `hyperd-wheel-tag` against `[wheel_tag].""`, so the values + are byte-for-byte identical. Copy them across verbatim. - **`hyperd-slug` is the join key** and it carries the *toml's* platform names (`macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`), not npm's (`darwin-arm64`, `darwin-x64`, …), which live in the sibling `platform:` @@ -119,19 +151,23 @@ was bumped, so npm `0.7.1` shipped with bundled engine `0.0.25080` while crates.io shipped `0.0.26359`. The guard turns that into a red check instead of a mystery bug report months later. -### 6. Verify the pin + the arm64 invariant +### 5. Verify the pin ```bash -make verify-hyperd-pin # all four platforms → HTTP 200 at the new pin +make verify-hyperd-pin # all four platforms → HTTP 200, and each + # pinned sha256 matches the digest PyPI + # publishes for that exact wheel filename make download-hyperd # re-verifies the macos-arm64 sha256 on download -.hyperd/current/hyperd --version # should report main.. -file .hyperd/current/hyperd # MUST say "Mach-O 64-bit executable arm64" on Apple Silicon +.hyperd/current/hyperd --version # should report the new version +file .hyperd/current/hyperd # sanity check: "Mach-O 64-bit executable + # arm64" on Apple Silicon ``` -If `file` reports `x86_64`, **stop** — the Java bundle no longer carries a native -arm64 binary and the bundle choice needs re-evaluation. +`verify` cross-checking the digests, not just HEAD-ing the URLs, is what makes +this step meaningful: it proves the pin names the exact bytes PyPI serves, rather +than merely that the CDN serves *something* at that path. -### 7. Run the full test suite against the NEW engine +### 6. Run the full test suite against the NEW engine Point `HYPERD_PATH` at the freshly downloaded binary — do **not** rely on the workstation default (`~/dev/bin/hyperd`), which may be an old or unversioned build. @@ -146,17 +182,24 @@ Require `failed=0`. Then the pre-commit gate: `cargo fmt --all -- --check` and `cargo clippy --workspace --all-targets --all-features -- -D warnings` (CI's exact clippy command). -### 8. A/B benchmark vs the previous pin +### 7. A/B benchmark vs the previous pin The canonical harness is the **unified suite** ([`hyperdb-api/benches/benchmark_suite.rs`](../../../hyperdb-api/benches/benchmark_suite.rs)). Download the **old** pin into a separate dir, then run the same suite on both. See [docs/BENCHMARK_GUIDE.md](../../../docs/BENCHMARK_GUIDE.md) for the harness details. +`--version ` on its own is enough for the baseline: it inherits the +builtin pin's `[wheel_tag]` values and carries no digests, so the download is +unverified and logs a WARN. That's fine for a throwaway baseline, and the four +tags are unchanged all the way back to `0.0.19484`. (If you ever need a baseline +from a release whose tags *do* differ, write a full pin file and use +`--version-file` instead.) + ```bash # Old engine into a scratch dir (sha256 skipped — that's fine for a throwaway baseline) cargo run --release -p hyperdb-bootstrap --bin hyperdb-bootstrap -- \ - download --version --build-id --dest .hyperd-old + download --version --dest .hyperd-old cargo build -q -p hyperdb-api --release --example benchmark_suite BIN=target/release/examples/benchmark_suite; ROWS=100000000 # 100M for signal over noise @@ -174,35 +217,38 @@ rm -rf .hyperd-old # clean up the scratch baseline (also add to .gitignore if - **Distrust `× 4` / parallel numbers on a laptop.** They throttle thermally — throughput declines monotonically across sequential runs because the machine is hotter for the second engine. Report single-connection deltas as the reliable signal; withhold multi-connection deltas unless run on a cooled/pinned host. - Report throughput as **M rows/s**, not wall time. -### 9. Log the release in the benchmark tracker +### 8. Log the release in the benchmark tracker Append a row per engine to [`docs/hyperd-release-benchmarks.md`](../../../docs/hyperd-release-benchmarks.md) (median single-connection numbers + the machine + the caveat). This builds the per-release history the BENCHMARK_GUIDE's by-platform tables don't capture. -### 10. Changelog +### 9. Changelog Add a `### Changed` bullet under `## [Unreleased]` in [`hyperdb-bootstrap/CHANGELOG.md`](../../../hyperdb-bootstrap/CHANGELOG.md): the new -version/build, "verified native arm64", and the headline performance A/B (with the -thermal caveat on multi-connection numbers). +version, the wheel tags if any of them moved, and the headline performance A/B +(with the thermal caveat on multi-connection numbers). If `## [Unreleased]` +already has a `### Changed`, merge into it — a second sibling heading is +markdownlint MD024. -### 11. Commit + PR +### 10. Commit + PR -- Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to ()`. +- Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to `. - **gh account:** the EMU account (`ssteiner_sfemu`) is Unauthorized on upstream. `gh auth switch --hostname github.com --user StefanSteiner`, then target upstream (it has the CI runners): `gh pr create --repo tableau/hyper-api-rust --base main --head StefanSteiner:`. - Put the verification checklist + performance table in the PR body. ## Verification checklist (what "done" means) -- [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 -- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, `HYPERD_BUILD_ID`, three matrix `hyperd-sha256`s, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 -- [ ] `.hyperd/current/hyperd --version` reports the new version/build -- [ ] `file` confirms macos-arm64 binary is native arm64 +- [ ] Four wheels listed on PyPI for the new version, tags matching the pin +- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, three matrix `hyperd-wheel-tag`/`hyperd-sha256` pairs, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 +- [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 **and** all four digests match PyPI's published sha256 +- [ ] `.hyperd/current/hyperd --version` reports the new version +- [ ] `file` confirms the macos-arm64 binary is native arm64 - [ ] `cargo test --workspace` → `failed=0` against the new engine - [ ] `cargo fmt --check` + CI-exact `cargo clippy` clean - [ ] A/B benchmark done (medians of ≥3 runs @ 100M rows); scratch `.hyperd-old` removed - [ ] Row appended to `docs/hyperd-release-benchmarks.md` -- [ ] CHANGELOG `[Unreleased]` bullet added +- [ ] CHANGELOG `[Unreleased]` bullet added (merged into the existing `### Changed`) - [ ] PR opened against `tableau/hyper-api-rust` from `StefanSteiner:` diff --git a/.claude/skills/update-hyperd-release/SKILL.md b/.claude/skills/update-hyperd-release/SKILL.md index e19b51fd..fd15183e 100644 --- a/.claude/skills/update-hyperd-release/SKILL.md +++ b/.claude/skills/update-hyperd-release/SKILL.md @@ -1,6 +1,6 @@ --- name: update-hyperd-release -description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version, updating hyperd-version.toml (version + build_id + 4 sha256s), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. +description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version on PyPI, updating hyperd-version.toml (version + 4 wheel tags + 4 sha256s, all read straight off the PyPI JSON API), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. --- # Update the pinned `hyperd` release @@ -12,11 +12,13 @@ gotchas learned in practice. ## Key facts (don't relearn these the hard way) -- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 5. - - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, `build_id`, and four per-platform sha256s. This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. - - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin, used for the `hyperd` bundled into the npm packages. -- **We download the Java bundle, NOT the C++ one.** The C++ `macos-arm64` zip ships an **x86_64** `hyperd` (upstream packaging defect) that only runs under Rosetta on Apple Silicon. The Java `macos-arm64` bundle carries a **native arm64** `hyperd`. Same URL template (only the `java`/`cxx` token differs), same internal layout (`lib/hyper/hyperd`). **Verify this invariant every bump** (step 6) — if a future Java bundle regresses to x86_64, the whole reason for using it is gone. -- **URL template:** `https://downloads.tableau.com/tssoftware/tableauhyperapi-java--release-main...zip` — platforms: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. +- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 4. + - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, four `[wheel_tag]` entries, and four per-platform sha256s. **There is no `build_id` any more.** This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. + - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin (`HYPERD_VERSION`, plus a `hyperd-wheel-tag` / `hyperd-sha256` pair per platform), used for the `hyperd` bundled into the npm packages. +- **We download the PyPI `tableauhyperapi` wheels.** Every input to the URL is either the version you're bumping to or a value already in the pin, so nothing has to be discovered by scraping. The wheel carries `hyperd` at `tableauhyperapi/bin/hyper/hyperd` (`hyperd.exe` plus `crashdumper.exe` on Windows). +- **PyPI publishes a sha256 per file.** You *read* the four digests off the JSON API (step 2) instead of downloading four ~80 MB archives and hashing them by hand. This is the single biggest time saving in the whole procedure. The digests still get committed — a hash in git is an attestation that's independent of the host serving the bytes. +- **URL template:** `https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi--py3-none-.whl` — this legacy path is constructible without an API call and 302-redirects to the content-addressed URL. Platform slugs: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. +- **The wheel tags live in the pin because they are not guaranteed stable.** arm64 wheels only exist from `0.0.19484` onward, and a future macOS floor bump would change `macosx_13_0_arm64`. A wrong tag is a **silent 404**, not a loud error — so keeping the tags as pin data makes any such change a visible diff in the pin file. (Empirically all four tags are unchanged from `0.0.19484` through `0.0.26479`.) - **Crate version is workspace-driven + release-please.** `hyperdb-bootstrap` uses `version.workspace = true`; do **not** hand-edit a crate version. The conventional-commit type drives the release — use `fix(bootstrap): ...` for a routine bump (patch release). - **Never invent `hyperd` flags** (AGENTS.md reminder #9) and **never report tests/benches green without real output** (#10). Tests start a real `hyperd` subprocess; a misconfigured server hangs rather than erroring. @@ -27,71 +29,101 @@ Track these as todos. Each step gates the next. ### 1. Create a branch ```bash -git checkout -b chore/bump-hyperd- # e.g. chore/bump-hyperd-0.0.26225 +git checkout -b chore/bump-hyperd- # e.g. chore/bump-hyperd-0.0.26479 ``` -### 2. Find the latest version + build id +### 2. Read the version and all four digests off PyPI + +Two commands. The first gives you the version to bump to; the second gives you +every wheel for that version with its published sha256, ready to paste into the +toml. ```bash -curl -sL "https://tableau.github.io/hyper-db/docs/releases" | rg -o "0\.0\.[0-9]+" | head -1 -curl -sL "https://tableau.github.io/hyper-db/docs/releases" | \ - rg -o "tableauhyperapi-java-[a-z0-9_-]+-release-main\.\.r[a-z0-9]+\.zip" | sort -u +# Latest version on PyPI +curl -s https://pypi.org/pypi/tableauhyperapi/json | jq -r .info.version + +# Every wheel + its published sha256 for that version — paste straight into the toml +curl -s https://pypi.org/pypi/tableauhyperapi//json \ + | jq -r '.urls[] | "\(.filename) \(.digests.sha256)"' ``` -Confirm all four platform zips are listed for that version and share one build id. +The second command prints four lines. Map filename → platform slug so each digest +lands on the right toml line: + +| Wheel filename suffix | Platform slug | `[wheel_tag]` value | +|--------------------------------|------------------|--------------------------| +| `macosx_13_0_arm64.whl` | `macos-arm64` | `macosx_13_0_arm64` | +| `macosx_10_11_x86_64.whl` | `macos-x86_64` | `macosx_10_11_x86_64` | +| `manylinux2014_x86_64.whl` | `linux-x86_64` | `manylinux2014_x86_64` | +| `win_amd64.whl` | `windows-x86_64` | `win_amd64` | + +**Confirm the four printed filenames still carry exactly those four tags.** A +changed tag is the silent-404 vector: the pin would still compile and `verify` +would be the only thing that catches it. If a tag *has* changed, update the +matching `[wheel_tag]` entry in the same edit as the digests. + +Expect exactly four wheels. If PyPI lists more (or fewer) for the version, stop +and work out why before pinning it. + (The [`hyper-api-release-verify-upcoming-packages`](../hyper-api-release-verify-upcoming-packages/SKILL.md) -skill — bundled `verify_release.py` — validates the whole page's downloadability and -zip integrity for a given `--version`.) +skill — bundled `verify_release.py` — validates downloadability and archive +integrity for a given `--version` if you want a second opinion.) -### 3. Compute the four sha256s +### 3. Edit `hyperd-version.toml` -Download each Java zip and hash it. The values go verbatim into the toml. +Update `version` and all four `[sha256]` entries from the step-2 output; update a +`[wheel_tag]` entry only if step 2 showed the tag changed. Record the **old** +version first — you need it for the A/B benchmark (step 7). -```bash -V=; B=; cd "$(mktemp -d)" -for p in macos-arm64 macos-x86_64 linux-x86_64 windows-x86_64; do - curl -sL --fail -o "$p.zip" \ - "https://downloads.tableau.com/tssoftware/tableauhyperapi-java-$p-release-main.$V.$B.zip" & -done; wait -for p in macos-arm64 macos-x86_64 linux-x86_64 windows-x86_64; do - printf '%-16s ' "$p"; shasum -a 256 "$p.zip" | awk '{print $1}' -done -``` +The file should end up looking like this: -### 4. Edit `hyperd-version.toml` +```toml +version = "0.0.26479" -Update `version`, `build_id`, and all four `[sha256]` entries. Record the **old** -version/build_id first — you need it for the A/B benchmark (step 8). +[wheel_tag] +"macos-arm64" = "macosx_13_0_arm64" +"macos-x86_64" = "macosx_10_11_x86_64" +"linux-x86_64" = "manylinux2014_x86_64" +"windows-x86_64" = "win_amd64" -### 5. Mirror the pin into the npm release workflow +[sha256] +"macos-arm64" = "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" +"macos-x86_64" = "960e276028137847a3870695d9c2d5a1392c173b1e119ff1146d24a75deca71a" +"linux-x86_64" = "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc" +"windows-x86_64" = "7a4f96d2a22351e944fea6db5d03ab5272ad4c0577acc987bfcb3739ed639502" +``` + +### 4. Mirror the pin into the npm release workflow **The step that is easy to miss, and it reddens CI every time it is missed.** [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) bundles `hyperd` into the npm packages from its own hardcoded pin, decoupled -from the toml. Update it in the same commit as step 4 — the two files must +from the toml. Update it in the same commit as step 3 — the two files must never be bumped separately. | Key to update | Where in the workflow (line numbers drift — grep) | |---|---| -| `HYPERD_VERSION` | top-level `env:` block, ~line 26 | -| `HYPERD_BUILD_ID` | top-level `env:` block, ~line 27 | -| `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 99 | -| `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 110 | -| `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 115 | -| `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 105 | +| `HYPERD_VERSION` | top-level `env:` block, ~line 30 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 106 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 119 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 125 | +| `hyperd-wheel-tag` + `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 113 | ```bash -grep -nE "HYPERD_VERSION|HYPERD_BUILD_ID|hyperd-slug|hyperd-sha256" \ +grep -nE "HYPERD_VERSION|hyperd-slug|hyperd-wheel-tag|hyperd-sha256" \ .github/workflows/npm-build-publish.yml ``` -- **The matrix hashes are the same Java-zip sha256s you computed in step 3** — - not hashes of the extracted binary or of some other artifact. The workflow - downloads the identical URL - (`tableauhyperapi-java-${SLUG}-release-main.${HYPERD_VERSION}.${HYPERD_BUILD_ID}.zip`), - and the guard compares each `hyperd-sha256` **directly** against - `[sha256].""` in the toml, so the values are byte-for-byte identical. - Copy them across verbatim. +- **There is no `HYPERD_BUILD_ID` to update.** The workflow builds the wheel URL + from `HYPERD_VERSION` plus the per-platform `hyperd-wheel-tag`, so the build + id the Java-zip pin needed has no counterpart here. +- **The matrix values are the ones you read off PyPI in step 2** — the + `hyperd-sha256`s are the published `.whl` digests, not hashes of the extracted + binary or of some other artifact. The workflow downloads the identical URL + (`tableauhyperapi-${HYPERD_VERSION}-py3-none-${WHEEL_TAG}.whl`), and the guard + compares each `hyperd-sha256` **directly** against `[sha256].""` in the + toml, and each `hyperd-wheel-tag` against `[wheel_tag].""`, so the values + are byte-for-byte identical. Copy them across verbatim. - **`hyperd-slug` is the join key** and it carries the *toml's* platform names (`macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`), not npm's (`darwin-arm64`, `darwin-x64`, …), which live in the sibling `platform:` @@ -119,19 +151,23 @@ was bumped, so npm `0.7.1` shipped with bundled engine `0.0.25080` while crates.io shipped `0.0.26359`. The guard turns that into a red check instead of a mystery bug report months later. -### 6. Verify the pin + the arm64 invariant +### 5. Verify the pin ```bash -make verify-hyperd-pin # all four platforms → HTTP 200 at the new pin +make verify-hyperd-pin # all four platforms → HTTP 200, and each + # pinned sha256 matches the digest PyPI + # publishes for that exact wheel filename make download-hyperd # re-verifies the macos-arm64 sha256 on download -.hyperd/current/hyperd --version # should report main.. -file .hyperd/current/hyperd # MUST say "Mach-O 64-bit executable arm64" on Apple Silicon +.hyperd/current/hyperd --version # should report the new version +file .hyperd/current/hyperd # sanity check: "Mach-O 64-bit executable + # arm64" on Apple Silicon ``` -If `file` reports `x86_64`, **stop** — the Java bundle no longer carries a native -arm64 binary and the bundle choice needs re-evaluation. +`verify` cross-checking the digests, not just HEAD-ing the URLs, is what makes +this step meaningful: it proves the pin names the exact bytes PyPI serves, rather +than merely that the CDN serves *something* at that path. -### 7. Run the full test suite against the NEW engine +### 6. Run the full test suite against the NEW engine Point `HYPERD_PATH` at the freshly downloaded binary — do **not** rely on the workstation default (`~/dev/bin/hyperd`), which may be an old or unversioned build. @@ -146,17 +182,24 @@ Require `failed=0`. Then the pre-commit gate: `cargo fmt --all -- --check` and `cargo clippy --workspace --all-targets --all-features -- -D warnings` (CI's exact clippy command). -### 8. A/B benchmark vs the previous pin +### 7. A/B benchmark vs the previous pin The canonical harness is the **unified suite** ([`hyperdb-api/benches/benchmark_suite.rs`](../../../hyperdb-api/benches/benchmark_suite.rs)). Download the **old** pin into a separate dir, then run the same suite on both. See [docs/BENCHMARK_GUIDE.md](../../../docs/BENCHMARK_GUIDE.md) for the harness details. +`--version ` on its own is enough for the baseline: it inherits the +builtin pin's `[wheel_tag]` values and carries no digests, so the download is +unverified and logs a WARN. That's fine for a throwaway baseline, and the four +tags are unchanged all the way back to `0.0.19484`. (If you ever need a baseline +from a release whose tags *do* differ, write a full pin file and use +`--version-file` instead.) + ```bash # Old engine into a scratch dir (sha256 skipped — that's fine for a throwaway baseline) cargo run --release -p hyperdb-bootstrap --bin hyperdb-bootstrap -- \ - download --version --build-id --dest .hyperd-old + download --version --dest .hyperd-old cargo build -q -p hyperdb-api --release --example benchmark_suite BIN=target/release/examples/benchmark_suite; ROWS=100000000 # 100M for signal over noise @@ -174,35 +217,38 @@ rm -rf .hyperd-old # clean up the scratch baseline (also add to .gitignore if - **Distrust `× 4` / parallel numbers on a laptop.** They throttle thermally — throughput declines monotonically across sequential runs because the machine is hotter for the second engine. Report single-connection deltas as the reliable signal; withhold multi-connection deltas unless run on a cooled/pinned host. - Report throughput as **M rows/s**, not wall time. -### 9. Log the release in the benchmark tracker +### 8. Log the release in the benchmark tracker Append a row per engine to [`docs/hyperd-release-benchmarks.md`](../../../docs/hyperd-release-benchmarks.md) (median single-connection numbers + the machine + the caveat). This builds the per-release history the BENCHMARK_GUIDE's by-platform tables don't capture. -### 10. Changelog +### 9. Changelog Add a `### Changed` bullet under `## [Unreleased]` in [`hyperdb-bootstrap/CHANGELOG.md`](../../../hyperdb-bootstrap/CHANGELOG.md): the new -version/build, "verified native arm64", and the headline performance A/B (with the -thermal caveat on multi-connection numbers). +version, the wheel tags if any of them moved, and the headline performance A/B +(with the thermal caveat on multi-connection numbers). If `## [Unreleased]` +already has a `### Changed`, merge into it — a second sibling heading is +markdownlint MD024. -### 11. Commit + PR +### 10. Commit + PR -- Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to ()`. +- Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to `. - **gh account:** the EMU account (`ssteiner_sfemu`) is Unauthorized on upstream. `gh auth switch --hostname github.com --user StefanSteiner`, then target upstream (it has the CI runners): `gh pr create --repo tableau/hyper-api-rust --base main --head StefanSteiner:`. - Put the verification checklist + performance table in the PR body. ## Verification checklist (what "done" means) -- [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 -- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, `HYPERD_BUILD_ID`, three matrix `hyperd-sha256`s, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 -- [ ] `.hyperd/current/hyperd --version` reports the new version/build -- [ ] `file` confirms macos-arm64 binary is native arm64 +- [ ] Four wheels listed on PyPI for the new version, tags matching the pin +- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, three matrix `hyperd-wheel-tag`/`hyperd-sha256` pairs, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 +- [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 **and** all four digests match PyPI's published sha256 +- [ ] `.hyperd/current/hyperd --version` reports the new version +- [ ] `file` confirms the macos-arm64 binary is native arm64 - [ ] `cargo test --workspace` → `failed=0` against the new engine - [ ] `cargo fmt --check` + CI-exact `cargo clippy` clean - [ ] A/B benchmark done (medians of ≥3 runs @ 100M rows); scratch `.hyperd-old` removed - [ ] Row appended to `docs/hyperd-release-benchmarks.md` -- [ ] CHANGELOG `[Unreleased]` bullet added +- [ ] CHANGELOG `[Unreleased]` bullet added (merged into the existing `### Changed`) - [ ] PR opened against `tableau/hyper-api-rust` from `StefanSteiner:` diff --git a/.github/scripts/verify-npm-hyperd-pin.py b/.github/scripts/verify-npm-hyperd-pin.py index 2803ec1d..864d6bcc 100644 --- a/.github/scripts/verify-npm-hyperd-pin.py +++ b/.github/scripts/verify-npm-hyperd-pin.py @@ -2,8 +2,9 @@ """Guard: keep the npm release workflow's hyperd pin in sync with the toml. `.github/workflows/npm-build-publish.yml` bundles `hyperd` into the npm -packages using its OWN hardcoded `HYPERD_VERSION` / `HYPERD_BUILD_ID` / -per-platform matrix `hyperd-sha256`s. Those are decoupled from +packages from the PyPI `tableauhyperapi` wheel, using its OWN hardcoded +`HYPERD_VERSION` plus per-platform matrix `hyperd-wheel-tag`s and +`hyperd-sha256`s. Those are decoupled from `hyperdb-bootstrap/hyperd-version.toml`, which is what `make download-hyperd` and the crates.io path use. @@ -12,10 +13,16 @@ This script fails CI whenever the two drift, so that can't recur silently. The platform slug (`macos-arm64`, `linux-x86_64`, `windows-x86_64`) is the join -key: it is identical between the toml's `[sha256]` table and the workflow's -`hyperd-slug` matrix field. Only slugs the workflow actually builds are checked, -so a commented-out matrix entry (invisible to the YAML parser) and any unused -extra toml sha are both fine. +key: it is identical between the toml's `[wheel_tag]` / `[sha256]` tables and +the workflow's `hyperd-slug` matrix field. Only slugs the workflow actually +builds are checked, so a commented-out matrix entry (invisible to the YAML +parser) and any unused extra toml entry are both fine. + +The wheel tag (`macosx_13_0_arm64`, `manylinux2014_x86_64`, ...) is checked +alongside the digest because it is a drift vector of its own: it is not +derivable from the version, and a wrong tag resolves to a URL that does not +exist, so it would surface as an opaque 404 mid-release rather than as a +mismatch. The sha256s are digests of the downloaded `.whl` archive. """ from __future__ import annotations @@ -36,12 +43,12 @@ def main() -> int: workflow = yaml.safe_load(WORKFLOW.read_text()) env = workflow.get("env", {}) + toml_wheel_tag = toml_data.get("wheel_tag", {}) toml_sha = toml_data.get("sha256", {}) # (label, expected-from-toml, actual-from-workflow) checks: list[tuple[str, str, str | None]] = [ ("HYPERD_VERSION", str(toml_data["version"]), env.get("HYPERD_VERSION")), - ("HYPERD_BUILD_ID", str(toml_data["build_id"]), env.get("HYPERD_BUILD_ID")), ] errors: list[str] = [] @@ -51,13 +58,22 @@ def main() -> int: slug = entry.get("hyperd-slug") if slug is None: continue - expected = toml_sha.get(slug) - if expected is None: + expected_tag = toml_wheel_tag.get(slug) + if expected_tag is None: + errors.append( + f'matrix slug "{slug}" has no [wheel_tag]."{slug}" entry in {TOML.name}' + ) + else: + checks.append( + (f"wheel_tag[{slug}]", expected_tag, entry.get("hyperd-wheel-tag")) + ) + expected_sha = toml_sha.get(slug) + if expected_sha is None: errors.append( f'matrix slug "{slug}" has no [sha256]."{slug}" entry in {TOML.name}' ) continue - checks.append((f"sha256[{slug}]", expected, entry.get("hyperd-sha256"))) + checks.append((f"sha256[{slug}]", expected_sha, entry.get("hyperd-sha256"))) for label, expected, actual in checks: if actual == expected: @@ -72,8 +88,9 @@ def main() -> int: sys.stdout.flush() print( f"\n{WORKFLOW.name} is out of sync with {TOML.name}. " - "Update the workflow's env vars and matrix sha256s to match the toml " - "(or vice versa) so npm bundles the same hyperd as crates.io.", + "Update the workflow's env vars and matrix wheel tags / sha256s to " + "match the toml (or vice versa) so npm bundles the same hyperd as " + "crates.io.", file=sys.stderr, ) return 1 diff --git a/.github/workflows/npm-build-publish.yml b/.github/workflows/npm-build-publish.yml index 15897370..6ab14573 100644 --- a/.github/workflows/npm-build-publish.yml +++ b/.github/workflows/npm-build-publish.yml @@ -23,8 +23,11 @@ permissions: env: CARGO_TERM_COLOR: always + # hyperd is bundled from the PyPI `tableauhyperapi` wheel. This version and + # the per-platform wheel tags / sha256s in the build-npm matrix below must + # match hyperdb-bootstrap/hyperd-version.toml — enforced by + # .github/scripts/verify-npm-hyperd-pin.py. HYPERD_VERSION: "0.0.26479" - HYPERD_BUILD_ID: "r96880f6a" jobs: verify-ci: @@ -92,27 +95,35 @@ jobs: fail-fast: false matrix: include: + # hyperd-wheel-tag is the PyPI platform tag for this slug. It is NOT + # derivable from HYPERD_VERSION, so it has to be carried explicitly; + # a wrong tag yields a 404 rather than a version mismatch. + # hyperd-sha256 is the digest of the downloaded .whl archive. - platform: darwin-arm64 os: macos-14 target: aarch64-apple-darwin hyperd-slug: macos-arm64 - hyperd-sha256: "65bd021b3d3470ac74728ec287866a3ee0dfd806daf2589670feb3580955ee95" + hyperd-wheel-tag: macosx_13_0_arm64 + hyperd-sha256: "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" # TODO: re-enable when macos-13 runners are more available # - platform: darwin-x64 # os: macos-13 # target: x86_64-apple-darwin # hyperd-slug: macos-x86_64 - # hyperd-sha256: "6690669c8a6a6c7c6794c101beb31b83e1589d76cb14b0c817523069591f694c" + # hyperd-wheel-tag: macosx_10_11_x86_64 + # hyperd-sha256: "960e276028137847a3870695d9c2d5a1392c173b1e119ff1146d24a75deca71a" - platform: linux-x64-gnu os: ubuntu-latest target: x86_64-unknown-linux-gnu hyperd-slug: linux-x86_64 - hyperd-sha256: "c20be5b6874d319c7db01dcec763d7e65ae2483c0becc75f2914a58accc4f932" + hyperd-wheel-tag: manylinux2014_x86_64 + hyperd-sha256: "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc" - platform: win32-x64-msvc os: windows-latest target: x86_64-pc-windows-msvc hyperd-slug: windows-x86_64 - hyperd-sha256: "3a400508e79c67ce9dcd8bd317ab164a61b5aed8ca880031ed4bbe6621514e1d" + hyperd-wheel-tag: win_amd64 + hyperd-sha256: "7a4f96d2a22351e944fea6db5d03ab5272ad4c0577acc987bfcb3739ed639502" runs-on: ${{ matrix.os }} defaults: run: @@ -154,16 +165,20 @@ jobs: - name: Download and verify hyperd env: SLUG: ${{ matrix.hyperd-slug }} + WHEEL_TAG: ${{ matrix.hyperd-wheel-tag }} EXPECTED_SHA256: ${{ matrix.hyperd-sha256 }} run: | set -euo pipefail - URL="https://downloads.tableau.com/tssoftware/tableauhyperapi-java-${SLUG}-release-main.${HYPERD_VERSION}.${HYPERD_BUILD_ID}.zip" - echo "Downloading: $URL" - curl --fail --silent --show-error --location --output hyperd-archive.zip "$URL" + # The engine ships inside the PyPI `tableauhyperapi` wheel, which is + # a zip. This URL 302-redirects to the content-addressed download. + URL="https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-${HYPERD_VERSION}-py3-none-${WHEEL_TAG}.whl" + echo "Downloading hyperd for ${SLUG}: $URL" + curl --fail --silent --show-error --location --output hyperd-archive.whl "$URL" + # Digest is over the downloaded archive, not the extracted binary. if command -v sha256sum &>/dev/null; then - ACTUAL_SHA256=$(sha256sum hyperd-archive.zip | awk '{print $1}') + ACTUAL_SHA256=$(sha256sum hyperd-archive.whl | awk '{print $1}') else - ACTUAL_SHA256=$(shasum -a 256 hyperd-archive.zip | awk '{print $1}') + ACTUAL_SHA256=$(shasum -a 256 hyperd-archive.whl | awk '{print $1}') fi if [[ "$ACTUAL_SHA256" != "$EXPECTED_SHA256" ]]; then echo "::error::SHA256 mismatch: expected $EXPECTED_SHA256, got $ACTUAL_SHA256" @@ -175,8 +190,11 @@ jobs: run: | set -euo pipefail mkdir -p hyperd-extracted - unzip -q hyperd-archive.zip -d hyperd-raw - # Find the lib/hyper or bin/hyper directory (matches extract.rs logic) + unzip -q hyperd-archive.whl -d hyperd-raw + # Find the lib/hyper or bin/hyper directory (matches extract.rs + # logic). The wheel lays the engine out at tableauhyperapi/bin/hyper/ + # on every platform, so in practice the bin/hyper branch matches; + # the lib/hyper branch is kept for parity with extract.rs. HYPER_DIR=$(find hyperd-raw -type d -name "hyper" -path "*/lib/hyper" -o -type d -name "hyper" -path "*/bin/hyper" | head -1) if [[ -z "$HYPER_DIR" ]]; then echo "::error::Could not find lib/hyper or bin/hyper in archive" @@ -195,6 +213,10 @@ jobs: - name: Extract LICENSE from hyperd archive run: | set -euo pipefail + # The wheel's tableauhyperapi-.dist-info/ carries LICENSE, + # NOTICES.txt and HYPER_API_OSS_disclosure.txt, so this glob matches. + # The heredoc below is the fallback if that ever changes, so a miss + # is non-fatal. LICENSE_FILE=$(find hyperd-raw -iname "LICENSE*" -o -iname "NOTICE*" | head -1) if [[ -n "$LICENSE_FILE" ]]; then cp "$LICENSE_FILE" LICENSE-HYPERD diff --git a/.github/workflows/verify-hyperd-pin.yml b/.github/workflows/verify-hyperd-pin.yml index 55ac3968..7086c466 100644 --- a/.github/workflows/verify-hyperd-pin.yml +++ b/.github/workflows/verify-hyperd-pin.yml @@ -1,8 +1,9 @@ name: verify-hyperd-pin -# HEAD every platform URL for the pinned Hyper release. Catches -# Tableau yanking / renaming an archive, and catches typos in -# hyperd-version.toml on PRs that touch the pin. +# HEAD every platform URL for the pinned Hyper release. The archives are the +# PyPI `tableauhyperapi` wheels, so this catches a wheel being yanked and +# catches typos in hyperd-version.toml (version or wheel tag — a wrong tag is +# just a URL that 404s) on PRs that touch the pin. on: push: @@ -46,7 +47,8 @@ jobs: - name: Verify npm-build-publish.yml hyperd pin matches the toml # The release workflow bundles hyperd into the npm packages from its - # OWN hardcoded version/build_id/sha256s, decoupled from the toml. + # OWN hardcoded version + per-platform wheel tags/sha256s, decoupled + # from the toml. # 0.7.1 shipped npm with the stale 0.0.25080 engine because only the # toml was bumped. This guard fails the build if they ever drift again. run: python3 .github/scripts/verify-npm-hyperd-pin.py diff --git a/AGENTS.md b/AGENTS.md index 644ced28..2d816a2d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,10 +9,14 @@ running `make download-hyperd` (or `.\build.ps1 download-hyperd`). The implementation lives in the [`hyperdb-bootstrap`](hyperdb-bootstrap/) crate; the pinned release is baked into [`hyperdb-bootstrap/hyperd-version.toml`](hyperdb-bootstrap/hyperd-version.toml). -Bumping `hyperd` = edit that file (version + build_id + per-platform sha256s), -then let the `fix(bootstrap):` commit drive the version via release-please (the -crate uses `version.workspace = true` — don't hand-edit a crate version). The -full repeatable procedure — verify the pin, run the suite, A/B benchmark +Bumping `hyperd` = edit that file (`version` + the four `[wheel_tag]` entries + +the four per-platform sha256s — there is no `build_id`), then let the +`fix(bootstrap):` commit drive the version via release-please (the crate uses +`version.workspace = true` — don't hand-edit a crate version). `hyperd` comes +out of the PyPI `tableauhyperapi` wheels, so **you don't compute the digests**: +read them straight off the JSON API with +`curl -s https://pypi.org/pypi/tableauhyperapi//json | jq -r '.urls[] | "\(.filename) \(.digests.sha256)"'`. +The full repeatable procedure — verify the pin, run the suite, A/B benchmark against the previous pin, and log the result — is captured in the [`update-hyperd-release`](.claude/skills/update-hyperd-release/SKILL.md) skill. diff --git a/Cargo.lock b/Cargo.lock index 951a2ea2..04bfe7b6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1893,10 +1893,8 @@ version = "1.0.0-rc.1" dependencies = [ "anyhow", "clap", - "regex", - "reqwest", - "rustls", "serde", + "serde_json", "sha2 0.11.0", "tempfile", "thiserror 2.0.20", @@ -3278,9 +3276,7 @@ checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" dependencies = [ "base64 0.22.1", "bytes", - "futures-channel", "futures-core", - "futures-util", "http", "http-body", "http-body-util", diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 0f83bc84..f1c59389 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -384,10 +384,9 @@ in companion crates: This installs `hyperd` at `.hyperd/current/hyperd` (auto-discovered by `make`/`build.ps1` — no `HYPERD_PATH` needed). The pinned release is baked into [`hyperdb-bootstrap/hyperd-version.toml`](hyperdb-bootstrap/hyperd-version.toml); - to upgrade, edit that file (version + build_id + per-platform sha256s) - and bump the crate version. Pass `ARGS="--latest"` to fetch the newest - release via best-effort scraping, or `ARGS="--version X --build-id Y"` - for an ad-hoc pin. + to upgrade, edit that file (version + the four `[wheel_tag]` entries + + per-platform sha256s) and bump the crate version. Pass + `ARGS="--version X"` for an ad-hoc pin. If you already have a `hyperd` elsewhere, set `HYPERD_PATH` instead: @@ -420,11 +419,9 @@ in companion crates: after the `build.ps1 download-hyperd` command): ```bash - # Best-effort scrape of the latest release (skips sha256 verification). - make download-hyperd ARGS="--latest" - - # Pin to a specific release ad-hoc. - make download-hyperd ARGS="--version 0.0.24457 --build-id rc36858b6" + # Pin to a specific release ad-hoc. Inherits the builtin pin's wheel tags + # and carries no digests, so the download is unverified (logs a WARN). + make download-hyperd ARGS="--version 0.0.26359" # Install to a custom location, e.g. shared across repos. make download-hyperd ARGS="--dest /opt/hyperd" @@ -435,9 +432,10 @@ in companion crates: Bumping the baked-in pin is an edit to [`hyperdb-bootstrap/hyperd-version.toml`](hyperdb-bootstrap/hyperd-version.toml) - (version + build_id + per-platform sha256s) plus a crate version bump. - `build.rs` validates the file on every compile, and the - `verify-hyperd-pin` CI workflow confirms the URLs resolve. + (version + the four `[wheel_tag]` entries + per-platform sha256s, all read + off the PyPI JSON API) plus a crate version bump. `build.rs` validates the + file on every compile, and the `verify-hyperd-pin` CI workflow confirms the + URLs resolve and that each pinned digest matches the one PyPI publishes. 4. **Windows only**: Install Visual Studio Build Tools with "Desktop development with C++" workload (provides the MSVC linker, not for C++ compilation). diff --git a/Makefile b/Makefile index dc547842..a6f3e954 100644 --- a/Makefile +++ b/Makefile @@ -141,13 +141,14 @@ test-api-release: examples: ./run_all_examples.sh -# Download hyperd from Tableau's Hyper C++ API release into .hyperd/current/ -# Forward extra flags via ARGS, e.g. `make download-hyperd ARGS="--latest"`. +# Download hyperd from the PyPI tableauhyperapi wheel into .hyperd/current/ +# Forward extra flags via ARGS, e.g. `make download-hyperd ARGS="--force"`. download-hyperd: cargo run --release -p hyperdb-bootstrap --bin hyperdb-bootstrap -- download $(ARGS) -# Network-only check: HEAD each supported platform URL for the pinned -# release. Intended for CI (nightly + on PRs touching hyperd-version.toml). +# Network-only check: probe each supported platform's wheel URL for the pinned +# release and cross-check its digest against PyPI. Intended for CI (nightly + +# on PRs touching hyperd-version.toml). verify-hyperd-pin: cargo run --release -p hyperdb-bootstrap --bin hyperdb-bootstrap -- verify $(ARGS) diff --git a/hyperdb-bootstrap/CHANGELOG.md b/hyperdb-bootstrap/CHANGELOG.md index 61cb6d64..e1c317aa 100644 --- a/hyperdb-bootstrap/CHANGELOG.md +++ b/hyperdb-bootstrap/CHANGELOG.md @@ -9,6 +9,56 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **BREAKING: `hyperd` is now downloaded from the PyPI `tableauhyperapi` + wheels instead of Tableau's Hyper Java API zips**, and the pin moves to + `0.0.26479`. The motivation is that a wheel URL is fully constructible from + the version plus the platform's wheel tag: + + ```text + https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-{version}-py3-none-{wheel_tag}.whl + ``` + + whereas the Tableau zip filenames embed an opaque `build_id` (`r07abb490`) + that cannot be derived from the version — which is why bumping the pin + used to require scraping an HTML page. And because **PyPI publishes a + sha256 per file**, the four pinned digests are now read off + `https://pypi.org/pypi/tableauhyperapi//json` rather than produced + by downloading four ~80 MB archives and hashing them by hand. The digests + are still committed: a hash in git is an attestation independent of the + host serving the bytes. + + **The bytes are unchanged.** The `hyperd` inside the `macosx_13_0_arm64` + wheel is bit-identical to the one extracted from the corresponding Java + zip — sha256 + `aef5c81970bb4d84d06fb9513d5ffd722526fce779632a0c5f63d87b6450e478`, + 277,836,448 bytes. Same build, different envelope; the wheels are ~3.6–4.5% + smaller. Both binaries report `minos 13.0`, so the `macosx_13_0` wheel tag + is **not** a raised support floor and no contributor loses support. + + **BREAKING** consequences for the pin format and the public API: + + - `hyperd-version.toml` now holds `version`, a `[wheel_tag]` table, and + `[sha256]`. **`build_id` is gone.** The wheel tags are pin data rather + than Rust constants because they are not stable across releases (arm64 + wheels only exist from `0.0.19484`; a future macOS floor bump would change + `macosx_13_0_arm64`) and a wrong tag yields a *silent 404* — keeping them + in the pin makes any such change a visible pin edit. + - `build.rs` now validates `version`, the `[wheel_tag]` entries, and the + sha256 shapes; it no longer validates `build_id`. + - The install layout is keyed on the version alone: the versioned cache + directory is `/0.0.26479/` (was `/0.0.26479.r96880f6a/`) and + `current/VERSION` contains just `0.0.26479`. + - `hyperd` is extracted from `tableauhyperapi/bin/hyper/` inside the wheel + rather than `lib/hyper/` inside the zip. + - New `PinnedRelease::wheel_tag_for(Platform) -> Option<&str>`. + - `url::build_download_url` is now **fallible**, returning + `Result` — a platform with no pinned wheel tag is an error + (`Error::MissingWheelTag`, a new variant). + - `verify` now additionally cross-checks every pinned sha256 against the + digest PyPI publishes for that exact wheel filename, on top of HEAD-ing + the four download URLs. It therefore validates the exact pinned bytes + rather than merely that the CDN serves *something* at that path. + - **BREAKING:** the minimum supported Rust version is now **1.88**, up from 1.81, and the crate is compiled with **edition 2024**. 1.88 is the version Red Hat Enterprise Linux 9.7 ships as `rust-toolset`. @@ -28,6 +78,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/). Guarded by a unit test that builds a `reqwest` client, which needs no network because the failure is a panic inside `build()`. + **This entire workaround is retired by the PyPI migration above**, which + drops `reqwest` and `rustls` from the crate: there is no in-process HTTP + client left, so there is no crypto provider to select. `ensure_crypto_provider`, + the `rustls-no-provider` feature, and the ring pin are all gone, and the + feature-unification pressure this bullet describes no longer exists for the + rest of the workspace. + - **Bump the pinned `hyperd` release to `0.0.26359` (`r07abb490`).** This supersedes the never-shipped `0.0.26225` bump attempt (PR #219), which was held because `0.0.26225` deadlocked on Apple Silicon Macs running macOS @@ -43,13 +100,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). thermally on a laptop. Verified native arm64 and all 1485 workspace tests pass; the macOS-14 CI runner is green (the deadlock is gone). -- **Bump the pinned `hyperd` release to `0.0.26479` (`r96880f6a`).** Updates the - version, build id, and all four per-platform sha256s in `hyperd-version.toml`. - **Verified native arm64** — `file` reports `Mach-O 64-bit executable arm64` - for the `macos-arm64` bundle's `lib/hyper/hyperd`, so the reason this crate - pulls the Java bundle rather than the C++ one still holds. All four platform - URLs return HTTP 200 at the new pin, and all **1586** workspace tests pass - against the new engine. Performance (interleaved same-session A/B vs the +- **Bump the pinned `hyperd` release to `0.0.26479` (engine build + `r96880f6a`).** The engine bump rides with the PyPI migration above, which is + what now carries the pin. **Verified native arm64** — `file` reports `Mach-O + 64-bit executable arm64` for the extracted `macos-arm64` binary. All four + platform URLs return HTTP 200 at the new pin, and the workspace test suite + passes against the new engine. Performance (interleaved same-session A/B vs the previous `0.0.26359` pin, medians of 5 runs per engine at 100M rows, single-connection): the async Arrow insert path **more than doubles** (`AsyncArrowInserter` **+127%**, 30.4 → 68.9 M rows/s), reproduced as **+75%** @@ -61,6 +117,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/). 17–41% run to run, enough to have read **+23% at 100M and −20% at 10M for the same workload in the same session**. +### Removed + +Everything here follows from the move to PyPI wheels described under +**Changed**: the build id no longer exists as a concept, and nothing needs to +be discovered by scraping. + +- **BREAKING: the `--build-id` CLI flag is removed.** `--version X` on its own + is now a complete version source: it inherits the builtin pin's + `[wheel_tag]` values and carries no digests, so the download is unverified + and logs a WARN. The four wheel tags are unchanged from `0.0.19484` through + `0.0.26479`, so this covers any realistic ad-hoc pin or A/B baseline; for a + release whose tags differ, use `--version-file` with a full pin. +- **BREAKING: the `--latest` CLI flag, the `VersionSource::ScrapeLatest` + variant, and the whole `scrape` module are removed.** The scraper had been + broken for three-plus releases without anyone noticing, because its tests + passed against a synthetic fixture rather than the live page. It had two + independent defects: the heading regex expected `

VERSION [DATE]

`, + but Docusaurus renders `0.0.26479 [September 3 2026]`, which `\s*` + cannot span; and the build-id capture hardcoded `(rc[a-z0-9]+)` while every + build id since `0.0.24457` has been `r` followed by hex. It is deleted + rather than fixed — with a constructible URL and published digests, there is + nothing left for it to do. Version-source precedence is now, highest to + lowest: `--version X`, `--version-file PATH`, an auto-discovered + `./hyperd-version.toml`, then the compiled-in default. +- **BREAKING: `PinnedRelease::build_id` and `InstalledHyperd::build_id` are + removed**, as is `PinnedRelease::version_tag()`. Use `.version`, which is + now the only release identifier, and `PinnedRelease::wheel_tag_for()` for + the per-platform tag. +- **BREAKING: the `Error::Http`, `Error::HttpStatus`, and `Error::ScrapeFailed` + variants are removed.** All three existed only to serve the scraper. +- **BREAKING: the `regex`, `reqwest`, and `rustls` dependencies are dropped + entirely.** `download.rs` and `verify.rs` already shell out to `curl` — + deliberately, because Akamai bot protection blocked non-browser TLS stacks + from GitHub runner IPs — so removing the scraper leaves no in-process HTTP + client behind. + ### Fixed - The "no hyperd installed" error suggested `hyperd-bootstrap download`, but @@ -68,17 +160,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/). Doc comments across the crate carried the same pre-rename name, as did the `hyperdb-bootstrap/hyperd-version.toml` path in `release.rs`. -- **Download `hyperd` from the Java API bundle instead of the C++ bundle.** - Tableau's C++ `macos-arm64` zip ships an **x86_64** `hyperd` (an upstream - packaging defect), so on Apple Silicon the extracted `hyperd` only ran - under Rosetta. The Java `macos-arm64` bundle carries a native arm64 - `hyperd`. The bundles share an identical URL template (only the - `java`/`cxx` token differs) and an identical internal layout - (`lib/hyper/hyperd`), so the switch is confined to the URL token and the - pinned per-platform sha256s in `hyperd-version.toml`. The other three - platforms (`macos-x86_64`, `linux-x86_64`, `windows-x86_64`) are - unaffected in architecture but now also come from the Java bundle for - consistency. +- **The extracted macOS arm64 `hyperd` is a native arm64 binary.** Earlier + releases of Tableau's C++ `macos-arm64` zip shipped an **x86_64** `hyperd` + (an upstream packaging defect), so on Apple Silicon the extracted binary + only ran under Rosetta; this crate switched to the Java bundle to get a + native one. Tableau fixed the C++ packaging in `0.0.26225`, and from that + release the C++ and Java binaries are byte-identical, so the bundle choice + stopped mattering — and the move to PyPI wheels under **Changed** supersedes + it entirely. The wheels carry the same native arm64 build. ## [0.1.1] - 2026-05-13 diff --git a/hyperdb-bootstrap/Cargo.toml b/hyperdb-bootstrap/Cargo.toml index 3cfa5ced..1119bc38 100644 --- a/hyperdb-bootstrap/Cargo.toml +++ b/hyperdb-bootstrap/Cargo.toml @@ -3,7 +3,7 @@ name = "hyperdb-bootstrap" version.workspace = true edition.workspace = true rust-version.workspace = true -description = "Download and install the hyperd executable from Tableau's Hyper C++ API release packages" +description = "Download and install the hyperd executable from the PyPI tableauhyperapi wheels" license.workspace = true repository.workspace = true homepage.workspace = true @@ -24,16 +24,14 @@ default = ["cli"] cli = ["dep:clap", "dep:anyhow", "dep:tracing-subscriber"] [dependencies] -reqwest = { version = "0.13", default-features = false, features = ["blocking", "rustls-no-provider"] } -# `rustls-no-provider` above ships no crypto provider, so we install `ring` -# ourselves before building a client. See `scrape::ensure_crypto_provider`. -rustls = { workspace = true } +# No in-process HTTP client: `download.rs` and `verify.rs` shell out to `curl`, +# so this crate needs no TLS stack (and no rustls crypto-provider dance). zip = { workspace = true } sha2 = { workspace = true } serde = { workspace = true } +serde_json = { workspace = true } toml = { workspace = true } thiserror = { workspace = true } -regex = { workspace = true } tempfile = { workspace = true } tracing = { workspace = true } clap = { workspace = true, optional = true } diff --git a/hyperdb-bootstrap/README.md b/hyperdb-bootstrap/README.md index 9bd9bd29..1bfc0c94 100644 --- a/hyperdb-bootstrap/README.md +++ b/hyperdb-bootstrap/README.md @@ -1,22 +1,42 @@ # hyperdb-bootstrap -Download and install the `hyperd` executable from Tableau's Hyper Java API -release packages. Ships as both a CLI binary and a library. - -The `hyperd` server isn't on crates.io — it's a prebuilt binary distributed -inside Tableau's Hyper API zips at -. This crate automates -the "download the right zip for your platform, extract `hyperd` out of -`lib/hyper/`, put it somewhere useful" step so contributors and CI can -bootstrap with a single command. - -> **Why the Java bundle, not C++?** Tableau publishes `hyperd` inside both -> the C++ and Java API zips. The C++ `macos-arm64` zip currently ships an -> **x86_64** `hyperd` (an upstream packaging defect), so on Apple Silicon it -> would only run under Rosetta. The Java `macos-arm64` zip carries a native -> arm64 `hyperd`. The two bundles are otherwise identical for our purposes -> (same URL template, same `lib/hyper/hyperd` layout), so this crate pulls -> from the Java bundle on every platform for consistency. +Download and install the `hyperd` executable from the PyPI `tableauhyperapi` +wheels. Ships as both a CLI binary and a library. + +The `hyperd` server isn't on crates.io — it's a prebuilt binary that Tableau +distributes inside its Hyper API packages. This crate takes the PyPI ones: +each `tableauhyperapi` wheel carries `hyperd` at +`tableauhyperapi/bin/hyper/hyperd` (`hyperd.exe` plus `crashdumper.exe` on +Windows). This crate automates the "download the right wheel for your +platform, extract `hyperd` out of it, put it somewhere useful" step so +contributors and CI can bootstrap with a single command. + +> **Why the PyPI wheels?** Two properties that no other distribution +> channel offers together. First, **the download URL is constructible** from +> nothing but the version and the platform's wheel tag: +> +> ```text +> https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-{version}-py3-none-{wheel_tag}.whl +> ``` +> +> Tableau's own download filenames embed an opaque build id (`r07abb490`) +> that cannot be derived from the version, so every bump previously required +> scraping an HTML page to discover it. Second, **PyPI publishes a sha256 per +> file**, so the pinned digests are now *read off an API* rather than produced +> by downloading four ~80 MB archives and hashing them by hand. +> +> The bytes are the same either way: the `hyperd` inside the +> `macosx_13_0_arm64` wheel is bit-identical (sha256 +> `aef5c819…6450e478`, 277,836,448 bytes) to the one this crate used to pull +> from Tableau. Same build, different envelope — and the wheels are ~3.6–4.5% +> smaller. Both report `minos 13.0`, so the `macosx_13_0` wheel tag is not a +> raised support floor and no contributor loses support. +> +> *Historical note, since older revisions of this file argued it at length:* +> Tableau's C++ `macos-arm64` zip did once ship an **x86_64** `hyperd`, which +> is why this crate used to take the Java bundle specifically. Tableau fixed +> that in `0.0.26225`; from that release the C++ and Java binaries are +> byte-identical, so the distinction no longer explains anything. ## Install @@ -36,18 +56,14 @@ hyperdb-bootstrap download --dest /opt/hyperd # Force a re-download even if the version is already cached hyperdb-bootstrap download --force -# Scrape the latest release off the public releases page (best-effort, -# skips sha256 verification). -hyperdb-bootstrap download --latest - -# Install a specific release by version + build id -hyperdb-bootstrap download --version 0.0.24457 --build-id rc36858b6 +# Install a specific release ad-hoc — just the version, no build id needed +hyperdb-bootstrap download --version 0.0.26359 # Use an external pinned-version TOML instead of the baked-in default hyperdb-bootstrap download --version-file ./my-hyperd.toml -# HEAD each supported platform's URL for the pinned release — used by CI -# to catch Tableau yanks/renames early. Exits non-zero on any failure. +# Check the pinned release for every supported platform — used by CI to +# catch yanks/renames early. Exits non-zero on any failure. hyperdb-bootstrap verify # Print the installed binary's path @@ -57,13 +73,18 @@ hyperdb-bootstrap which hyperdb-bootstrap version ``` +`--version X` on its own inherits the builtin pin's `[wheel_tag]` values and +carries **no** digests, so the download is unverified and a WARN is logged. The +four wheel tags are unchanged from `0.0.19484` through `0.0.26479`, so this works +for any realistic ad-hoc pin or benchmark baseline. For a release whose wheel +tags differ, write a full pin file and pass `--version-file`. + **Version-source precedence (highest → lowest):** -1. `--version X --build-id Y` -2. `--latest` -3. `--version-file PATH` -4. `./hyperd-version.toml` (auto-discovered in current dir) -5. Compiled-in default shipped with this crate +1. `--version X` +2. `--version-file PATH` +3. `./hyperd-version.toml` (auto-discovered in current dir) +4. Compiled-in default shipped with this crate ## Library @@ -88,39 +109,54 @@ synchronous applications. ## Build-time guarantees - **Compile-time pin validation.** `build.rs` parses `hyperd-version.toml` - on every build and fails fast if `version`/`build_id` are missing or - malformed, if a sha256 isn't 64 hex chars, or if an unknown platform - key appears. Empty sha256 strings are allowed (skip verification for + on every build and fails fast if `version` is missing or malformed, if a + `[wheel_tag]` entry is missing or empty for a supported platform, if a + sha256 isn't 64 hex chars, or if an unknown platform key appears in + either table. Empty sha256 strings are allowed (skip verification for that platform) but surface a `cargo:warning` so nobody ships a release with missing hashes by accident. -- **URL reachability (`verify` subcommand).** `hyperdb-bootstrap verify` - HEADs every supported platform's download URL for the pinned release. - It's wired into CI (see +- **Pin verification (`verify` subcommand).** `hyperdb-bootstrap verify` + does two things for every supported platform: it HEADs the download URL + for the pinned release, and it cross-checks the pinned sha256 against the + digest PyPI publishes for that exact wheel filename (via + `https://pypi.org/pypi/tableauhyperapi//json`). The second check + is what makes this meaningful — it validates the exact pinned bytes rather + than merely that the CDN serves *something* at that path. It's wired into + CI (see [`.github/workflows/verify-hyperd-pin.yml`](../.github/workflows/verify-hyperd-pin.yml)) so yanked or renamed archives fail a PR instead of the next contributor's `make download-hyperd`. ## Supported platforms -| OS | Arch | Slug | -|---------|---------|-------------------| -| macOS | arm64 | `macos-arm64` | -| macOS | x86_64 | `macos-x86_64` | -| Linux | x86_64 | `linux-x86_64` | -| Windows | x86_64 | `windows-x86_64` | +| OS | Arch | Slug | Wheel tag | +|---------|---------|-------------------|--------------------------| +| macOS | arm64 | `macos-arm64` | `macosx_13_0_arm64` | +| macOS | x86_64 | `macos-x86_64` | `macosx_10_11_x86_64` | +| Linux | x86_64 | `linux-x86_64` | `manylinux2014_x86_64` | +| Windows | x86_64 | `windows-x86_64` | `win_amd64` | Any other `(OS, ARCH)` errors out with a clear message. +The wheel tags are pin *data*, not constants in the Rust source: they aren't +guaranteed stable across releases (arm64 wheels only exist from `0.0.19484`, +and a future macOS floor bump would change `macosx_13_0_arm64`), and a wrong +tag produces a **silent 404** rather than a clear error. Keeping them in +`hyperd-version.toml` makes any such change a visible pin edit. + ## Install layout +The versioned cache directory and `current/VERSION` are keyed on the version +alone — there is no build id in the path any more. + ```text / -├── 0.0.24457.rc36858b6/ # versioned cache +├── 0.0.26479/ # versioned cache │ ├── hyperd # hyperd.exe on Windows -│ └── ... # other files shipped under lib/hyper/ +│ └── ... # other files shipped under the wheel's bin/hyper/ └── current/ # fresh copy on each successful run ├── hyperd - └── VERSION # text: "0.0.24457.rc36858b6" + └── VERSION # text: "0.0.26479" ``` `current/` is a file copy, not a symlink — this avoids needing admin diff --git a/hyperdb-bootstrap/build.rs b/hyperdb-bootstrap/build.rs index d6c47ca1..44633e4c 100644 --- a/hyperdb-bootstrap/build.rs +++ b/hyperdb-bootstrap/build.rs @@ -3,9 +3,9 @@ //! Compile-time sanity check for the pinned release metadata. //! -//! If `hyperd-version.toml` fails to parse, is missing `version`/`build_id`, -//! or has a shape the runtime code can't handle, the build fails here — -//! before any contributor ships a broken bump. +//! If `hyperd-version.toml` fails to parse, is missing `version`, lacks a +//! wheel tag for a supported platform, or has a shape the runtime code can't +//! handle, the build fails here — before any contributor ships a broken bump. //! //! Keep this in sync with `release.rs` / `platform.rs`. The check is //! deliberately lightweight (no network, no crates-io deps beyond what @@ -73,22 +73,12 @@ fn main() { "{}: `version` is empty", pin_path.display() ); - assert!( - !pin.build_id.trim().is_empty(), - "{}: `build_id` is empty", - pin_path.display() - ); // Reject stray whitespace / accidental newlines in the URL components. assert!( !pin.version.contains(char::is_whitespace), "{}: `version` contains whitespace", pin_path.display() ); - assert!( - !pin.build_id.contains(char::is_whitespace), - "{}: `build_id` contains whitespace", - pin_path.display() - ); const SUPPORTED: &[&str] = &[ "macos-arm64", @@ -96,13 +86,43 @@ fn main() { "linux-x86_64", "windows-x86_64", ]; - for key in pin.sha256.keys() { + for (table, keys) in [ + ("wheel_tag", pin.wheel_tag.keys()), + ("sha256", pin.sha256.keys()), + ] { + for key in keys { + assert!( + SUPPORTED.contains(&key.as_str()), + "{}: unknown platform key `{}` in [{}]; supported: {:?}", + pin_path.display(), + key, + table, + SUPPORTED + ); + } + } + + // Every supported platform needs a wheel tag: without one the wheel file + // name is not constructible, so `download` would fail at runtime on that + // platform only — exactly the kind of break that should not reach a + // contributor's machine. + for platform in SUPPORTED { + let tag = pin + .wheel_tag + .get(*platform) + .map(|t| t.trim()) + .unwrap_or_default(); assert!( - SUPPORTED.contains(&key.as_str()), - "{}: unknown platform key `{}` in [sha256]; supported: {:?}", + !tag.is_empty(), + "{}: no [wheel_tag] entry for `{}`; every supported platform needs one", pin_path.display(), - key, - SUPPORTED + platform + ); + assert!( + !tag.contains(char::is_whitespace), + "{}: wheel tag for `{}` contains whitespace", + pin_path.display(), + platform ); } for (plat, sha) in &pin.sha256 { @@ -129,7 +149,8 @@ fn main() { #[derive(serde::Deserialize)] struct PinCheck { version: String, - build_id: String, + #[serde(default)] + wheel_tag: HashMap, #[serde(default)] sha256: HashMap, } diff --git a/hyperdb-bootstrap/hyperd-version.toml b/hyperdb-bootstrap/hyperd-version.toml index b6191cb9..3c81ce05 100644 --- a/hyperdb-bootstrap/hyperd-version.toml +++ b/hyperdb-bootstrap/hyperd-version.toml @@ -1,21 +1,33 @@ -# Pinned Hyper **Java** API release used by hyperdb-bootstrap. +# Pinned Hyper API release used by hyperdb-bootstrap. # -# We pull `hyperd` from the Java binding's bundle, NOT the C++ one: the -# C++ macos-arm64 zip ships an x86_64 `hyperd` (upstream packaging defect), -# which only runs under Rosetta on Apple Silicon. The Java macos-arm64 -# bundle carries a native arm64 `hyperd`. Same URL template and same -# internal layout (lib/hyper/hyperd) — only the binding token + sha256s differ. +# `hyperd` is extracted from the PyPI `tableauhyperapi` wheels. The wheel +# carries the same `hyperd` build as Tableau's Java/C++ API bundles, but its +# filename is fully constructible from `version` + the platform's wheel tag, +# so bumping this pin needs no build id and no HTML scraping. # -# Bump these values (and sha256s) when upgrading. Contributors without -# an override get this exact release, so reproducibility depends on it. -version = "0.0.26479" -build_id = "r96880f6a" +# Digests are copied verbatim from the PyPI JSON API — do not compute by hand: +# curl -s https://pypi.org/pypi/tableauhyperapi//json \ +# | jq -r '.urls[]|"\(.filename) \(.digests.sha256)"' +# +# Contributors without an override get this exact release, so reproducibility +# depends on it. +version = "0.0.26479" + +# Wheel platform tag per target. These are NOT derivable from the version and +# are NOT stable across releases: arm64 wheels only exist from 0.0.19484, and a +# future macOS floor bump would change `macosx_13_0_arm64`. A wrong tag yields a +# silent 404, so the tag lives here — where changing it is a visible pin edit — +# rather than hardcoded in Rust. +[wheel_tag] +"macos-arm64" = "macosx_13_0_arm64" +"macos-x86_64" = "macosx_10_11_x86_64" +"linux-x86_64" = "manylinux2014_x86_64" +"windows-x86_64" = "win_amd64" -# sha256 of each platform's .zip. Omit a platform to skip verification -# for that platform (not recommended). Compute with: -# shasum -a 256 tableauhyperapi-java--release-main...zip +# sha256 of each platform's .whl, as published by PyPI. Omit a platform to skip +# verification for that platform (not recommended). [sha256] -"macos-arm64" = "65bd021b3d3470ac74728ec287866a3ee0dfd806daf2589670feb3580955ee95" -"macos-x86_64" = "6690669c8a6a6c7c6794c101beb31b83e1589d76cb14b0c817523069591f694c" -"linux-x86_64" = "c20be5b6874d319c7db01dcec763d7e65ae2483c0becc75f2914a58accc4f932" -"windows-x86_64" = "3a400508e79c67ce9dcd8bd317ab164a61b5aed8ca880031ed4bbe6621514e1d" +"macos-arm64" = "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" +"macos-x86_64" = "960e276028137847a3870695d9c2d5a1392c173b1e119ff1146d24a75deca71a" +"linux-x86_64" = "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc" +"windows-x86_64" = "7a4f96d2a22351e944fea6db5d03ab5272ad4c0577acc987bfcb3739ed639502" diff --git a/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs b/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs index 101c3fae..0e7da3fe 100644 --- a/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs +++ b/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs @@ -6,8 +6,8 @@ //! Subcommands: //! - `download` — install `hyperd` under `.hyperd//` and refresh //! `.hyperd/current/`. -//! - `verify` — HEAD each platform URL for the pinned release to confirm -//! the CDN is still serving it (CI guard against silent yanks). +//! - `verify` — probe each platform's wheel URL and cross-check its pinned +//! digest against PyPI (CI guard against silent yanks and digest drift). //! - `which` — print the path of the currently-installed `hyperd`. //! - `version` — print the pinned release metadata. @@ -36,8 +36,9 @@ struct Cli { enum Command { /// Download and install hyperd into `.hyperd/` (or --dest). Download(DownloadArgs), - /// HEAD every platform URL for the pinned release to check they're - /// still reachable. Useful as a CI guard against silent yanks. + /// Check every platform's wheel URL is still reachable and that its + /// pinned digest still matches what PyPI publishes. Useful as a CI + /// guard against silent yanks and digest drift. Verify(VerifyArgs), /// Print the path of the currently-installed hyperd (if any). Which(WhichArgs), @@ -48,7 +49,7 @@ enum Command { #[derive(Args)] #[command(group( ArgGroup::new("version_src") - .args(["latest", "version", "version_file"]) + .args(["version", "version_file"]) .required(false) .multiple(false) ))] @@ -61,19 +62,11 @@ struct DownloadArgs { #[arg(long)] force: bool, - /// Scrape the latest release from the Tableau releases page. - /// Best-effort; skips sha256 verification. + /// Explicit version to install (e.g. 0.0.26359). Reuses the pinned wheel + /// tags and skips sha256 verification, since digests are version-specific. #[arg(long)] - latest: bool, - - /// Explicit version to install (e.g. 0.0.24457). Requires --build-id. - #[arg(long, requires = "build_id")] version: Option, - /// Build id for --version (e.g. rc36858b6). - #[arg(long, requires = "version")] - build_id: Option, - /// Path to an external pinned-version TOML file. #[arg(long, value_name = "PATH")] version_file: Option, @@ -128,22 +121,22 @@ fn run_download(args: DownloadArgs) -> Result<()> { )] fn pick_version_source(args: &DownloadArgs) -> Result { // Precedence: - // 1. --version + --build-id (Explicit) - // 2. --latest (ScrapeLatest) - // 3. --version-file PATH (TomlFile) - // 4. ./hyperd-version.toml (auto-discovered TomlFile) - // 5. builtin - if let (Some(v), Some(b)) = (&args.version, &args.build_id) { + // 1. --version X (Explicit) + // 2. --version-file PATH (TomlFile) + // 3. ./hyperd-version.toml (auto-discovered TomlFile) + // 4. builtin + if let Some(v) = &args.version { + // Inherit the builtin pin's wheel tags — they are stable across every + // release that publishes an arm64 wheel (0.0.19484 onward), so an + // ad-hoc version override does not also need to restate them. Drop the + // digests: they are specific to the pinned version's files. let release = PinnedRelease { version: v.clone(), - build_id: b.clone(), + wheel_tag: PinnedRelease::builtin().wheel_tag, sha256: std::collections::HashMap::default(), }; return Ok(VersionSource::Explicit(release)); } - if args.latest { - return Ok(VersionSource::ScrapeLatest); - } if let Some(path) = &args.version_file { return Ok(VersionSource::TomlFile(path.clone())); } @@ -178,9 +171,19 @@ fn run_which(args: WhichArgs) -> Result<()> { )] fn run_version() -> Result<()> { let r = PinnedRelease::builtin(); - println!("pinned version: {}", r.version); - println!("pinned build_id: {}", r.build_id); - println!("version tag: {}", r.version_tag()); + println!("pinned version: {}", r.version); + for platform in [ + Platform::MacosArm64, + Platform::MacosX86_64, + Platform::LinuxX86_64, + Platform::WindowsX86_64, + ] { + println!( + " {:<16} {}", + platform.to_string(), + r.wheel_tag_for(platform).unwrap_or("") + ); + } Ok(()) } @@ -190,51 +193,39 @@ fn run_verify(args: VerifyArgs) -> Result<()> { .with_context(|| format!("loading {}", path.display()))?, None => PinnedRelease::builtin(), }; - println!("verifying hyperd {}...", release.version_tag()); - let outcomes = verify_release(&release).context("HEAD requests failed")?; + println!("verifying hyperd {}...", release.version); + let outcomes = verify_release(&release).context("probing platform URLs failed")?; let mut all_ok = true; for o in &outcomes { - match (o.status, &o.error) { - (Some(status), _) if o.ok() => { - println!( - " OK {:<16} [{status}] {}", - o.platform.to_string(), - o.url - ); - } - (Some(status), _) => { - all_ok = false; - println!( - " FAIL {:<16} [{status}] {}", - o.platform.to_string(), - o.url - ); - } - (None, Some(err)) => { - all_ok = false; - println!( - " FAIL {:<16} [network error] {} ({err})", - o.platform.to_string(), - o.url - ); - } - (None, None) => unreachable!("verify_release always sets status or error"), + let label = if o.ok() { "OK " } else { "FAIL" }; + if !o.ok() { + all_ok = false; } + let http = match (o.status, &o.error) { + (Some(status), _) => format!("{status}"), + (None, Some(err)) => format!("network error: {err}"), + (None, None) => unreachable!("verify_release always sets status or error"), + }; + println!( + " {label} {:<16} [{http}] {}", + o.platform.to_string(), + o.url + ); + println!(" {}", o.digest); } if !all_ok { - anyhow::bail!("one or more platform URLs failed to resolve"); + anyhow::bail!("one or more platforms failed URL or digest verification"); } - println!("all platforms reachable."); + println!("all platforms reachable with matching digests."); Ok(()) } fn print_installed(i: &InstalledHyperd) { let status = if i.cache_hit { "cached" } else { "installed" }; println!( - "{status}: hyperd {version}.{build_id} ({platform}) -> {path}", + "{status}: hyperd {version} ({platform}) -> {path}", status = status, version = i.version, - build_id = i.build_id, platform = i.platform, path = i.binary_path.display(), ); diff --git a/hyperdb-bootstrap/src/error.rs b/hyperdb-bootstrap/src/error.rs index 407c3d83..e17c18af 100644 --- a/hyperdb-bootstrap/src/error.rs +++ b/hyperdb-bootstrap/src/error.rs @@ -8,9 +8,9 @@ use thiserror::Error; /// Errors produced while downloading, verifying, and installing `hyperd`. /// /// Every fallible function in this crate returns a `Result`. The -/// variants line up with the phases of bootstrap: platform detection, -/// HTTP/curl fetching, TOML parsing, archive extraction, and checksum -/// verification. +/// variants line up with the phases of bootstrap: platform detection, URL +/// construction, `curl` fetching, TOML parsing, archive extraction, and +/// checksum verification. #[derive(Debug, Error)] pub enum Error { /// The host (`os` / `arch` combination) has no published `hyperd` build. @@ -36,20 +36,17 @@ pub enum Error { source: std::io::Error, }, - /// A `reqwest` HTTP client error (connection failure, TLS issue, etc.). - #[error("HTTP error: {0}")] - Http(#[source] reqwest::Error), - - /// A server returned a non-success HTTP status while fetching `url`. - #[error("HTTP {status} when fetching {url}")] - HttpStatus { - /// URL that was being fetched when the failure occurred. - url: String, - /// HTTP response status code. - status: u16, + /// The pinned release carries no wheel tag for this platform, so the + /// wheel file name cannot be constructed. + #[error( + "no wheel tag pinned for platform {platform}; add it to the [wheel_tag] table in hyperd-version.toml" + )] + MissingWheelTag { + /// Platform whose `[wheel_tag]` entry is missing. + platform: crate::platform::Platform, }, - /// The fallback `curl` subprocess exited with a non-zero status. + /// The `curl` subprocess exited with a non-zero status. #[error("curl exited with code {code} when fetching {url}")] CurlFailed { /// URL passed to `curl`. @@ -78,10 +75,6 @@ pub enum Error { /// The archive did not contain a recognizable `hyperd` executable. #[error("hyperd executable not found in extracted archive")] HyperdNotInArchive, - - /// Scraping the public releases page for the latest version failed. - #[error("failed to scrape latest release: {0}")] - ScrapeFailed(&'static str), } impl Error { @@ -106,12 +99,10 @@ impl Error { } } - /// Constructs an [`Self::HttpStatus`] error. - pub fn http_status(url: impl Into, status: u16) -> Self { - Error::HttpStatus { - url: url.into(), - status, - } + /// Constructs an [`Self::MissingWheelTag`] error. + #[must_use] + pub fn missing_wheel_tag(platform: crate::platform::Platform) -> Self { + Error::MissingWheelTag { platform } } /// Constructs an [`Self::CurlFailed`] error. diff --git a/hyperdb-bootstrap/src/extract.rs b/hyperdb-bootstrap/src/extract.rs index fb349784..16ec9fdd 100644 --- a/hyperdb-bootstrap/src/extract.rs +++ b/hyperdb-bootstrap/src/extract.rs @@ -3,14 +3,18 @@ //! ZIP-archive extraction for the Hyper API release bundle. //! -//! The upstream archive nests `hyperd` plus its shared libraries inside a -//! versioned top-level directory (e.g. -//! `tableauhyperapi-java-macos-arm64-release-main.0.0.24457.rc36858b6/`) and -//! then under `lib/hyper/` on Linux/macOS or `bin/hyper/` on Windows. This -//! module flattens both layers so downstream consumers only see the -//! `hyperd` runtime files. The layout is identical across the Java and C++ -//! bundles, so this extractor is agnostic to which binding we download -//! (we use Java — see `url.rs` for why). +//! The upstream archive nests `hyperd` plus its shared libraries inside one +//! top-level directory and then under `lib/hyper/` or `bin/hyper/` inside it. +//! This module flattens both layers so downstream consumers only see the +//! `hyperd` runtime files. +//! +//! A wheel is a zip, and the same two-layer shape holds: the PyPI +//! `tableauhyperapi` wheels put the executable at +//! `tableauhyperapi/bin/hyper/hyperd` (plus `hyperd.exe` and +//! `crashdumper.exe` on Windows), so the `tableauhyperapi` package directory +//! lands in the "one optional top-level wrapper" slot and needs no special +//! case. The `lib/hyper/` spelling is retained because Tableau's Java and C++ +//! zips use it on Linux/macOS. use std::fs::{self, File}; use std::io; @@ -18,9 +22,9 @@ use std::path::{Path, PathBuf}; use crate::Error; -/// Extract everything under `lib/hyper/` (or `bin/hyper/` on Windows) from -/// the Hyper API zip into `dest_dir`, flattening the wrapper prefixes -/// away. Returns the list of extracted file paths relative to `dest_dir`. +/// Extract everything under `lib/hyper/` or `bin/hyper/` from the Hyper API +/// archive into `dest_dir`, flattening the wrapper prefixes away. Returns the +/// list of extracted file paths relative to `dest_dir`. /// /// # Errors /// @@ -89,14 +93,13 @@ pub fn extract_hyperd(zip_path: &Path, dest_dir: &Path) -> Result, } /// Return the path stripped of a leading `lib/hyper/` or `bin/hyper/` prefix, -/// or `None` if the entry is outside those directories. The Hyper API zip -/// wraps everything in a top-level `tableauhyperapi--...` directory and -/// nests the runtime under `lib/hyper/` (Linux/macOS) or `bin/hyper/` -/// (Windows) inside it. +/// or `None` if the entry is outside those directories. The archive wraps +/// everything in one top-level directory (`tableauhyperapi/` in a wheel) and +/// nests the runtime under `bin/hyper/` — or `lib/hyper/` in Tableau's +/// Linux/macOS zips — inside it. fn strip_lib_hyper_prefix(path: &Path) -> Option { let mut comps = path.components(); - // Skip one optional top-level wrapper component (e.g. - // `tableauhyperapi-java-macos-arm64-release-main.0.0.24457.rc36858b6`) + // Skip one optional top-level wrapper component (e.g. `tableauhyperapi`) // before looking for the `lib/hyper` or `bin/hyper` pair. let first = comps.next()?; let (a, b) = if first.as_os_str() == "lib" || first.as_os_str() == "bin" { @@ -156,6 +159,153 @@ mod tests { assert_eq!(strip_lib_hyper_prefix(Path::new("other/file")), None); } + /// Entry names taken verbatim from the real + /// `tableauhyperapi-0.0.26479-py3-none-macosx_13_0_arm64.whl` (42 entries; + /// this is the shape-representative subset). Guards the claim that the + /// wheel needs no extractor changes: the `tableauhyperapi` package + /// directory occupies the optional-wrapper slot, and `bin/hyper` is + /// already an accepted pair. + #[test] + fn strip_prefix_matches_real_wheel_entries() { + assert_eq!( + strip_lib_hyper_prefix(Path::new("tableauhyperapi/bin/hyper/hyperd")), + Some(PathBuf::from("hyperd")) + ); + // `libtableauhyperapi.dylib` sits under `bin/` but NOT `bin/hyper/`: + // it is the Python binding's own library, not part of the engine. + assert_eq!( + strip_lib_hyper_prefix(Path::new("tableauhyperapi/bin/libtableauhyperapi.dylib")), + None + ); + for outside in [ + "tableauhyperapi/__init__.py", + "tableauhyperapi/impl/dll.py", + "tableauhyperapi-0.0.26479.dist-info/LICENSE", + "tableauhyperapi-0.0.26479.dist-info/RECORD", + ] { + assert_eq!( + strip_lib_hyper_prefix(Path::new(outside)), + None, + "{outside} should not be extracted" + ); + } + } + + /// The Windows wheel ships `crashdumper.exe` beside `hyperd.exe` under the + /// same `bin/hyper/` directory, so both must survive the flattening. + #[test] + fn strip_prefix_matches_windows_wheel_entries() { + assert_eq!( + strip_lib_hyper_prefix(Path::new("tableauhyperapi/bin/hyper/hyperd.exe")), + Some(PathBuf::from("hyperd.exe")) + ); + assert_eq!( + strip_lib_hyper_prefix(Path::new("tableauhyperapi/bin/hyper/crashdumper.exe")), + Some(PathBuf::from("crashdumper.exe")) + ); + } + + /// End-to-end extraction over a fixture that mirrors the real macOS wheel + /// entry list, including the entries that must be skipped. + #[test] + fn extract_wheel_layout() -> Result<(), Box> { + use std::io::Write; + let tmp = tempfile::tempdir()?; + let wheel_path = tmp + .path() + .join("tableauhyperapi-0.0.26479-py3-none-test.whl"); + { + let file = File::create(&wheel_path)?; + let mut zw = zip::ZipWriter::new(file); + let opts = zip::write::SimpleFileOptions::default(); + for (name, body) in [ + ("tableauhyperapi/__init__.py", "python"), + ("tableauhyperapi/impl/dll.py", "python"), + ( + "tableauhyperapi/bin/libtableauhyperapi.dylib", + "binding lib", + ), + ("tableauhyperapi/bin/hyper/hyperd", "fake hyperd"), + ("tableauhyperapi-0.0.26479.dist-info/LICENSE", "license"), + ("tableauhyperapi-0.0.26479.dist-info/RECORD", "record"), + ] { + zw.start_file(name, opts)?; + zw.write_all(body.as_bytes())?; + } + zw.finish()?; + } + let out = tmp.path().join("out"); + let files = extract_hyperd(&wheel_path, &out)?; + + assert_eq!(files, vec![PathBuf::from("hyperd")]); + assert_eq!(std::fs::read_to_string(out.join("hyperd"))?, "fake hyperd"); + // Nothing outside bin/hyper/ leaks into the install dir. + for skipped in [ + "__init__.py", + "libtableauhyperapi.dylib", + "LICENSE", + "RECORD", + ] { + assert!(!out.join(skipped).exists(), "{skipped} should be skipped"); + } + Ok(()) + } + + /// The Windows wheel case, including `crashdumper.exe`. + #[test] + fn extract_windows_wheel_layout() -> Result<(), Box> { + use std::io::Write; + let tmp = tempfile::tempdir()?; + let wheel_path = tmp + .path() + .join("tableauhyperapi-0.0.26479-py3-none-win_amd64.whl"); + { + let file = File::create(&wheel_path)?; + let mut zw = zip::ZipWriter::new(file); + let opts = zip::write::SimpleFileOptions::default(); + for name in [ + "tableauhyperapi/bin/hyper/hyperd.exe", + "tableauhyperapi/bin/hyper/crashdumper.exe", + "tableauhyperapi/bin/tableauhyperapi.dll", + ] { + zw.start_file(name, opts)?; + zw.write_all(b"fake")?; + } + zw.finish()?; + } + let out = tmp.path().join("out"); + let files = extract_hyperd(&wheel_path, &out)?; + + assert!(files.iter().any(|p| p == Path::new("hyperd.exe"))); + assert!(files.iter().any(|p| p == Path::new("crashdumper.exe"))); + assert!(!out.join("tableauhyperapi.dll").exists()); + Ok(()) + } + + /// A wheel with no engine inside must fail loudly rather than install an + /// empty directory. + #[test] + fn extract_errors_when_no_hyperd() -> Result<(), Box> { + use std::io::Write; + let tmp = tempfile::tempdir()?; + let wheel_path = tmp.path().join("no-engine.whl"); + { + let file = File::create(&wheel_path)?; + let mut zw = zip::ZipWriter::new(file); + zw.start_file( + "tableauhyperapi/__init__.py", + zip::write::SimpleFileOptions::default(), + )?; + zw.write_all(b"python")?; + zw.finish()?; + } + assert!(matches!( + extract_hyperd(&wheel_path, &tmp.path().join("out")), + Err(Error::HyperdNotInArchive) + )); + Ok(()) + } + #[test] fn extract_fixture_zip() -> Result<(), Box> { use std::io::Write; diff --git a/hyperdb-bootstrap/src/install.rs b/hyperdb-bootstrap/src/install.rs index 2cf89317..131e367f 100644 --- a/hyperdb-bootstrap/src/install.rs +++ b/hyperdb-bootstrap/src/install.rs @@ -2,8 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 OR MIT //! High-level [`install()`] entry point: resolves a release, downloads and -//! verifies the archive, extracts the executable into -//! `//`, then refreshes the +//! verifies the wheel, extracts the executable into +//! `//`, then refreshes the //! `/current/` pointer so downstream tooling can always find the //! active install at a stable path. @@ -15,12 +15,11 @@ use crate::download::download_and_verify; use crate::extract::extract_hyperd; use crate::platform::Platform; use crate::release::PinnedRelease; -use crate::scrape::scrape_latest; -use crate::url::build_download_url; +use crate::url::{build_download_url, wheel_filename}; /// Default directory (relative to CWD) used when [`InstallOptions::dest_root`] /// is left at its default. Laid out as -/// `.hyperd//hyperd[.exe]` plus a mirror at `.hyperd/current/`. +/// `.hyperd//hyperd[.exe]` plus a mirror at `.hyperd/current/`. pub const DEFAULT_DEST_ROOT: &str = ".hyperd"; /// Which `hyperd` release [`install`] should resolve. @@ -34,10 +33,8 @@ pub enum VersionSource { Builtin, /// Load pinned metadata from a specific TOML file. TomlFile(PathBuf), - /// Caller-supplied release (e.g. CLI `--version`/`--build-id` flags). + /// Caller-supplied release (e.g. the CLI `--version` flag). Explicit(PinnedRelease), - /// Best-effort scrape of the public releases page. - ScrapeLatest, } /// Configuration passed to [`install`]. @@ -71,10 +68,8 @@ pub struct InstalledHyperd { /// Absolute or relative path to the installed `hyperd` executable. /// Always under `/current/`. pub binary_path: PathBuf, - /// Installed release version (for example, `"0.0.24457"`). + /// Installed release version (for example, `"0.0.26479"`). pub version: String, - /// Installed release build id (for example, `"rc36858b6"`). - pub build_id: String, /// Host platform this install targets. pub platform: Platform, /// `true` if the versioned directory already existed and we skipped the @@ -96,7 +91,7 @@ pub struct InstalledHyperd { /// # Errors /// /// Returns any [`Error`] variant produced by the phases it drives — -/// platform detection, release resolution (TOML parsing or scraping), +/// platform detection, release resolution (TOML parsing), URL construction, /// download, checksum verification, ZIP extraction, or filesystem /// operations under `dest_root`. pub fn install(opts: InstallOptions) -> Result { @@ -104,9 +99,8 @@ pub fn install(opts: InstallOptions) -> Result { Some(p) => p, None => Platform::current()?, }; - let release = resolve_release(&opts.version_source, platform)?; - let version_tag = release.version_tag(); - let versioned_dir = opts.dest_root.join(&version_tag); + let release = resolve_release(&opts.version_source)?; + let versioned_dir = opts.dest_root.join(&release.version); let current_dir = opts.dest_root.join("current"); let exe_name = platform.executable_name(); @@ -120,23 +114,21 @@ pub fn install(opts: InstallOptions) -> Result { download_and_extract(&release, platform, &versioned_dir)?; } - refresh_current(¤t_dir, &versioned_dir, &version_tag)?; + refresh_current(¤t_dir, &versioned_dir, &release.version)?; let binary_path = current_dir.join(exe_name); Ok(InstalledHyperd { binary_path, version: release.version, - build_id: release.build_id, platform, cache_hit, }) } -fn resolve_release(source: &VersionSource, platform: Platform) -> Result { +fn resolve_release(source: &VersionSource) -> Result { match source { VersionSource::Builtin => Ok(PinnedRelease::builtin()), VersionSource::TomlFile(path) => PinnedRelease::from_toml_file(path), VersionSource::Explicit(r) => Ok(r.clone()), - VersionSource::ScrapeLatest => scrape_latest(platform), } } @@ -152,15 +144,15 @@ fn download_and_extract( fs::create_dir_all(versioned_dir) .map_err(|source| Error::io(format!("creating {}", versioned_dir.display()), source))?; - let url = build_download_url(release, platform); + let url = build_download_url(release, platform)?; let tmp = tempfile::tempdir().map_err(|source| Error::io("creating temp dir", source))?; - let zip_path = tmp.path().join("hyperapi-java.zip"); - download_and_verify(&url, release.sha256_for(platform), &zip_path)?; - extract_hyperd(&zip_path, versioned_dir)?; + let wheel_path = tmp.path().join(wheel_filename(release, platform)?); + download_and_verify(&url, release.sha256_for(platform), &wheel_path)?; + extract_hyperd(&wheel_path, versioned_dir)?; Ok(()) } -fn refresh_current(current: &Path, source: &Path, version_tag: &str) -> Result<(), Error> { +fn refresh_current(current: &Path, source: &Path, version: &str) -> Result<(), Error> { // current/ is a fresh file copy every run — avoids Windows symlink // privileges and keeps the Makefile auto-discovery path stable. if current.exists() { @@ -170,7 +162,7 @@ fn refresh_current(current: &Path, source: &Path, version_tag: &str) -> Result<( fs::create_dir_all(current) .map_err(|source| Error::io(format!("creating {}", current.display()), source))?; copy_dir_contents(source, current)?; - fs::write(current.join("VERSION"), version_tag) + fs::write(current.join("VERSION"), version) .map_err(|source| Error::io(format!("writing {}/VERSION", current.display()), source))?; Ok(()) } diff --git a/hyperdb-bootstrap/src/lib.rs b/hyperdb-bootstrap/src/lib.rs index 24364501..3f7daaeb 100644 --- a/hyperdb-bootstrap/src/lib.rs +++ b/hyperdb-bootstrap/src/lib.rs @@ -1,10 +1,12 @@ // Copyright (c) 2026, Salesforce, Inc. All rights reserved. // SPDX-License-Identifier: Apache-2.0 OR MIT -//! Download and install the `hyperd` executable from Tableau's Hyper Java -//! API release packages. (The Java bundle is used rather than the C++ one -//! because the C++ `macos-arm64` zip ships an x86_64 `hyperd`; see the -//! `url` module for the full rationale.) +//! Download and install the `hyperd` executable from the PyPI +//! `tableauhyperapi` wheels. The wheel carries the same `hyperd` build as +//! Tableau's Java/C++ API bundles, but its file name is constructible from +//! the release version alone — no opaque build id, no page scraping — and +//! PyPI publishes a sha256 per file. See the `url` module for the full +//! rationale. //! //! The crate ships both a CLI binary (`hyperdb-bootstrap`) and a small //! library. The library is blocking (no async runtime required) and has @@ -21,8 +23,8 @@ //! ``` //! //! See [`InstallOptions`] and [`VersionSource`] for how to override the -//! destination, pin a specific release, load metadata from an external -//! TOML file, or scrape the latest release from the public releases page. +//! destination, pin a specific release, or load metadata from an external +//! TOML file. /// HTTP (via `curl`) download of release archives + SHA-256 verification. pub mod download; @@ -36,16 +38,13 @@ pub mod install; pub mod platform; /// Pinned-release metadata loaded from `hyperd-version.toml`. pub mod release; -/// Best-effort scraping of the public releases page to discover the latest -/// version when no pin is supplied. -pub mod scrape; -/// URL construction for Tableau's public download endpoint. +/// URL construction for the PyPI wheel download endpoint. pub mod url; -/// Reachability probes that HEAD each platform URL of a pinned release. +/// Reachability and digest checks for each platform of a pinned release. pub mod verify; pub use error::Error; pub use install::{DEFAULT_DEST_ROOT, InstallOptions, InstalledHyperd, VersionSource, install}; pub use platform::Platform; pub use release::PinnedRelease; -pub use verify::{VerifyOutcome, verify_release}; +pub use verify::{DigestStatus, VerifyOutcome, verify_release}; diff --git a/hyperdb-bootstrap/src/release.rs b/hyperdb-bootstrap/src/release.rs index e528cba5..aaeec1c3 100644 --- a/hyperdb-bootstrap/src/release.rs +++ b/hyperdb-bootstrap/src/release.rs @@ -3,9 +3,9 @@ //! A pinned `hyperd` release descriptor loaded from `hyperd-version.toml`. //! -//! Each `PinnedRelease` records a specific `version` + `build_id` pair -//! (the two components that make up a Hyper release tag) and the expected -//! SHA-256 checksums for each platform. +//! Each `PinnedRelease` records the release `version`, the wheel platform tag +//! to request for each target, and the expected SHA-256 checksum of each +//! platform's wheel. use serde::{Deserialize, Serialize}; use std::collections::HashMap; @@ -16,8 +16,8 @@ use crate::platform::Platform; const BUILTIN_TOML: &str = include_str!("../hyperd-version.toml"); -/// A concrete `hyperd` release pinned to a specific version and build, with -/// optional per-platform SHA-256 checksums. +/// A concrete `hyperd` release pinned to a specific version, with per-platform +/// wheel tags and optional per-platform SHA-256 checksums. /// /// The "built-in" pin shipped with the crate lives in /// `hyperdb-bootstrap/hyperd-version.toml` and is available via @@ -26,10 +26,13 @@ const BUILTIN_TOML: &str = include_str!("../hyperd-version.toml"); /// a literal TOML string to [`PinnedRelease::from_toml_str`]. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct PinnedRelease { - /// Upstream release version (for example, `"0.0.24457"`). + /// Upstream release version (for example, `"0.0.26479"`). pub version: String, - /// Upstream build identifier suffix (for example, `"rc36858b6"`). - pub build_id: String, + /// Wheel platform tag keyed by platform (for example, + /// `"macosx_13_0_arm64"`). Kept as pin data rather than hardcoded because + /// the tags are not stable across releases and a wrong tag 404s silently. + #[serde(default)] + pub wheel_tag: HashMap, /// Expected SHA-256 digests keyed by platform. Empty strings are treated /// as "no digest" so that partially-filled tables skip verification for /// the missing targets instead of failing outright. @@ -78,17 +81,24 @@ impl PinnedRelease { /// strings (common in pre-release metadata) are treated as absent. #[must_use] pub fn sha256_for(&self, platform: Platform) -> Option<&str> { - self.sha256 - .get(&platform) - .map(|s| s.trim()) - .filter(|s| !s.is_empty()) + Self::lookup(&self.sha256, platform) } - /// Returns the full Hyper release tag — `version.build_id` — used in - /// download URLs and install directory names. + /// Returns the wheel platform tag for `platform` (for example, + /// `"macosx_13_0_arm64"`), or `None` if the pin does not carry one. + /// + /// Without a tag the wheel filename cannot be constructed, so callers + /// treat `None` as [`Error::MissingWheelTag`] rather than guessing. #[must_use] - pub fn version_tag(&self) -> String { - format!("{}.{}", self.version, self.build_id) + pub fn wheel_tag_for(&self, platform: Platform) -> Option<&str> { + Self::lookup(&self.wheel_tag, platform) + } + + fn lookup(table: &HashMap, platform: Platform) -> Option<&str> { + table + .get(&platform) + .map(|s| s.trim()) + .filter(|s| !s.is_empty()) } } @@ -100,14 +110,29 @@ mod tests { fn builtin_parses() { let r = PinnedRelease::builtin(); assert!(!r.version.is_empty()); - assert!(!r.build_id.is_empty()); } #[test] - fn empty_sha_is_ignored() { + fn builtin_pins_a_wheel_tag_and_digest_for_every_platform() { + let r = PinnedRelease::builtin(); + for p in [ + Platform::MacosArm64, + Platform::MacosX86_64, + Platform::LinuxX86_64, + Platform::WindowsX86_64, + ] { + assert!(r.wheel_tag_for(p).is_some(), "{p} has no wheel tag"); + assert!(r.sha256_for(p).is_some(), "{p} has no sha256"); + } + } + + #[test] + fn empty_entries_are_ignored() { let toml_str = r#" version = "0.0.1" -build_id = "rc1" +[wheel_tag] +"macos-arm64" = "" +"linux-x86_64" = "manylinux2014_x86_64" [sha256] "macos-arm64" = "" "linux-x86_64" = "abc" @@ -115,15 +140,17 @@ build_id = "rc1" let r = PinnedRelease::from_toml_str(toml_str).unwrap(); assert!(r.sha256_for(Platform::MacosArm64).is_none()); assert_eq!(r.sha256_for(Platform::LinuxX86_64), Some("abc")); + assert!(r.wheel_tag_for(Platform::MacosArm64).is_none()); + assert_eq!( + r.wheel_tag_for(Platform::LinuxX86_64), + Some("manylinux2014_x86_64") + ); } #[test] - fn version_tag_format() { - let r = PinnedRelease { - version: "0.0.24457".to_string(), - build_id: "rc36858b6".to_string(), - sha256: HashMap::new(), - }; - assert_eq!(r.version_tag(), "0.0.24457.rc36858b6"); + fn tables_default_to_empty_when_absent() { + let r = PinnedRelease::from_toml_str("version = \"0.0.1\"").unwrap(); + assert!(r.wheel_tag_for(Platform::MacosArm64).is_none()); + assert!(r.sha256_for(Platform::MacosArm64).is_none()); } } diff --git a/hyperdb-bootstrap/src/scrape.rs b/hyperdb-bootstrap/src/scrape.rs deleted file mode 100644 index 2defd3f2..00000000 --- a/hyperdb-bootstrap/src/scrape.rs +++ /dev/null @@ -1,154 +0,0 @@ -// Copyright (c) 2026, Salesforce, Inc. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 OR MIT - -//! Best-effort scraper for the public Hyper releases page. -//! -//! When no version pin is supplied, [`crate::scrape::scrape_latest`] fetches -//! `https://tableau.github.io/hyper-db/docs/releases` and parses out the -//! most recent version + build id for the given platform. This bypasses -//! the compile-time pin and can lag or break when the page layout changes -//! — prefer an explicit [`crate::VersionSource::Builtin`] or -//! [`crate::VersionSource::TomlFile`] in production. - -use regex::Regex; -use std::collections::HashMap; - -use crate::Error; -use crate::platform::Platform; -use crate::release::PinnedRelease; - -const RELEASES_URL: &str = "https://tableau.github.io/hyper-db/docs/releases"; - -/// Installs `ring` as the process-wide rustls crypto provider. -/// -/// Our `reqwest` uses the `rustls-no-provider` feature, which links no -/// provider of its own. `reqwest` resolves one via -/// `CryptoProvider::get_default()`, which has no crate-feature fallback, and -/// *panics* while building a `Client` if nothing is installed. So this must -/// run before the first client is built. -/// -/// An `Err` from `install_default` means something else — most likely the host -/// application — installed a provider first. That is deliberately ignored: a -/// library must not override an embedder's choice. -fn ensure_crypto_provider() { - static INIT: std::sync::OnceLock<()> = std::sync::OnceLock::new(); - INIT.get_or_init(|| { - let _ = rustls::crypto::ring::default_provider().install_default(); - }); -} - -/// Fetches the public releases page and returns the newest `PinnedRelease` -/// that has a Java download for `platform`. -/// -/// The returned `PinnedRelease` has an empty SHA-256 map — scraping only -/// recovers the version + build id, never a digest. -/// -/// # Errors -/// -/// - [`Error::Http`] on network / TLS failure. -/// - [`Error::HttpStatus`] on a non-success HTTP response. -/// - [`Error::ScrapeFailed`] when the page layout no longer matches the -/// expected structure. -pub fn scrape_latest(platform: Platform) -> Result { - tracing::info!(url = RELEASES_URL, "scraping latest release"); - ensure_crypto_provider(); - let client = reqwest::blocking::Client::builder() - .user_agent(concat!("hyperd-bootstrap/", env!("CARGO_PKG_VERSION"))) - .build() - .map_err(Error::Http)?; - let resp = client.get(RELEASES_URL).send().map_err(Error::Http)?; - if !resp.status().is_success() { - return Err(Error::http_status(RELEASES_URL, resp.status().as_u16())); - } - let html = resp.text().map_err(Error::Http)?; - parse_latest(&html, platform) -} - -fn parse_latest(html: &str, platform: Platform) -> Result { - // The releases page lists entries in reverse-chronological order as - // `

VERSION [DATE]

`. The first match is the newest. - let h3_re = - Regex::new(r"]*>\s*([0-9]+(?:\.[0-9]+){1,3})\s*\[[^\]]+\]").expect("valid regex"); - let version = h3_re - .captures(html) - .and_then(|c| c.get(1)) - .map(|m| m.as_str().to_string()) - .ok_or(Error::ScrapeFailed( - "no

VERSION [DATE]

heading found", - ))?; - - // For that version, find the Java zip for the requested platform to - // recover the build id. - let href_re = Regex::new(&format!( - r"tableauhyperapi-java-{plat}-release-main\.{ver}\.(rc[a-z0-9]+)\.zip", - plat = regex::escape(platform.slug()), - ver = regex::escape(&version), - )) - .expect("valid regex"); - let build_id = href_re - .captures(html) - .and_then(|c| c.get(1)) - .map(|m| m.as_str().to_string()) - .ok_or(Error::ScrapeFailed( - "no matching java zip href for scraped version", - ))?; - - Ok(PinnedRelease { - version, - build_id, - sha256: HashMap::new(), - }) -} - -#[cfg(test)] -mod tests { - use super::*; - - /// Guards the `rustls-no-provider` feature choice. - /// - /// `reqwest` panics inside `build()` when no crypto provider is installed, - /// so this needs no network to fail. Without - /// [`ensure_crypto_provider`] it aborts the test thread outright. - #[test] - fn client_builds_with_a_crypto_provider_installed() { - ensure_crypto_provider(); - assert!( - reqwest::blocking::Client::builder() - .user_agent("hyperd-bootstrap-test") - .build() - .is_ok() - ); - } - - #[test] - fn parse_real_page_snippet() { - let html = r#" -

0.0.24457 [February 12 2026]

-
  • Release notes
- Java (macOS arm64) - Java (Linux) -

0.0.20000 [January 1 2025]

- "#; - let release = parse_latest(html, Platform::MacosArm64).unwrap(); - assert_eq!(release.version, "0.0.24457"); - assert_eq!(release.build_id, "rc36858b6"); - } - - #[test] - fn parse_errors_when_no_heading() { - let html = "

nothing here

"; - assert!(matches!( - parse_latest(html, Platform::LinuxX86_64), - Err(Error::ScrapeFailed(_)) - )); - } - - #[test] - fn parse_errors_when_no_matching_href() { - let html = r"

0.0.24457 [Feb 12 2026]

"; - assert!(matches!( - parse_latest(html, Platform::WindowsX86_64), - Err(Error::ScrapeFailed(_)) - )); - } -} diff --git a/hyperdb-bootstrap/src/url.rs b/hyperdb-bootstrap/src/url.rs index 60edd126..25a64c8b 100644 --- a/hyperdb-bootstrap/src/url.rs +++ b/hyperdb-bootstrap/src/url.rs @@ -1,36 +1,59 @@ // Copyright (c) 2026, Salesforce, Inc. All rights reserved. // SPDX-License-Identifier: Apache-2.0 OR MIT -//! Builds canonical download URLs for Tableau's public Hyper **Java** API -//! release bundles. +//! Builds download URLs for the PyPI `tableauhyperapi` wheels, which is where +//! this crate sources the `hyperd` executable. //! -//! We deliberately use the Java binding's bundle rather than the C++ one: -//! the C++ `macos-arm64` zip ships an **x86_64** `hyperd` (an upstream -//! packaging defect), so on Apple Silicon it would only run under Rosetta. -//! The Java `macos-arm64` bundle carries a native arm64 `hyperd`. Both -//! bundles share the identical URL template (only the `java`/`cxx` token -//! differs) and the identical internal layout (`lib/hyper/hyperd`), so the -//! switch is confined to this token and the pinned sha256s. +//! We use the wheel rather than Tableau's Java/C++ API zip on +//! `downloads.tableau.com` because the wheel filename is fully constructible +//! from the release version plus a platform tag. The zip filenames embed an +//! opaque build id (e.g. `r96880f6a`) that cannot be derived from the version, +//! which previously forced every bump through an HTML scraper. The `hyperd` +//! inside the wheel is the same build — byte-identical to the one the Java +//! bundle ships — so this is a change of envelope, not of engine. +//! +//! The legacy `/packages/py3/t//` path used here is +//! constructible without querying the PyPI index; it 302-redirects to the +//! content-addressed URL. PyPI also publishes a sha256 per file, which is +//! where the digests in `hyperd-version.toml` come from. +use crate::Error; use crate::platform::Platform; use crate::release::PinnedRelease; -const BASE_URL: &str = "https://downloads.tableau.com/tssoftware"; +const BASE_URL: &str = "https://files.pythonhosted.org/packages/py3/t/tableauhyperapi"; -/// Builds the `downloads.tableau.com` URL for the given release / platform -/// combination. +/// Builds the wheel file name for the given release / platform combination, +/// for example `tableauhyperapi-0.0.26479-py3-none-macosx_13_0_arm64.whl`. +/// +/// # Errors /// -/// The URL template matches -/// `https://downloads.tableau.com/tssoftware/tableauhyperapi-java--release-main...zip`. -#[must_use] -pub fn build_download_url(release: &PinnedRelease, platform: Platform) -> String { - format!( - "{base}/tableauhyperapi-java-{plat}-release-main.{version}.{build_id}.zip", - base = BASE_URL, - plat = platform.slug(), +/// Returns [`Error::MissingWheelTag`] if the release does not pin a wheel tag +/// for `platform`. The tag is pin data (see [`PinnedRelease::wheel_tag_for`]) +/// precisely so that a missing or changed tag fails loudly here instead of +/// producing a 404 at download time. +pub fn wheel_filename(release: &PinnedRelease, platform: Platform) -> Result { + let tag = release + .wheel_tag_for(platform) + .ok_or_else(|| Error::missing_wheel_tag(platform))?; + Ok(format!( + "tableauhyperapi-{version}-py3-none-{tag}.whl", version = release.version, - build_id = release.build_id, - ) + )) +} + +/// Builds the `files.pythonhosted.org` URL for the given release / platform +/// combination. +/// +/// # Errors +/// +/// Returns [`Error::MissingWheelTag`] if the release does not pin a wheel tag +/// for `platform`. +pub fn build_download_url(release: &PinnedRelease, platform: Platform) -> Result { + Ok(format!( + "{BASE_URL}/{file}", + file = wheel_filename(release, platform)? + )) } #[cfg(test)] @@ -38,17 +61,73 @@ mod tests { use super::*; use std::collections::HashMap; + fn release() -> PinnedRelease { + PinnedRelease { + version: "0.0.26479".to_string(), + wheel_tag: HashMap::from([ + (Platform::MacosArm64, "macosx_13_0_arm64".to_string()), + (Platform::MacosX86_64, "macosx_10_11_x86_64".to_string()), + (Platform::LinuxX86_64, "manylinux2014_x86_64".to_string()), + (Platform::WindowsX86_64, "win_amd64".to_string()), + ]), + sha256: HashMap::new(), + } + } + #[test] fn url_matches_expected_template() { + assert_eq!( + build_download_url(&release(), Platform::MacosArm64).unwrap(), + "https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-0.0.26479-py3-none-macosx_13_0_arm64.whl" + ); + } + + #[test] + fn every_platform_gets_its_own_tag() { + let r = release(); + for (platform, tag) in [ + (Platform::MacosArm64, "macosx_13_0_arm64"), + (Platform::MacosX86_64, "macosx_10_11_x86_64"), + (Platform::LinuxX86_64, "manylinux2014_x86_64"), + (Platform::WindowsX86_64, "win_amd64"), + ] { + assert_eq!( + wheel_filename(&r, platform).unwrap(), + format!("tableauhyperapi-0.0.26479-py3-none-{tag}.whl") + ); + } + } + + #[test] + fn missing_wheel_tag_is_an_error_not_a_guess() { let r = PinnedRelease { - version: "0.0.24457".to_string(), - build_id: "rc36858b6".to_string(), + version: "0.0.26479".to_string(), + wheel_tag: HashMap::new(), sha256: HashMap::new(), }; - let url = build_download_url(&r, Platform::MacosArm64); - assert_eq!( - url, - "https://downloads.tableau.com/tssoftware/tableauhyperapi-java-macos-arm64-release-main.0.0.24457.rc36858b6.zip" - ); + assert!(matches!( + build_download_url(&r, Platform::LinuxX86_64), + Err(Error::MissingWheelTag { .. }) + )); + } + + #[test] + fn builtin_pin_builds_a_url_for_every_platform() { + let r = PinnedRelease::builtin(); + for p in [ + Platform::MacosArm64, + Platform::MacosX86_64, + Platform::LinuxX86_64, + Platform::WindowsX86_64, + ] { + let url = build_download_url(&r, p).expect("builtin pin has every wheel tag"); + assert!(url.starts_with(BASE_URL)); + assert!( + std::path::Path::new(&url) + .extension() + .is_some_and(|ext| ext.eq_ignore_ascii_case("whl")) + ); + assert!(url.contains(&r.version)); + } } } diff --git a/hyperdb-bootstrap/src/verify.rs b/hyperdb-bootstrap/src/verify.rs index 49fc9875..7764a1dd 100644 --- a/hyperdb-bootstrap/src/verify.rs +++ b/hyperdb-bootstrap/src/verify.rs @@ -1,17 +1,32 @@ // Copyright (c) 2026, Salesforce, Inc. All rights reserved. // SPDX-License-Identifier: Apache-2.0 OR MIT -//! HEAD each supported platform's download URL to confirm the pinned -//! release is still reachable on Tableau's CDN. Used by the `verify` -//! CLI subcommand and by CI workflows that guard against silent yanks -//! or URL-scheme changes. +//! Checks that a pinned release is still fetchable *and* still the bytes we +//! pinned. Used by the `verify` CLI subcommand and by CI workflows that guard +//! against silent yanks, URL-scheme changes, and digest drift. +//! +//! Two independent checks run per platform: +//! +//! 1. **Reachability** — HEAD the constructed +//! `files.pythonhosted.org/packages/py3/...` URL. This is the legacy PyPI +//! path, which is constructible without an index query but only reaches the +//! file via a redirect, so it is worth probing directly. +//! 2. **Digest** — cross-check the pinned sha256 against the digest PyPI +//! publishes for that exact wheel file name. This is the stronger check: +//! reachability only proves the CDN serves *something*, whereas the digest +//! proves the pinned bytes are the published bytes. It also catches a stale +//! `[wheel_tag]`, which would otherwise 404 silently on one platform only. +use std::collections::HashMap; +use std::fmt; use std::process::Command; +use serde::Deserialize; + use crate::Error; use crate::platform::Platform; use crate::release::PinnedRelease; -use crate::url::build_download_url; +use crate::url::{build_download_url, wheel_filename}; const PLATFORMS: &[Platform] = &[ Platform::MacosArm64, @@ -20,8 +35,53 @@ const PLATFORMS: &[Platform] = &[ Platform::WindowsX86_64, ]; -/// Result of a single platform reachability probe performed by -/// [`verify_release`]. +/// Outcome of cross-checking one platform's pinned digest against PyPI. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum DigestStatus { + /// The pinned digest matches the one PyPI publishes for the wheel. + Match, + /// PyPI publishes a different digest for the pinned file name. + Mismatch { + /// Digest PyPI publishes for the wheel. + published: String, + }, + /// PyPI's file list has no entry for the pinned wheel file name — + /// typically a stale `[wheel_tag]` or a yanked release. + FileMissing, + /// The pin carries no digest for this platform, so there is nothing to + /// compare. Not a failure: an empty digest is documented as "skip". + NotPinned, + /// The PyPI index could not be consulted. Not treated as a failure, since + /// an index outage says nothing about the pin. + Unknown(String), +} + +impl DigestStatus { + /// Returns `true` unless PyPI actively contradicts the pin. + /// + /// [`Self::Unknown`] and [`Self::NotPinned`] are "no information", not + /// failure; [`Self::Mismatch`] and [`Self::FileMissing`] are hard failures. + #[must_use] + pub fn ok(&self) -> bool { + !matches!(self, Self::Mismatch { .. } | Self::FileMissing) + } +} + +impl fmt::Display for DigestStatus { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Match => f.write_str("digest matches PyPI"), + Self::Mismatch { published } => { + write!(f, "digest MISMATCH — PyPI publishes {published}") + } + Self::FileMissing => f.write_str("PyPI has no file with this name (stale wheel tag?)"), + Self::NotPinned => f.write_str("no digest pinned"), + Self::Unknown(why) => write!(f, "digest unchecked ({why})"), + } + } +} + +/// Result of a single platform's probe performed by [`verify_release`]. #[derive(Debug, Clone)] pub struct VerifyOutcome { /// Platform the probe targeted. @@ -31,47 +91,145 @@ pub struct VerifyOutcome { /// HTTP status returned by the CDN, or `None` if the probe itself failed. pub status: Option, /// Error message when `status` is `None` (spawn failure, parse failure, - /// stderr from a failed `curl` invocation). + /// stderr from a failed `curl` invocation, or URL construction failure). pub error: Option, + /// How the pinned digest compares to the one PyPI publishes. + pub digest: DigestStatus, } impl VerifyOutcome { - /// Returns `true` when the probe observed a 2xx/3xx HTTP status, which - /// is what `curl --head --location` returns when the CDN serves the - /// release. + /// Returns `true` when the probe observed a 2xx/3xx HTTP status (what + /// `curl --head --location` returns when the CDN serves the file) and the + /// digest check did not contradict the pin. #[must_use] pub fn ok(&self) -> bool { - matches!(self.status, Some(s) if (200..400).contains(&s)) + matches!(self.status, Some(s) if (200..400).contains(&s)) && self.digest.ok() } } -/// HEAD every supported platform URL for `release`. Returns one outcome -/// per platform; callers decide how to surface failures. +/// Probe every supported platform for `release`: HEAD its wheel URL and +/// cross-check its pinned digest against PyPI. Returns one outcome per +/// platform; callers decide how to surface failures. /// -/// Uses `curl --head` rather than reqwest so that Akamai's bot-protection -/// layer (which blocks reqwest's TLS fingerprint from GitHub-hosted runner -/// IPs) does not cause false 403 failures. +/// Uses `curl` rather than an in-process HTTP client so that Akamai-style bot +/// protection (which blocks some TLS fingerprints from CI runner IPs) does not +/// cause false failures, and so the crate needs no TLS stack of its own. /// /// # Errors /// /// Currently always returns `Ok(_)` — individual platform failures are -/// surfaced through [`VerifyOutcome::error`]. The `Result` wrapper is -/// kept for forward compatibility if a top-level failure mode is added. +/// surfaced through [`VerifyOutcome::error`] and [`VerifyOutcome::digest`]. +/// The `Result` wrapper is kept for forward compatibility if a top-level +/// failure mode is added. pub fn verify_release(release: &PinnedRelease) -> Result, Error> { + let digests = digest_statuses(release); let mut out = Vec::with_capacity(PLATFORMS.len()); for &platform in PLATFORMS { - let url = build_download_url(release, platform); - let outcome = curl_head(&url); - out.push(VerifyOutcome { - platform, - url, - status: outcome.0, - error: outcome.1, - }); + let digest = digests + .get(&platform) + .cloned() + .unwrap_or_else(|| DigestStatus::Unknown("platform not probed".to_string())); + let outcome = match build_download_url(release, platform) { + Ok(url) => { + let (status, error) = curl_head(&url); + VerifyOutcome { + platform, + url, + status, + error, + digest, + } + } + Err(e) => VerifyOutcome { + platform, + url: String::from(""), + status: None, + error: Some(e.to_string()), + digest, + }, + }; + out.push(outcome); } Ok(out) } +/// Fetch PyPI's file list for the pinned version and compare each platform's +/// pinned digest against it. +fn digest_statuses(release: &PinnedRelease) -> HashMap { + let url = format!( + "https://pypi.org/pypi/tableauhyperapi/{}/json", + release.version + ); + match curl_body(&url) { + Ok(body) => compare_digests(&body, release), + Err(why) => PLATFORMS + .iter() + .map(|&p| (p, DigestStatus::Unknown(why.clone()))) + .collect(), + } +} + +/// The subset of PyPI's per-version JSON payload we care about. Unknown +/// fields are ignored, so this survives additions to the API response. +#[derive(Deserialize)] +struct PypiVersion { + urls: Vec, +} + +#[derive(Deserialize)] +struct PypiFile { + filename: String, + digests: PypiDigests, +} + +#[derive(Deserialize)] +struct PypiDigests { + sha256: String, +} + +/// Pure comparison of a PyPI per-version JSON payload against a pin. Split out +/// from the network fetch so it can be tested against a fixture. +fn compare_digests(index_json: &str, release: &PinnedRelease) -> HashMap { + let parsed: PypiVersion = match serde_json::from_str(index_json) { + Ok(p) => p, + Err(e) => { + let why = format!("could not parse PyPI response: {e}"); + return PLATFORMS + .iter() + .map(|&p| (p, DigestStatus::Unknown(why.clone()))) + .collect(); + } + }; + let published: HashMap<&str, &str> = parsed + .urls + .iter() + .map(|f| (f.filename.as_str(), f.digests.sha256.as_str())) + .collect(); + + PLATFORMS + .iter() + .map(|&platform| { + let status = match wheel_filename(release, platform) { + Err(e) => DigestStatus::Unknown(e.to_string()), + Ok(filename) => match ( + published.get(filename.as_str()), + release.sha256_for(platform), + ) { + (None, _) => DigestStatus::FileMissing, + (Some(_), None) => DigestStatus::NotPinned, + (Some(actual), Some(pinned)) if actual.eq_ignore_ascii_case(pinned) => { + DigestStatus::Match + } + (Some(actual), Some(_)) => DigestStatus::Mismatch { + published: (*actual).to_string(), + }, + }, + }; + (platform, status) + }) + .collect() +} + /// Run `curl --head --silent --show-error --location ` and parse the /// HTTP status from the first status line. Returns `(Some(status), None)` /// on success and `(None, Some(error))` on spawn/parse failure. @@ -88,22 +246,8 @@ fn curl_head(url: &str) -> (Option, Option) { let stderr = String::from_utf8_lossy(&output.stderr).into_owned(); return (None, Some(stderr)); } - // Parse the last "HTTP/x.x NNN" status line (curl --location may - // produce multiple status lines when following redirects). let stdout = String::from_utf8_lossy(&output.stdout); - let status = stdout - .lines() - .filter_map(|line| { - let line = line.trim(); - // Matches "HTTP/1.1 200 OK", "HTTP/2 403", etc. - if line.starts_with("HTTP/") { - line.split_whitespace().nth(1)?.parse::().ok() - } else { - None - } - }) - .next_back(); - match status { + match last_http_status(&stdout) { Some(s) => (Some(s), None), None => ( None, @@ -116,3 +260,184 @@ fn curl_head(url: &str) -> (Option, Option) { } } } + +/// Parse the last `HTTP/x.x NNN` status line. `curl --location` emits one per +/// hop, and the final hop is the one that matters. +fn last_http_status(stdout: &str) -> Option { + stdout + .lines() + .filter_map(|line| { + let line = line.trim(); + // Matches "HTTP/1.1 200 OK", "HTTP/2 403", etc. + if line.starts_with("HTTP/") { + line.split_whitespace().nth(1)?.parse::().ok() + } else { + None + } + }) + .next_back() +} + +/// GET `url` with `curl` and return the response body. +fn curl_body(url: &str) -> Result { + let output = Command::new("curl") + .args(["--silent", "--show-error", "--location", "--fail"]) + .arg(url) + .output() + .map_err(|e| format!("failed to spawn curl: {e}"))?; + if !output.status.success() { + return Err(format!( + "curl exited {}: {}", + output.status.code().unwrap_or(-1), + String::from_utf8_lossy(&output.stderr).trim() + )); + } + Ok(String::from_utf8_lossy(&output.stdout).into_owned()) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Trimmed-down shape of a real `pypi.org/pypi/tableauhyperapi//json` + /// response: only the fields `compare_digests` reads, plus an extra field + /// to prove unknown keys are tolerated. + const INDEX_JSON: &str = r#"{ + "info": {"version": "0.0.26479"}, + "urls": [ + {"filename": "tableauhyperapi-0.0.26479-py3-none-macosx_13_0_arm64.whl", + "digests": {"sha256": "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118", "md5": "x"}, + "size": 80316638}, + {"filename": "tableauhyperapi-0.0.26479-py3-none-manylinux2014_x86_64.whl", + "digests": {"sha256": "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc"}, + "size": 89743672} + ] + }"#; + + fn pin(sha_arm64: &str) -> PinnedRelease { + PinnedRelease { + version: "0.0.26479".to_string(), + wheel_tag: HashMap::from([ + (Platform::MacosArm64, "macosx_13_0_arm64".to_string()), + (Platform::LinuxX86_64, "manylinux2014_x86_64".to_string()), + (Platform::WindowsX86_64, "win_amd64".to_string()), + ]), + sha256: HashMap::from([(Platform::MacosArm64, sha_arm64.to_string())]), + } + } + + #[test] + fn matching_digest_is_reported_as_match() { + let statuses = compare_digests( + INDEX_JSON, + &pin("e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118"), + ); + assert_eq!(statuses[&Platform::MacosArm64], DigestStatus::Match); + assert!(statuses[&Platform::MacosArm64].ok()); + } + + #[test] + fn digest_comparison_is_case_insensitive() { + let statuses = compare_digests( + INDEX_JSON, + &pin("E80E4DAC6D8437AD8C20F36ADD7E523B18BC06D90D4C605A256C57DF8DF2C118"), + ); + assert_eq!(statuses[&Platform::MacosArm64], DigestStatus::Match); + } + + #[test] + fn drifted_digest_is_a_failure() { + let statuses = compare_digests(INDEX_JSON, &pin(&"a".repeat(64))); + assert_eq!( + statuses[&Platform::MacosArm64], + DigestStatus::Mismatch { + published: "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" + .to_string() + } + ); + assert!(!statuses[&Platform::MacosArm64].ok()); + } + + /// A wheel tag that PyPI does not publish for this version is the silent-404 + /// vector the `[wheel_tag]` pin exists to make visible. + #[test] + fn unpublished_wheel_tag_is_a_failure() { + let statuses = compare_digests(INDEX_JSON, &pin(&"a".repeat(64))); + assert_eq!( + statuses[&Platform::WindowsX86_64], + DigestStatus::FileMissing + ); + assert!(!statuses[&Platform::WindowsX86_64].ok()); + } + + #[test] + fn published_but_unpinned_digest_is_not_a_failure() { + let statuses = compare_digests(INDEX_JSON, &pin(&"a".repeat(64))); + assert_eq!(statuses[&Platform::LinuxX86_64], DigestStatus::NotPinned); + assert!(statuses[&Platform::LinuxX86_64].ok()); + } + + /// A platform with no pinned wheel tag can't be checked, but must not be + /// reported as a digest failure — `verify_release` surfaces the missing tag + /// through the URL-construction error instead. + #[test] + fn missing_wheel_tag_is_unknown_not_failure() { + let statuses = compare_digests(INDEX_JSON, &pin(&"a".repeat(64))); + assert!(matches!( + statuses[&Platform::MacosX86_64], + DigestStatus::Unknown(_) + )); + assert!(statuses[&Platform::MacosX86_64].ok()); + } + + #[test] + fn unparseable_response_is_unknown_for_every_platform() { + let statuses = compare_digests("503", &pin(&"a".repeat(64))); + assert_eq!(statuses.len(), PLATFORMS.len()); + for status in statuses.values() { + assert!(matches!(status, DigestStatus::Unknown(_))); + assert!(status.ok()); + } + } + + #[test] + fn last_status_line_wins_across_redirects() { + let stdout = "HTTP/2 302\r\nlocation: elsewhere\r\n\r\nHTTP/2 200\r\n"; + assert_eq!(last_http_status(stdout), Some(200)); + assert_eq!(last_http_status("no status here"), None); + } + + #[test] + fn outcome_ok_requires_both_reachability_and_digest() { + let base = VerifyOutcome { + platform: Platform::MacosArm64, + url: "https://example.invalid/x.whl".to_string(), + status: Some(200), + error: None, + digest: DigestStatus::Match, + }; + assert!(base.ok()); + assert!( + !VerifyOutcome { + status: Some(404), + ..base.clone() + } + .ok() + ); + assert!( + !VerifyOutcome { + digest: DigestStatus::FileMissing, + ..base.clone() + } + .ok() + ); + // An index outage must not fail an otherwise-reachable pin. + assert!( + VerifyOutcome { + digest: DigestStatus::Unknown("offline".to_string()), + ..base + } + .ok() + ); + } +} diff --git a/hyperdb-bootstrap/tests/integration.rs b/hyperdb-bootstrap/tests/integration.rs index 2ad4eabd..55e7f08f 100644 --- a/hyperdb-bootstrap/tests/integration.rs +++ b/hyperdb-bootstrap/tests/integration.rs @@ -5,21 +5,67 @@ use hyperdb_bootstrap::{ InstallOptions, PinnedRelease, Platform, VersionSource, install, url::build_download_url, + url::wheel_filename, }; +const PLATFORMS: [Platform; 4] = [ + Platform::MacosArm64, + Platform::MacosX86_64, + Platform::LinuxX86_64, + Platform::WindowsX86_64, +]; + #[test] -fn builtin_release_builds_a_valid_url() { +fn builtin_release_builds_a_valid_wheel_url_for_every_platform() { let r = PinnedRelease::builtin(); - let url = build_download_url(&r, Platform::LinuxX86_64); - assert!(url.starts_with("https://downloads.tableau.com/tssoftware/")); - assert!(url.contains("java-linux-x86_64")); - assert!( - std::path::Path::new(&url) - .extension() - .is_some_and(|ext| ext.eq_ignore_ascii_case("zip")) - ); - assert!(url.contains(&r.version)); - assert!(url.contains(&r.build_id)); + for platform in PLATFORMS { + let url = build_download_url(&r, platform).expect("builtin pin has every wheel tag"); + assert!( + url.starts_with("https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/"), + "unexpected base for {platform}: {url}" + ); + assert!( + std::path::Path::new(&url) + .extension() + .is_some_and(|ext| ext.eq_ignore_ascii_case("whl")) + ); + assert!(url.contains(&r.version)); + assert!(url.contains(r.wheel_tag_for(platform).expect("wheel tag"))); + } +} + +/// The wheel file name is what PyPI publishes a digest against, so it has to +/// match the `tableauhyperapi--py3-none-.whl` convention exactly. +#[test] +fn builtin_wheel_filenames_follow_the_pypi_convention() { + let r = PinnedRelease::builtin(); + for platform in PLATFORMS { + let name = wheel_filename(&r, platform).expect("builtin pin has every wheel tag"); + assert_eq!( + name, + format!( + "tableauhyperapi-{}-py3-none-{}.whl", + r.version, + r.wheel_tag_for(platform).expect("wheel tag") + ) + ); + } +} + +/// Every platform must carry both a wheel tag and a digest, or `download` +/// breaks (or silently skips verification) on that platform alone. +#[test] +fn builtin_release_pins_every_platform() { + let r = PinnedRelease::builtin(); + for platform in PLATFORMS { + assert!( + r.wheel_tag_for(platform).is_some(), + "{platform} has no wheel tag" + ); + let sha = r.sha256_for(platform).expect("digest"); + assert_eq!(sha.len(), 64, "{platform} digest is not 64 hex chars"); + assert!(sha.chars().all(|c| c.is_ascii_hexdigit())); + } } #[test] @@ -32,7 +78,7 @@ fn install_options_defaults_are_sensible() { } #[test] -#[ignore = "hits the public Tableau downloads CDN; run with --ignored"] +#[ignore = "downloads an ~80 MB wheel from PyPI; run with --ignored"] fn install_end_to_end_with_builtin() { let tmp = tempfile::tempdir().unwrap(); let installed = install(InstallOptions { @@ -50,13 +96,34 @@ fn install_end_to_end_with_builtin() { .and_then(|n| n.to_str()) .is_some_and(|n| n.starts_with("hyperd")) ); + // The install dir and VERSION marker are keyed on the version alone. + assert_eq!( + std::fs::read_to_string(tmp.path().join("current").join("VERSION")).unwrap(), + installed.version + ); + assert!(tmp.path().join(&installed.version).is_dir()); } #[test] -#[ignore = "scrapes a live web page; run with --ignored"] -fn scrape_latest_real_page() { - let platform = Platform::current().expect("supported platform"); - let r = hyperdb_bootstrap::scrape::scrape_latest(platform).expect("scrape succeeds"); - assert!(!r.version.is_empty()); - assert!(r.build_id.starts_with("rc")); +#[ignore = "hits the PyPI JSON API and the wheel CDN; run with --ignored"] +fn verify_builtin_release_against_pypi() { + let r = PinnedRelease::builtin(); + let outcomes = hyperdb_bootstrap::verify_release(&r).expect("verify runs"); + assert_eq!(outcomes.len(), PLATFORMS.len()); + for o in &outcomes { + assert!( + o.ok(), + "{} failed: status={:?} error={:?} digest={}", + o.platform, + o.status, + o.error, + o.digest + ); + assert_eq!( + o.digest, + hyperdb_bootstrap::DigestStatus::Match, + "{} digest not confirmed against PyPI", + o.platform + ); + } } From d7da987ba4045a4a4a1965e091ce40a5810f52d3 Mon Sep 17 00:00:00 2001 From: Stefan Steiner Date: Sat, 5 Sep 2026 13:40:57 -0700 Subject: [PATCH 2/3] fix(bootstrap): use the post-rename crate name in module docs and errors Five references still said `hyperd-bootstrap` rather than `hyperdb-bootstrap`, including the `which` failure message, which told the user to run a command that does not exist. These land in files the PyPI-wheel migration already rewrites. `chore/post-rc-cleanup` fixes the same five in 0cc65e2; making the identical edit here means the two branches merge to the correct text either way, rather than risking a conflict resolution that takes this branch's side and silently reverts that fix. From a9fe1b04dd2e2c2aafb12346c0b5b41436f0d787 Mon Sep 17 00:00:00 2001 From: Stefan Steiner Date: Sat, 5 Sep 2026 14:34:23 -0700 Subject: [PATCH 3/3] chore: release 1.0.0-rc.2 Cuts the second release candidate for 1.0.0. The explicit footer is required rather than optional. The preceding commit is marked `feat(bootstrap)!`, and the workspace is already on 1.x, so release-please's DefaultVersioningStrategy would take the MajorVersionUpdate branch and compute 2.0.0. The `bump-minor-pre-major` setting in release-please-config.json does not help here: it is gated on `version.isPreMajor`, which is `major < 1` and therefore false at 1.0.0-rc.1. A Release-As note short-circuits `determineReleaseType` before the breaking-change tally is read, so it pins the version exactly. Staying inside the 1.0.0-rc line is a release-strategy decision, not a claim that nothing broke. The removals are enumerated in the preceding commit's BREAKING CHANGE footer and in hyperdb-bootstrap/CHANGELOG.md under Unreleased -> Removed. Release-As: 1.0.0-rc.2