diff --git a/.agents/skills/update-hyperd-release/SKILL.md b/.agents/skills/update-hyperd-release/SKILL.md index e19b51fd..fd15183e 100644 --- a/.agents/skills/update-hyperd-release/SKILL.md +++ b/.agents/skills/update-hyperd-release/SKILL.md @@ -1,6 +1,6 @@ --- name: update-hyperd-release -description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version, updating hyperd-version.toml (version + build_id + 4 sha256s), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. +description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version on PyPI, updating hyperd-version.toml (version + 4 wheel tags + 4 sha256s, all read straight off the PyPI JSON API), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. --- # Update the pinned `hyperd` release @@ -12,11 +12,13 @@ gotchas learned in practice. ## Key facts (don't relearn these the hard way) -- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 5. - - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, `build_id`, and four per-platform sha256s. This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. - - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin, used for the `hyperd` bundled into the npm packages. -- **We download the Java bundle, NOT the C++ one.** The C++ `macos-arm64` zip ships an **x86_64** `hyperd` (upstream packaging defect) that only runs under Rosetta on Apple Silicon. The Java `macos-arm64` bundle carries a **native arm64** `hyperd`. Same URL template (only the `java`/`cxx` token differs), same internal layout (`lib/hyper/hyperd`). **Verify this invariant every bump** (step 6) — if a future Java bundle regresses to x86_64, the whole reason for using it is gone. -- **URL template:** `https://downloads.tableau.com/tssoftware/tableauhyperapi-java--release-main...zip` — platforms: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. +- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 4. + - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, four `[wheel_tag]` entries, and four per-platform sha256s. **There is no `build_id` any more.** This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. + - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin (`HYPERD_VERSION`, plus a `hyperd-wheel-tag` / `hyperd-sha256` pair per platform), used for the `hyperd` bundled into the npm packages. +- **We download the PyPI `tableauhyperapi` wheels.** Every input to the URL is either the version you're bumping to or a value already in the pin, so nothing has to be discovered by scraping. The wheel carries `hyperd` at `tableauhyperapi/bin/hyper/hyperd` (`hyperd.exe` plus `crashdumper.exe` on Windows). +- **PyPI publishes a sha256 per file.** You *read* the four digests off the JSON API (step 2) instead of downloading four ~80 MB archives and hashing them by hand. This is the single biggest time saving in the whole procedure. The digests still get committed — a hash in git is an attestation that's independent of the host serving the bytes. +- **URL template:** `https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi--py3-none-.whl` — this legacy path is constructible without an API call and 302-redirects to the content-addressed URL. Platform slugs: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. +- **The wheel tags live in the pin because they are not guaranteed stable.** arm64 wheels only exist from `0.0.19484` onward, and a future macOS floor bump would change `macosx_13_0_arm64`. A wrong tag is a **silent 404**, not a loud error — so keeping the tags as pin data makes any such change a visible diff in the pin file. (Empirically all four tags are unchanged from `0.0.19484` through `0.0.26479`.) - **Crate version is workspace-driven + release-please.** `hyperdb-bootstrap` uses `version.workspace = true`; do **not** hand-edit a crate version. The conventional-commit type drives the release — use `fix(bootstrap): ...` for a routine bump (patch release). - **Never invent `hyperd` flags** (AGENTS.md reminder #9) and **never report tests/benches green without real output** (#10). Tests start a real `hyperd` subprocess; a misconfigured server hangs rather than erroring. @@ -27,71 +29,101 @@ Track these as todos. Each step gates the next. ### 1. Create a branch ```bash -git checkout -b chore/bump-hyperd- # e.g. chore/bump-hyperd-0.0.26225 +git checkout -b chore/bump-hyperd- # e.g. chore/bump-hyperd-0.0.26479 ``` -### 2. Find the latest version + build id +### 2. Read the version and all four digests off PyPI + +Two commands. The first gives you the version to bump to; the second gives you +every wheel for that version with its published sha256, ready to paste into the +toml. ```bash -curl -sL "https://tableau.github.io/hyper-db/docs/releases" | rg -o "0\.0\.[0-9]+" | head -1 -curl -sL "https://tableau.github.io/hyper-db/docs/releases" | \ - rg -o "tableauhyperapi-java-[a-z0-9_-]+-release-main\.\.r[a-z0-9]+\.zip" | sort -u +# Latest version on PyPI +curl -s https://pypi.org/pypi/tableauhyperapi/json | jq -r .info.version + +# Every wheel + its published sha256 for that version — paste straight into the toml +curl -s https://pypi.org/pypi/tableauhyperapi//json \ + | jq -r '.urls[] | "\(.filename) \(.digests.sha256)"' ``` -Confirm all four platform zips are listed for that version and share one build id. +The second command prints four lines. Map filename → platform slug so each digest +lands on the right toml line: + +| Wheel filename suffix | Platform slug | `[wheel_tag]` value | +|--------------------------------|------------------|--------------------------| +| `macosx_13_0_arm64.whl` | `macos-arm64` | `macosx_13_0_arm64` | +| `macosx_10_11_x86_64.whl` | `macos-x86_64` | `macosx_10_11_x86_64` | +| `manylinux2014_x86_64.whl` | `linux-x86_64` | `manylinux2014_x86_64` | +| `win_amd64.whl` | `windows-x86_64` | `win_amd64` | + +**Confirm the four printed filenames still carry exactly those four tags.** A +changed tag is the silent-404 vector: the pin would still compile and `verify` +would be the only thing that catches it. If a tag *has* changed, update the +matching `[wheel_tag]` entry in the same edit as the digests. + +Expect exactly four wheels. If PyPI lists more (or fewer) for the version, stop +and work out why before pinning it. + (The [`hyper-api-release-verify-upcoming-packages`](../hyper-api-release-verify-upcoming-packages/SKILL.md) -skill — bundled `verify_release.py` — validates the whole page's downloadability and -zip integrity for a given `--version`.) +skill — bundled `verify_release.py` — validates downloadability and archive +integrity for a given `--version` if you want a second opinion.) -### 3. Compute the four sha256s +### 3. Edit `hyperd-version.toml` -Download each Java zip and hash it. The values go verbatim into the toml. +Update `version` and all four `[sha256]` entries from the step-2 output; update a +`[wheel_tag]` entry only if step 2 showed the tag changed. Record the **old** +version first — you need it for the A/B benchmark (step 7). -```bash -V=; B=; cd "$(mktemp -d)" -for p in macos-arm64 macos-x86_64 linux-x86_64 windows-x86_64; do - curl -sL --fail -o "$p.zip" \ - "https://downloads.tableau.com/tssoftware/tableauhyperapi-java-$p-release-main.$V.$B.zip" & -done; wait -for p in macos-arm64 macos-x86_64 linux-x86_64 windows-x86_64; do - printf '%-16s ' "$p"; shasum -a 256 "$p.zip" | awk '{print $1}' -done -``` +The file should end up looking like this: -### 4. Edit `hyperd-version.toml` +```toml +version = "0.0.26479" -Update `version`, `build_id`, and all four `[sha256]` entries. Record the **old** -version/build_id first — you need it for the A/B benchmark (step 8). +[wheel_tag] +"macos-arm64" = "macosx_13_0_arm64" +"macos-x86_64" = "macosx_10_11_x86_64" +"linux-x86_64" = "manylinux2014_x86_64" +"windows-x86_64" = "win_amd64" -### 5. Mirror the pin into the npm release workflow +[sha256] +"macos-arm64" = "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" +"macos-x86_64" = "960e276028137847a3870695d9c2d5a1392c173b1e119ff1146d24a75deca71a" +"linux-x86_64" = "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc" +"windows-x86_64" = "7a4f96d2a22351e944fea6db5d03ab5272ad4c0577acc987bfcb3739ed639502" +``` + +### 4. Mirror the pin into the npm release workflow **The step that is easy to miss, and it reddens CI every time it is missed.** [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) bundles `hyperd` into the npm packages from its own hardcoded pin, decoupled -from the toml. Update it in the same commit as step 4 — the two files must +from the toml. Update it in the same commit as step 3 — the two files must never be bumped separately. | Key to update | Where in the workflow (line numbers drift — grep) | |---|---| -| `HYPERD_VERSION` | top-level `env:` block, ~line 26 | -| `HYPERD_BUILD_ID` | top-level `env:` block, ~line 27 | -| `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 99 | -| `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 110 | -| `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 115 | -| `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 105 | +| `HYPERD_VERSION` | top-level `env:` block, ~line 30 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 106 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 119 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 125 | +| `hyperd-wheel-tag` + `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 113 | ```bash -grep -nE "HYPERD_VERSION|HYPERD_BUILD_ID|hyperd-slug|hyperd-sha256" \ +grep -nE "HYPERD_VERSION|hyperd-slug|hyperd-wheel-tag|hyperd-sha256" \ .github/workflows/npm-build-publish.yml ``` -- **The matrix hashes are the same Java-zip sha256s you computed in step 3** — - not hashes of the extracted binary or of some other artifact. The workflow - downloads the identical URL - (`tableauhyperapi-java-${SLUG}-release-main.${HYPERD_VERSION}.${HYPERD_BUILD_ID}.zip`), - and the guard compares each `hyperd-sha256` **directly** against - `[sha256].""` in the toml, so the values are byte-for-byte identical. - Copy them across verbatim. +- **There is no `HYPERD_BUILD_ID` to update.** The workflow builds the wheel URL + from `HYPERD_VERSION` plus the per-platform `hyperd-wheel-tag`, so the build + id the Java-zip pin needed has no counterpart here. +- **The matrix values are the ones you read off PyPI in step 2** — the + `hyperd-sha256`s are the published `.whl` digests, not hashes of the extracted + binary or of some other artifact. The workflow downloads the identical URL + (`tableauhyperapi-${HYPERD_VERSION}-py3-none-${WHEEL_TAG}.whl`), and the guard + compares each `hyperd-sha256` **directly** against `[sha256].""` in the + toml, and each `hyperd-wheel-tag` against `[wheel_tag].""`, so the values + are byte-for-byte identical. Copy them across verbatim. - **`hyperd-slug` is the join key** and it carries the *toml's* platform names (`macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`), not npm's (`darwin-arm64`, `darwin-x64`, …), which live in the sibling `platform:` @@ -119,19 +151,23 @@ was bumped, so npm `0.7.1` shipped with bundled engine `0.0.25080` while crates.io shipped `0.0.26359`. The guard turns that into a red check instead of a mystery bug report months later. -### 6. Verify the pin + the arm64 invariant +### 5. Verify the pin ```bash -make verify-hyperd-pin # all four platforms → HTTP 200 at the new pin +make verify-hyperd-pin # all four platforms → HTTP 200, and each + # pinned sha256 matches the digest PyPI + # publishes for that exact wheel filename make download-hyperd # re-verifies the macos-arm64 sha256 on download -.hyperd/current/hyperd --version # should report main.. -file .hyperd/current/hyperd # MUST say "Mach-O 64-bit executable arm64" on Apple Silicon +.hyperd/current/hyperd --version # should report the new version +file .hyperd/current/hyperd # sanity check: "Mach-O 64-bit executable + # arm64" on Apple Silicon ``` -If `file` reports `x86_64`, **stop** — the Java bundle no longer carries a native -arm64 binary and the bundle choice needs re-evaluation. +`verify` cross-checking the digests, not just HEAD-ing the URLs, is what makes +this step meaningful: it proves the pin names the exact bytes PyPI serves, rather +than merely that the CDN serves *something* at that path. -### 7. Run the full test suite against the NEW engine +### 6. Run the full test suite against the NEW engine Point `HYPERD_PATH` at the freshly downloaded binary — do **not** rely on the workstation default (`~/dev/bin/hyperd`), which may be an old or unversioned build. @@ -146,17 +182,24 @@ Require `failed=0`. Then the pre-commit gate: `cargo fmt --all -- --check` and `cargo clippy --workspace --all-targets --all-features -- -D warnings` (CI's exact clippy command). -### 8. A/B benchmark vs the previous pin +### 7. A/B benchmark vs the previous pin The canonical harness is the **unified suite** ([`hyperdb-api/benches/benchmark_suite.rs`](../../../hyperdb-api/benches/benchmark_suite.rs)). Download the **old** pin into a separate dir, then run the same suite on both. See [docs/BENCHMARK_GUIDE.md](../../../docs/BENCHMARK_GUIDE.md) for the harness details. +`--version ` on its own is enough for the baseline: it inherits the +builtin pin's `[wheel_tag]` values and carries no digests, so the download is +unverified and logs a WARN. That's fine for a throwaway baseline, and the four +tags are unchanged all the way back to `0.0.19484`. (If you ever need a baseline +from a release whose tags *do* differ, write a full pin file and use +`--version-file` instead.) + ```bash # Old engine into a scratch dir (sha256 skipped — that's fine for a throwaway baseline) cargo run --release -p hyperdb-bootstrap --bin hyperdb-bootstrap -- \ - download --version --build-id --dest .hyperd-old + download --version --dest .hyperd-old cargo build -q -p hyperdb-api --release --example benchmark_suite BIN=target/release/examples/benchmark_suite; ROWS=100000000 # 100M for signal over noise @@ -174,35 +217,38 @@ rm -rf .hyperd-old # clean up the scratch baseline (also add to .gitignore if - **Distrust `× 4` / parallel numbers on a laptop.** They throttle thermally — throughput declines monotonically across sequential runs because the machine is hotter for the second engine. Report single-connection deltas as the reliable signal; withhold multi-connection deltas unless run on a cooled/pinned host. - Report throughput as **M rows/s**, not wall time. -### 9. Log the release in the benchmark tracker +### 8. Log the release in the benchmark tracker Append a row per engine to [`docs/hyperd-release-benchmarks.md`](../../../docs/hyperd-release-benchmarks.md) (median single-connection numbers + the machine + the caveat). This builds the per-release history the BENCHMARK_GUIDE's by-platform tables don't capture. -### 10. Changelog +### 9. Changelog Add a `### Changed` bullet under `## [Unreleased]` in [`hyperdb-bootstrap/CHANGELOG.md`](../../../hyperdb-bootstrap/CHANGELOG.md): the new -version/build, "verified native arm64", and the headline performance A/B (with the -thermal caveat on multi-connection numbers). +version, the wheel tags if any of them moved, and the headline performance A/B +(with the thermal caveat on multi-connection numbers). If `## [Unreleased]` +already has a `### Changed`, merge into it — a second sibling heading is +markdownlint MD024. -### 11. Commit + PR +### 10. Commit + PR -- Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to ()`. +- Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to `. - **gh account:** the EMU account (`ssteiner_sfemu`) is Unauthorized on upstream. `gh auth switch --hostname github.com --user StefanSteiner`, then target upstream (it has the CI runners): `gh pr create --repo tableau/hyper-api-rust --base main --head StefanSteiner:`. - Put the verification checklist + performance table in the PR body. ## Verification checklist (what "done" means) -- [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 -- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, `HYPERD_BUILD_ID`, three matrix `hyperd-sha256`s, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 -- [ ] `.hyperd/current/hyperd --version` reports the new version/build -- [ ] `file` confirms macos-arm64 binary is native arm64 +- [ ] Four wheels listed on PyPI for the new version, tags matching the pin +- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, three matrix `hyperd-wheel-tag`/`hyperd-sha256` pairs, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 +- [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 **and** all four digests match PyPI's published sha256 +- [ ] `.hyperd/current/hyperd --version` reports the new version +- [ ] `file` confirms the macos-arm64 binary is native arm64 - [ ] `cargo test --workspace` → `failed=0` against the new engine - [ ] `cargo fmt --check` + CI-exact `cargo clippy` clean - [ ] A/B benchmark done (medians of ≥3 runs @ 100M rows); scratch `.hyperd-old` removed - [ ] Row appended to `docs/hyperd-release-benchmarks.md` -- [ ] CHANGELOG `[Unreleased]` bullet added +- [ ] CHANGELOG `[Unreleased]` bullet added (merged into the existing `### Changed`) - [ ] PR opened against `tableau/hyper-api-rust` from `StefanSteiner:` diff --git a/.claude/skills/update-hyperd-release/SKILL.md b/.claude/skills/update-hyperd-release/SKILL.md index e19b51fd..fd15183e 100644 --- a/.claude/skills/update-hyperd-release/SKILL.md +++ b/.claude/skills/update-hyperd-release/SKILL.md @@ -1,6 +1,6 @@ --- name: update-hyperd-release -description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version, updating hyperd-version.toml (version + build_id + 4 sha256s), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. +description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version on PyPI, updating hyperd-version.toml (version + 4 wheel tags + 4 sha256s, all read straight off the PyPI JSON API), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. --- # Update the pinned `hyperd` release @@ -12,11 +12,13 @@ gotchas learned in practice. ## Key facts (don't relearn these the hard way) -- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 5. - - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, `build_id`, and four per-platform sha256s. This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. - - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin, used for the `hyperd` bundled into the npm packages. -- **We download the Java bundle, NOT the C++ one.** The C++ `macos-arm64` zip ships an **x86_64** `hyperd` (upstream packaging defect) that only runs under Rosetta on Apple Silicon. The Java `macos-arm64` bundle carries a **native arm64** `hyperd`. Same URL template (only the `java`/`cxx` token differs), same internal layout (`lib/hyper/hyperd`). **Verify this invariant every bump** (step 6) — if a future Java bundle regresses to x86_64, the whole reason for using it is gone. -- **URL template:** `https://downloads.tableau.com/tssoftware/tableauhyperapi-java--release-main...zip` — platforms: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. +- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 4. + - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, four `[wheel_tag]` entries, and four per-platform sha256s. **There is no `build_id` any more.** This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. + - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin (`HYPERD_VERSION`, plus a `hyperd-wheel-tag` / `hyperd-sha256` pair per platform), used for the `hyperd` bundled into the npm packages. +- **We download the PyPI `tableauhyperapi` wheels.** Every input to the URL is either the version you're bumping to or a value already in the pin, so nothing has to be discovered by scraping. The wheel carries `hyperd` at `tableauhyperapi/bin/hyper/hyperd` (`hyperd.exe` plus `crashdumper.exe` on Windows). +- **PyPI publishes a sha256 per file.** You *read* the four digests off the JSON API (step 2) instead of downloading four ~80 MB archives and hashing them by hand. This is the single biggest time saving in the whole procedure. The digests still get committed — a hash in git is an attestation that's independent of the host serving the bytes. +- **URL template:** `https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi--py3-none-.whl` — this legacy path is constructible without an API call and 302-redirects to the content-addressed URL. Platform slugs: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. +- **The wheel tags live in the pin because they are not guaranteed stable.** arm64 wheels only exist from `0.0.19484` onward, and a future macOS floor bump would change `macosx_13_0_arm64`. A wrong tag is a **silent 404**, not a loud error — so keeping the tags as pin data makes any such change a visible diff in the pin file. (Empirically all four tags are unchanged from `0.0.19484` through `0.0.26479`.) - **Crate version is workspace-driven + release-please.** `hyperdb-bootstrap` uses `version.workspace = true`; do **not** hand-edit a crate version. The conventional-commit type drives the release — use `fix(bootstrap): ...` for a routine bump (patch release). - **Never invent `hyperd` flags** (AGENTS.md reminder #9) and **never report tests/benches green without real output** (#10). Tests start a real `hyperd` subprocess; a misconfigured server hangs rather than erroring. @@ -27,71 +29,101 @@ Track these as todos. Each step gates the next. ### 1. Create a branch ```bash -git checkout -b chore/bump-hyperd- # e.g. chore/bump-hyperd-0.0.26225 +git checkout -b chore/bump-hyperd- # e.g. chore/bump-hyperd-0.0.26479 ``` -### 2. Find the latest version + build id +### 2. Read the version and all four digests off PyPI + +Two commands. The first gives you the version to bump to; the second gives you +every wheel for that version with its published sha256, ready to paste into the +toml. ```bash -curl -sL "https://tableau.github.io/hyper-db/docs/releases" | rg -o "0\.0\.[0-9]+" | head -1 -curl -sL "https://tableau.github.io/hyper-db/docs/releases" | \ - rg -o "tableauhyperapi-java-[a-z0-9_-]+-release-main\.\.r[a-z0-9]+\.zip" | sort -u +# Latest version on PyPI +curl -s https://pypi.org/pypi/tableauhyperapi/json | jq -r .info.version + +# Every wheel + its published sha256 for that version — paste straight into the toml +curl -s https://pypi.org/pypi/tableauhyperapi//json \ + | jq -r '.urls[] | "\(.filename) \(.digests.sha256)"' ``` -Confirm all four platform zips are listed for that version and share one build id. +The second command prints four lines. Map filename → platform slug so each digest +lands on the right toml line: + +| Wheel filename suffix | Platform slug | `[wheel_tag]` value | +|--------------------------------|------------------|--------------------------| +| `macosx_13_0_arm64.whl` | `macos-arm64` | `macosx_13_0_arm64` | +| `macosx_10_11_x86_64.whl` | `macos-x86_64` | `macosx_10_11_x86_64` | +| `manylinux2014_x86_64.whl` | `linux-x86_64` | `manylinux2014_x86_64` | +| `win_amd64.whl` | `windows-x86_64` | `win_amd64` | + +**Confirm the four printed filenames still carry exactly those four tags.** A +changed tag is the silent-404 vector: the pin would still compile and `verify` +would be the only thing that catches it. If a tag *has* changed, update the +matching `[wheel_tag]` entry in the same edit as the digests. + +Expect exactly four wheels. If PyPI lists more (or fewer) for the version, stop +and work out why before pinning it. + (The [`hyper-api-release-verify-upcoming-packages`](../hyper-api-release-verify-upcoming-packages/SKILL.md) -skill — bundled `verify_release.py` — validates the whole page's downloadability and -zip integrity for a given `--version`.) +skill — bundled `verify_release.py` — validates downloadability and archive +integrity for a given `--version` if you want a second opinion.) -### 3. Compute the four sha256s +### 3. Edit `hyperd-version.toml` -Download each Java zip and hash it. The values go verbatim into the toml. +Update `version` and all four `[sha256]` entries from the step-2 output; update a +`[wheel_tag]` entry only if step 2 showed the tag changed. Record the **old** +version first — you need it for the A/B benchmark (step 7). -```bash -V=; B=; cd "$(mktemp -d)" -for p in macos-arm64 macos-x86_64 linux-x86_64 windows-x86_64; do - curl -sL --fail -o "$p.zip" \ - "https://downloads.tableau.com/tssoftware/tableauhyperapi-java-$p-release-main.$V.$B.zip" & -done; wait -for p in macos-arm64 macos-x86_64 linux-x86_64 windows-x86_64; do - printf '%-16s ' "$p"; shasum -a 256 "$p.zip" | awk '{print $1}' -done -``` +The file should end up looking like this: -### 4. Edit `hyperd-version.toml` +```toml +version = "0.0.26479" -Update `version`, `build_id`, and all four `[sha256]` entries. Record the **old** -version/build_id first — you need it for the A/B benchmark (step 8). +[wheel_tag] +"macos-arm64" = "macosx_13_0_arm64" +"macos-x86_64" = "macosx_10_11_x86_64" +"linux-x86_64" = "manylinux2014_x86_64" +"windows-x86_64" = "win_amd64" -### 5. Mirror the pin into the npm release workflow +[sha256] +"macos-arm64" = "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" +"macos-x86_64" = "960e276028137847a3870695d9c2d5a1392c173b1e119ff1146d24a75deca71a" +"linux-x86_64" = "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc" +"windows-x86_64" = "7a4f96d2a22351e944fea6db5d03ab5272ad4c0577acc987bfcb3739ed639502" +``` + +### 4. Mirror the pin into the npm release workflow **The step that is easy to miss, and it reddens CI every time it is missed.** [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) bundles `hyperd` into the npm packages from its own hardcoded pin, decoupled -from the toml. Update it in the same commit as step 4 — the two files must +from the toml. Update it in the same commit as step 3 — the two files must never be bumped separately. | Key to update | Where in the workflow (line numbers drift — grep) | |---|---| -| `HYPERD_VERSION` | top-level `env:` block, ~line 26 | -| `HYPERD_BUILD_ID` | top-level `env:` block, ~line 27 | -| `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 99 | -| `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 110 | -| `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 115 | -| `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 105 | +| `HYPERD_VERSION` | top-level `env:` block, ~line 30 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 106 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 119 | +| `hyperd-wheel-tag` + `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 125 | +| `hyperd-wheel-tag` + `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 113 | ```bash -grep -nE "HYPERD_VERSION|HYPERD_BUILD_ID|hyperd-slug|hyperd-sha256" \ +grep -nE "HYPERD_VERSION|hyperd-slug|hyperd-wheel-tag|hyperd-sha256" \ .github/workflows/npm-build-publish.yml ``` -- **The matrix hashes are the same Java-zip sha256s you computed in step 3** — - not hashes of the extracted binary or of some other artifact. The workflow - downloads the identical URL - (`tableauhyperapi-java-${SLUG}-release-main.${HYPERD_VERSION}.${HYPERD_BUILD_ID}.zip`), - and the guard compares each `hyperd-sha256` **directly** against - `[sha256].""` in the toml, so the values are byte-for-byte identical. - Copy them across verbatim. +- **There is no `HYPERD_BUILD_ID` to update.** The workflow builds the wheel URL + from `HYPERD_VERSION` plus the per-platform `hyperd-wheel-tag`, so the build + id the Java-zip pin needed has no counterpart here. +- **The matrix values are the ones you read off PyPI in step 2** — the + `hyperd-sha256`s are the published `.whl` digests, not hashes of the extracted + binary or of some other artifact. The workflow downloads the identical URL + (`tableauhyperapi-${HYPERD_VERSION}-py3-none-${WHEEL_TAG}.whl`), and the guard + compares each `hyperd-sha256` **directly** against `[sha256].""` in the + toml, and each `hyperd-wheel-tag` against `[wheel_tag].""`, so the values + are byte-for-byte identical. Copy them across verbatim. - **`hyperd-slug` is the join key** and it carries the *toml's* platform names (`macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`), not npm's (`darwin-arm64`, `darwin-x64`, …), which live in the sibling `platform:` @@ -119,19 +151,23 @@ was bumped, so npm `0.7.1` shipped with bundled engine `0.0.25080` while crates.io shipped `0.0.26359`. The guard turns that into a red check instead of a mystery bug report months later. -### 6. Verify the pin + the arm64 invariant +### 5. Verify the pin ```bash -make verify-hyperd-pin # all four platforms → HTTP 200 at the new pin +make verify-hyperd-pin # all four platforms → HTTP 200, and each + # pinned sha256 matches the digest PyPI + # publishes for that exact wheel filename make download-hyperd # re-verifies the macos-arm64 sha256 on download -.hyperd/current/hyperd --version # should report main.. -file .hyperd/current/hyperd # MUST say "Mach-O 64-bit executable arm64" on Apple Silicon +.hyperd/current/hyperd --version # should report the new version +file .hyperd/current/hyperd # sanity check: "Mach-O 64-bit executable + # arm64" on Apple Silicon ``` -If `file` reports `x86_64`, **stop** — the Java bundle no longer carries a native -arm64 binary and the bundle choice needs re-evaluation. +`verify` cross-checking the digests, not just HEAD-ing the URLs, is what makes +this step meaningful: it proves the pin names the exact bytes PyPI serves, rather +than merely that the CDN serves *something* at that path. -### 7. Run the full test suite against the NEW engine +### 6. Run the full test suite against the NEW engine Point `HYPERD_PATH` at the freshly downloaded binary — do **not** rely on the workstation default (`~/dev/bin/hyperd`), which may be an old or unversioned build. @@ -146,17 +182,24 @@ Require `failed=0`. Then the pre-commit gate: `cargo fmt --all -- --check` and `cargo clippy --workspace --all-targets --all-features -- -D warnings` (CI's exact clippy command). -### 8. A/B benchmark vs the previous pin +### 7. A/B benchmark vs the previous pin The canonical harness is the **unified suite** ([`hyperdb-api/benches/benchmark_suite.rs`](../../../hyperdb-api/benches/benchmark_suite.rs)). Download the **old** pin into a separate dir, then run the same suite on both. See [docs/BENCHMARK_GUIDE.md](../../../docs/BENCHMARK_GUIDE.md) for the harness details. +`--version ` on its own is enough for the baseline: it inherits the +builtin pin's `[wheel_tag]` values and carries no digests, so the download is +unverified and logs a WARN. That's fine for a throwaway baseline, and the four +tags are unchanged all the way back to `0.0.19484`. (If you ever need a baseline +from a release whose tags *do* differ, write a full pin file and use +`--version-file` instead.) + ```bash # Old engine into a scratch dir (sha256 skipped — that's fine for a throwaway baseline) cargo run --release -p hyperdb-bootstrap --bin hyperdb-bootstrap -- \ - download --version --build-id --dest .hyperd-old + download --version --dest .hyperd-old cargo build -q -p hyperdb-api --release --example benchmark_suite BIN=target/release/examples/benchmark_suite; ROWS=100000000 # 100M for signal over noise @@ -174,35 +217,38 @@ rm -rf .hyperd-old # clean up the scratch baseline (also add to .gitignore if - **Distrust `× 4` / parallel numbers on a laptop.** They throttle thermally — throughput declines monotonically across sequential runs because the machine is hotter for the second engine. Report single-connection deltas as the reliable signal; withhold multi-connection deltas unless run on a cooled/pinned host. - Report throughput as **M rows/s**, not wall time. -### 9. Log the release in the benchmark tracker +### 8. Log the release in the benchmark tracker Append a row per engine to [`docs/hyperd-release-benchmarks.md`](../../../docs/hyperd-release-benchmarks.md) (median single-connection numbers + the machine + the caveat). This builds the per-release history the BENCHMARK_GUIDE's by-platform tables don't capture. -### 10. Changelog +### 9. Changelog Add a `### Changed` bullet under `## [Unreleased]` in [`hyperdb-bootstrap/CHANGELOG.md`](../../../hyperdb-bootstrap/CHANGELOG.md): the new -version/build, "verified native arm64", and the headline performance A/B (with the -thermal caveat on multi-connection numbers). +version, the wheel tags if any of them moved, and the headline performance A/B +(with the thermal caveat on multi-connection numbers). If `## [Unreleased]` +already has a `### Changed`, merge into it — a second sibling heading is +markdownlint MD024. -### 11. Commit + PR +### 10. Commit + PR -- Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to ()`. +- Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to `. - **gh account:** the EMU account (`ssteiner_sfemu`) is Unauthorized on upstream. `gh auth switch --hostname github.com --user StefanSteiner`, then target upstream (it has the CI runners): `gh pr create --repo tableau/hyper-api-rust --base main --head StefanSteiner:`. - Put the verification checklist + performance table in the PR body. ## Verification checklist (what "done" means) -- [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 -- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, `HYPERD_BUILD_ID`, three matrix `hyperd-sha256`s, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 -- [ ] `.hyperd/current/hyperd --version` reports the new version/build -- [ ] `file` confirms macos-arm64 binary is native arm64 +- [ ] Four wheels listed on PyPI for the new version, tags matching the pin +- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, three matrix `hyperd-wheel-tag`/`hyperd-sha256` pairs, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 +- [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 **and** all four digests match PyPI's published sha256 +- [ ] `.hyperd/current/hyperd --version` reports the new version +- [ ] `file` confirms the macos-arm64 binary is native arm64 - [ ] `cargo test --workspace` → `failed=0` against the new engine - [ ] `cargo fmt --check` + CI-exact `cargo clippy` clean - [ ] A/B benchmark done (medians of ≥3 runs @ 100M rows); scratch `.hyperd-old` removed - [ ] Row appended to `docs/hyperd-release-benchmarks.md` -- [ ] CHANGELOG `[Unreleased]` bullet added +- [ ] CHANGELOG `[Unreleased]` bullet added (merged into the existing `### Changed`) - [ ] PR opened against `tableau/hyper-api-rust` from `StefanSteiner:` diff --git a/.github/scripts/verify-npm-hyperd-pin.py b/.github/scripts/verify-npm-hyperd-pin.py index 2803ec1d..864d6bcc 100644 --- a/.github/scripts/verify-npm-hyperd-pin.py +++ b/.github/scripts/verify-npm-hyperd-pin.py @@ -2,8 +2,9 @@ """Guard: keep the npm release workflow's hyperd pin in sync with the toml. `.github/workflows/npm-build-publish.yml` bundles `hyperd` into the npm -packages using its OWN hardcoded `HYPERD_VERSION` / `HYPERD_BUILD_ID` / -per-platform matrix `hyperd-sha256`s. Those are decoupled from +packages from the PyPI `tableauhyperapi` wheel, using its OWN hardcoded +`HYPERD_VERSION` plus per-platform matrix `hyperd-wheel-tag`s and +`hyperd-sha256`s. Those are decoupled from `hyperdb-bootstrap/hyperd-version.toml`, which is what `make download-hyperd` and the crates.io path use. @@ -12,10 +13,16 @@ This script fails CI whenever the two drift, so that can't recur silently. The platform slug (`macos-arm64`, `linux-x86_64`, `windows-x86_64`) is the join -key: it is identical between the toml's `[sha256]` table and the workflow's -`hyperd-slug` matrix field. Only slugs the workflow actually builds are checked, -so a commented-out matrix entry (invisible to the YAML parser) and any unused -extra toml sha are both fine. +key: it is identical between the toml's `[wheel_tag]` / `[sha256]` tables and +the workflow's `hyperd-slug` matrix field. Only slugs the workflow actually +builds are checked, so a commented-out matrix entry (invisible to the YAML +parser) and any unused extra toml entry are both fine. + +The wheel tag (`macosx_13_0_arm64`, `manylinux2014_x86_64`, ...) is checked +alongside the digest because it is a drift vector of its own: it is not +derivable from the version, and a wrong tag resolves to a URL that does not +exist, so it would surface as an opaque 404 mid-release rather than as a +mismatch. The sha256s are digests of the downloaded `.whl` archive. """ from __future__ import annotations @@ -36,12 +43,12 @@ def main() -> int: workflow = yaml.safe_load(WORKFLOW.read_text()) env = workflow.get("env", {}) + toml_wheel_tag = toml_data.get("wheel_tag", {}) toml_sha = toml_data.get("sha256", {}) # (label, expected-from-toml, actual-from-workflow) checks: list[tuple[str, str, str | None]] = [ ("HYPERD_VERSION", str(toml_data["version"]), env.get("HYPERD_VERSION")), - ("HYPERD_BUILD_ID", str(toml_data["build_id"]), env.get("HYPERD_BUILD_ID")), ] errors: list[str] = [] @@ -51,13 +58,22 @@ def main() -> int: slug = entry.get("hyperd-slug") if slug is None: continue - expected = toml_sha.get(slug) - if expected is None: + expected_tag = toml_wheel_tag.get(slug) + if expected_tag is None: + errors.append( + f'matrix slug "{slug}" has no [wheel_tag]."{slug}" entry in {TOML.name}' + ) + else: + checks.append( + (f"wheel_tag[{slug}]", expected_tag, entry.get("hyperd-wheel-tag")) + ) + expected_sha = toml_sha.get(slug) + if expected_sha is None: errors.append( f'matrix slug "{slug}" has no [sha256]."{slug}" entry in {TOML.name}' ) continue - checks.append((f"sha256[{slug}]", expected, entry.get("hyperd-sha256"))) + checks.append((f"sha256[{slug}]", expected_sha, entry.get("hyperd-sha256"))) for label, expected, actual in checks: if actual == expected: @@ -72,8 +88,9 @@ def main() -> int: sys.stdout.flush() print( f"\n{WORKFLOW.name} is out of sync with {TOML.name}. " - "Update the workflow's env vars and matrix sha256s to match the toml " - "(or vice versa) so npm bundles the same hyperd as crates.io.", + "Update the workflow's env vars and matrix wheel tags / sha256s to " + "match the toml (or vice versa) so npm bundles the same hyperd as " + "crates.io.", file=sys.stderr, ) return 1 diff --git a/.github/workflows/npm-build-publish.yml b/.github/workflows/npm-build-publish.yml index 15897370..6ab14573 100644 --- a/.github/workflows/npm-build-publish.yml +++ b/.github/workflows/npm-build-publish.yml @@ -23,8 +23,11 @@ permissions: env: CARGO_TERM_COLOR: always + # hyperd is bundled from the PyPI `tableauhyperapi` wheel. This version and + # the per-platform wheel tags / sha256s in the build-npm matrix below must + # match hyperdb-bootstrap/hyperd-version.toml — enforced by + # .github/scripts/verify-npm-hyperd-pin.py. HYPERD_VERSION: "0.0.26479" - HYPERD_BUILD_ID: "r96880f6a" jobs: verify-ci: @@ -92,27 +95,35 @@ jobs: fail-fast: false matrix: include: + # hyperd-wheel-tag is the PyPI platform tag for this slug. It is NOT + # derivable from HYPERD_VERSION, so it has to be carried explicitly; + # a wrong tag yields a 404 rather than a version mismatch. + # hyperd-sha256 is the digest of the downloaded .whl archive. - platform: darwin-arm64 os: macos-14 target: aarch64-apple-darwin hyperd-slug: macos-arm64 - hyperd-sha256: "65bd021b3d3470ac74728ec287866a3ee0dfd806daf2589670feb3580955ee95" + hyperd-wheel-tag: macosx_13_0_arm64 + hyperd-sha256: "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" # TODO: re-enable when macos-13 runners are more available # - platform: darwin-x64 # os: macos-13 # target: x86_64-apple-darwin # hyperd-slug: macos-x86_64 - # hyperd-sha256: "6690669c8a6a6c7c6794c101beb31b83e1589d76cb14b0c817523069591f694c" + # hyperd-wheel-tag: macosx_10_11_x86_64 + # hyperd-sha256: "960e276028137847a3870695d9c2d5a1392c173b1e119ff1146d24a75deca71a" - platform: linux-x64-gnu os: ubuntu-latest target: x86_64-unknown-linux-gnu hyperd-slug: linux-x86_64 - hyperd-sha256: "c20be5b6874d319c7db01dcec763d7e65ae2483c0becc75f2914a58accc4f932" + hyperd-wheel-tag: manylinux2014_x86_64 + hyperd-sha256: "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc" - platform: win32-x64-msvc os: windows-latest target: x86_64-pc-windows-msvc hyperd-slug: windows-x86_64 - hyperd-sha256: "3a400508e79c67ce9dcd8bd317ab164a61b5aed8ca880031ed4bbe6621514e1d" + hyperd-wheel-tag: win_amd64 + hyperd-sha256: "7a4f96d2a22351e944fea6db5d03ab5272ad4c0577acc987bfcb3739ed639502" runs-on: ${{ matrix.os }} defaults: run: @@ -154,16 +165,20 @@ jobs: - name: Download and verify hyperd env: SLUG: ${{ matrix.hyperd-slug }} + WHEEL_TAG: ${{ matrix.hyperd-wheel-tag }} EXPECTED_SHA256: ${{ matrix.hyperd-sha256 }} run: | set -euo pipefail - URL="https://downloads.tableau.com/tssoftware/tableauhyperapi-java-${SLUG}-release-main.${HYPERD_VERSION}.${HYPERD_BUILD_ID}.zip" - echo "Downloading: $URL" - curl --fail --silent --show-error --location --output hyperd-archive.zip "$URL" + # The engine ships inside the PyPI `tableauhyperapi` wheel, which is + # a zip. This URL 302-redirects to the content-addressed download. + URL="https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-${HYPERD_VERSION}-py3-none-${WHEEL_TAG}.whl" + echo "Downloading hyperd for ${SLUG}: $URL" + curl --fail --silent --show-error --location --output hyperd-archive.whl "$URL" + # Digest is over the downloaded archive, not the extracted binary. if command -v sha256sum &>/dev/null; then - ACTUAL_SHA256=$(sha256sum hyperd-archive.zip | awk '{print $1}') + ACTUAL_SHA256=$(sha256sum hyperd-archive.whl | awk '{print $1}') else - ACTUAL_SHA256=$(shasum -a 256 hyperd-archive.zip | awk '{print $1}') + ACTUAL_SHA256=$(shasum -a 256 hyperd-archive.whl | awk '{print $1}') fi if [[ "$ACTUAL_SHA256" != "$EXPECTED_SHA256" ]]; then echo "::error::SHA256 mismatch: expected $EXPECTED_SHA256, got $ACTUAL_SHA256" @@ -175,8 +190,11 @@ jobs: run: | set -euo pipefail mkdir -p hyperd-extracted - unzip -q hyperd-archive.zip -d hyperd-raw - # Find the lib/hyper or bin/hyper directory (matches extract.rs logic) + unzip -q hyperd-archive.whl -d hyperd-raw + # Find the lib/hyper or bin/hyper directory (matches extract.rs + # logic). The wheel lays the engine out at tableauhyperapi/bin/hyper/ + # on every platform, so in practice the bin/hyper branch matches; + # the lib/hyper branch is kept for parity with extract.rs. HYPER_DIR=$(find hyperd-raw -type d -name "hyper" -path "*/lib/hyper" -o -type d -name "hyper" -path "*/bin/hyper" | head -1) if [[ -z "$HYPER_DIR" ]]; then echo "::error::Could not find lib/hyper or bin/hyper in archive" @@ -195,6 +213,10 @@ jobs: - name: Extract LICENSE from hyperd archive run: | set -euo pipefail + # The wheel's tableauhyperapi-.dist-info/ carries LICENSE, + # NOTICES.txt and HYPER_API_OSS_disclosure.txt, so this glob matches. + # The heredoc below is the fallback if that ever changes, so a miss + # is non-fatal. LICENSE_FILE=$(find hyperd-raw -iname "LICENSE*" -o -iname "NOTICE*" | head -1) if [[ -n "$LICENSE_FILE" ]]; then cp "$LICENSE_FILE" LICENSE-HYPERD diff --git a/.github/workflows/verify-hyperd-pin.yml b/.github/workflows/verify-hyperd-pin.yml index 55ac3968..7086c466 100644 --- a/.github/workflows/verify-hyperd-pin.yml +++ b/.github/workflows/verify-hyperd-pin.yml @@ -1,8 +1,9 @@ name: verify-hyperd-pin -# HEAD every platform URL for the pinned Hyper release. Catches -# Tableau yanking / renaming an archive, and catches typos in -# hyperd-version.toml on PRs that touch the pin. +# HEAD every platform URL for the pinned Hyper release. The archives are the +# PyPI `tableauhyperapi` wheels, so this catches a wheel being yanked and +# catches typos in hyperd-version.toml (version or wheel tag — a wrong tag is +# just a URL that 404s) on PRs that touch the pin. on: push: @@ -46,7 +47,8 @@ jobs: - name: Verify npm-build-publish.yml hyperd pin matches the toml # The release workflow bundles hyperd into the npm packages from its - # OWN hardcoded version/build_id/sha256s, decoupled from the toml. + # OWN hardcoded version + per-platform wheel tags/sha256s, decoupled + # from the toml. # 0.7.1 shipped npm with the stale 0.0.25080 engine because only the # toml was bumped. This guard fails the build if they ever drift again. run: python3 .github/scripts/verify-npm-hyperd-pin.py diff --git a/AGENTS.md b/AGENTS.md index 644ced28..2d816a2d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,10 +9,14 @@ running `make download-hyperd` (or `.\build.ps1 download-hyperd`). The implementation lives in the [`hyperdb-bootstrap`](hyperdb-bootstrap/) crate; the pinned release is baked into [`hyperdb-bootstrap/hyperd-version.toml`](hyperdb-bootstrap/hyperd-version.toml). -Bumping `hyperd` = edit that file (version + build_id + per-platform sha256s), -then let the `fix(bootstrap):` commit drive the version via release-please (the -crate uses `version.workspace = true` — don't hand-edit a crate version). The -full repeatable procedure — verify the pin, run the suite, A/B benchmark +Bumping `hyperd` = edit that file (`version` + the four `[wheel_tag]` entries + +the four per-platform sha256s — there is no `build_id`), then let the +`fix(bootstrap):` commit drive the version via release-please (the crate uses +`version.workspace = true` — don't hand-edit a crate version). `hyperd` comes +out of the PyPI `tableauhyperapi` wheels, so **you don't compute the digests**: +read them straight off the JSON API with +`curl -s https://pypi.org/pypi/tableauhyperapi//json | jq -r '.urls[] | "\(.filename) \(.digests.sha256)"'`. +The full repeatable procedure — verify the pin, run the suite, A/B benchmark against the previous pin, and log the result — is captured in the [`update-hyperd-release`](.claude/skills/update-hyperd-release/SKILL.md) skill. diff --git a/Cargo.lock b/Cargo.lock index 951a2ea2..04bfe7b6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1893,10 +1893,8 @@ version = "1.0.0-rc.1" dependencies = [ "anyhow", "clap", - "regex", - "reqwest", - "rustls", "serde", + "serde_json", "sha2 0.11.0", "tempfile", "thiserror 2.0.20", @@ -3278,9 +3276,7 @@ checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" dependencies = [ "base64 0.22.1", "bytes", - "futures-channel", "futures-core", - "futures-util", "http", "http-body", "http-body-util", diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 0f83bc84..f1c59389 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -384,10 +384,9 @@ in companion crates: This installs `hyperd` at `.hyperd/current/hyperd` (auto-discovered by `make`/`build.ps1` — no `HYPERD_PATH` needed). The pinned release is baked into [`hyperdb-bootstrap/hyperd-version.toml`](hyperdb-bootstrap/hyperd-version.toml); - to upgrade, edit that file (version + build_id + per-platform sha256s) - and bump the crate version. Pass `ARGS="--latest"` to fetch the newest - release via best-effort scraping, or `ARGS="--version X --build-id Y"` - for an ad-hoc pin. + to upgrade, edit that file (version + the four `[wheel_tag]` entries + + per-platform sha256s) and bump the crate version. Pass + `ARGS="--version X"` for an ad-hoc pin. If you already have a `hyperd` elsewhere, set `HYPERD_PATH` instead: @@ -420,11 +419,9 @@ in companion crates: after the `build.ps1 download-hyperd` command): ```bash - # Best-effort scrape of the latest release (skips sha256 verification). - make download-hyperd ARGS="--latest" - - # Pin to a specific release ad-hoc. - make download-hyperd ARGS="--version 0.0.24457 --build-id rc36858b6" + # Pin to a specific release ad-hoc. Inherits the builtin pin's wheel tags + # and carries no digests, so the download is unverified (logs a WARN). + make download-hyperd ARGS="--version 0.0.26359" # Install to a custom location, e.g. shared across repos. make download-hyperd ARGS="--dest /opt/hyperd" @@ -435,9 +432,10 @@ in companion crates: Bumping the baked-in pin is an edit to [`hyperdb-bootstrap/hyperd-version.toml`](hyperdb-bootstrap/hyperd-version.toml) - (version + build_id + per-platform sha256s) plus a crate version bump. - `build.rs` validates the file on every compile, and the - `verify-hyperd-pin` CI workflow confirms the URLs resolve. + (version + the four `[wheel_tag]` entries + per-platform sha256s, all read + off the PyPI JSON API) plus a crate version bump. `build.rs` validates the + file on every compile, and the `verify-hyperd-pin` CI workflow confirms the + URLs resolve and that each pinned digest matches the one PyPI publishes. 4. **Windows only**: Install Visual Studio Build Tools with "Desktop development with C++" workload (provides the MSVC linker, not for C++ compilation). diff --git a/Makefile b/Makefile index dc547842..a6f3e954 100644 --- a/Makefile +++ b/Makefile @@ -141,13 +141,14 @@ test-api-release: examples: ./run_all_examples.sh -# Download hyperd from Tableau's Hyper C++ API release into .hyperd/current/ -# Forward extra flags via ARGS, e.g. `make download-hyperd ARGS="--latest"`. +# Download hyperd from the PyPI tableauhyperapi wheel into .hyperd/current/ +# Forward extra flags via ARGS, e.g. `make download-hyperd ARGS="--force"`. download-hyperd: cargo run --release -p hyperdb-bootstrap --bin hyperdb-bootstrap -- download $(ARGS) -# Network-only check: HEAD each supported platform URL for the pinned -# release. Intended for CI (nightly + on PRs touching hyperd-version.toml). +# Network-only check: probe each supported platform's wheel URL for the pinned +# release and cross-check its digest against PyPI. Intended for CI (nightly + +# on PRs touching hyperd-version.toml). verify-hyperd-pin: cargo run --release -p hyperdb-bootstrap --bin hyperdb-bootstrap -- verify $(ARGS) diff --git a/hyperdb-bootstrap/CHANGELOG.md b/hyperdb-bootstrap/CHANGELOG.md index 61cb6d64..e1c317aa 100644 --- a/hyperdb-bootstrap/CHANGELOG.md +++ b/hyperdb-bootstrap/CHANGELOG.md @@ -9,6 +9,56 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- **BREAKING: `hyperd` is now downloaded from the PyPI `tableauhyperapi` + wheels instead of Tableau's Hyper Java API zips**, and the pin moves to + `0.0.26479`. The motivation is that a wheel URL is fully constructible from + the version plus the platform's wheel tag: + + ```text + https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-{version}-py3-none-{wheel_tag}.whl + ``` + + whereas the Tableau zip filenames embed an opaque `build_id` (`r07abb490`) + that cannot be derived from the version — which is why bumping the pin + used to require scraping an HTML page. And because **PyPI publishes a + sha256 per file**, the four pinned digests are now read off + `https://pypi.org/pypi/tableauhyperapi//json` rather than produced + by downloading four ~80 MB archives and hashing them by hand. The digests + are still committed: a hash in git is an attestation independent of the + host serving the bytes. + + **The bytes are unchanged.** The `hyperd` inside the `macosx_13_0_arm64` + wheel is bit-identical to the one extracted from the corresponding Java + zip — sha256 + `aef5c81970bb4d84d06fb9513d5ffd722526fce779632a0c5f63d87b6450e478`, + 277,836,448 bytes. Same build, different envelope; the wheels are ~3.6–4.5% + smaller. Both binaries report `minos 13.0`, so the `macosx_13_0` wheel tag + is **not** a raised support floor and no contributor loses support. + + **BREAKING** consequences for the pin format and the public API: + + - `hyperd-version.toml` now holds `version`, a `[wheel_tag]` table, and + `[sha256]`. **`build_id` is gone.** The wheel tags are pin data rather + than Rust constants because they are not stable across releases (arm64 + wheels only exist from `0.0.19484`; a future macOS floor bump would change + `macosx_13_0_arm64`) and a wrong tag yields a *silent 404* — keeping them + in the pin makes any such change a visible pin edit. + - `build.rs` now validates `version`, the `[wheel_tag]` entries, and the + sha256 shapes; it no longer validates `build_id`. + - The install layout is keyed on the version alone: the versioned cache + directory is `/0.0.26479/` (was `/0.0.26479.r96880f6a/`) and + `current/VERSION` contains just `0.0.26479`. + - `hyperd` is extracted from `tableauhyperapi/bin/hyper/` inside the wheel + rather than `lib/hyper/` inside the zip. + - New `PinnedRelease::wheel_tag_for(Platform) -> Option<&str>`. + - `url::build_download_url` is now **fallible**, returning + `Result` — a platform with no pinned wheel tag is an error + (`Error::MissingWheelTag`, a new variant). + - `verify` now additionally cross-checks every pinned sha256 against the + digest PyPI publishes for that exact wheel filename, on top of HEAD-ing + the four download URLs. It therefore validates the exact pinned bytes + rather than merely that the CDN serves *something* at that path. + - **BREAKING:** the minimum supported Rust version is now **1.88**, up from 1.81, and the crate is compiled with **edition 2024**. 1.88 is the version Red Hat Enterprise Linux 9.7 ships as `rust-toolset`. @@ -28,6 +78,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/). Guarded by a unit test that builds a `reqwest` client, which needs no network because the failure is a panic inside `build()`. + **This entire workaround is retired by the PyPI migration above**, which + drops `reqwest` and `rustls` from the crate: there is no in-process HTTP + client left, so there is no crypto provider to select. `ensure_crypto_provider`, + the `rustls-no-provider` feature, and the ring pin are all gone, and the + feature-unification pressure this bullet describes no longer exists for the + rest of the workspace. + - **Bump the pinned `hyperd` release to `0.0.26359` (`r07abb490`).** This supersedes the never-shipped `0.0.26225` bump attempt (PR #219), which was held because `0.0.26225` deadlocked on Apple Silicon Macs running macOS @@ -43,13 +100,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/). thermally on a laptop. Verified native arm64 and all 1485 workspace tests pass; the macOS-14 CI runner is green (the deadlock is gone). -- **Bump the pinned `hyperd` release to `0.0.26479` (`r96880f6a`).** Updates the - version, build id, and all four per-platform sha256s in `hyperd-version.toml`. - **Verified native arm64** — `file` reports `Mach-O 64-bit executable arm64` - for the `macos-arm64` bundle's `lib/hyper/hyperd`, so the reason this crate - pulls the Java bundle rather than the C++ one still holds. All four platform - URLs return HTTP 200 at the new pin, and all **1586** workspace tests pass - against the new engine. Performance (interleaved same-session A/B vs the +- **Bump the pinned `hyperd` release to `0.0.26479` (engine build + `r96880f6a`).** The engine bump rides with the PyPI migration above, which is + what now carries the pin. **Verified native arm64** — `file` reports `Mach-O + 64-bit executable arm64` for the extracted `macos-arm64` binary. All four + platform URLs return HTTP 200 at the new pin, and the workspace test suite + passes against the new engine. Performance (interleaved same-session A/B vs the previous `0.0.26359` pin, medians of 5 runs per engine at 100M rows, single-connection): the async Arrow insert path **more than doubles** (`AsyncArrowInserter` **+127%**, 30.4 → 68.9 M rows/s), reproduced as **+75%** @@ -61,6 +117,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/). 17–41% run to run, enough to have read **+23% at 100M and −20% at 10M for the same workload in the same session**. +### Removed + +Everything here follows from the move to PyPI wheels described under +**Changed**: the build id no longer exists as a concept, and nothing needs to +be discovered by scraping. + +- **BREAKING: the `--build-id` CLI flag is removed.** `--version X` on its own + is now a complete version source: it inherits the builtin pin's + `[wheel_tag]` values and carries no digests, so the download is unverified + and logs a WARN. The four wheel tags are unchanged from `0.0.19484` through + `0.0.26479`, so this covers any realistic ad-hoc pin or A/B baseline; for a + release whose tags differ, use `--version-file` with a full pin. +- **BREAKING: the `--latest` CLI flag, the `VersionSource::ScrapeLatest` + variant, and the whole `scrape` module are removed.** The scraper had been + broken for three-plus releases without anyone noticing, because its tests + passed against a synthetic fixture rather than the live page. It had two + independent defects: the heading regex expected `

VERSION [DATE]

`, + but Docusaurus renders `0.0.26479 [September 3 2026]`, which `\s*` + cannot span; and the build-id capture hardcoded `(rc[a-z0-9]+)` while every + build id since `0.0.24457` has been `r` followed by hex. It is deleted + rather than fixed — with a constructible URL and published digests, there is + nothing left for it to do. Version-source precedence is now, highest to + lowest: `--version X`, `--version-file PATH`, an auto-discovered + `./hyperd-version.toml`, then the compiled-in default. +- **BREAKING: `PinnedRelease::build_id` and `InstalledHyperd::build_id` are + removed**, as is `PinnedRelease::version_tag()`. Use `.version`, which is + now the only release identifier, and `PinnedRelease::wheel_tag_for()` for + the per-platform tag. +- **BREAKING: the `Error::Http`, `Error::HttpStatus`, and `Error::ScrapeFailed` + variants are removed.** All three existed only to serve the scraper. +- **BREAKING: the `regex`, `reqwest`, and `rustls` dependencies are dropped + entirely.** `download.rs` and `verify.rs` already shell out to `curl` — + deliberately, because Akamai bot protection blocked non-browser TLS stacks + from GitHub runner IPs — so removing the scraper leaves no in-process HTTP + client behind. + ### Fixed - The "no hyperd installed" error suggested `hyperd-bootstrap download`, but @@ -68,17 +160,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/). Doc comments across the crate carried the same pre-rename name, as did the `hyperdb-bootstrap/hyperd-version.toml` path in `release.rs`. -- **Download `hyperd` from the Java API bundle instead of the C++ bundle.** - Tableau's C++ `macos-arm64` zip ships an **x86_64** `hyperd` (an upstream - packaging defect), so on Apple Silicon the extracted `hyperd` only ran - under Rosetta. The Java `macos-arm64` bundle carries a native arm64 - `hyperd`. The bundles share an identical URL template (only the - `java`/`cxx` token differs) and an identical internal layout - (`lib/hyper/hyperd`), so the switch is confined to the URL token and the - pinned per-platform sha256s in `hyperd-version.toml`. The other three - platforms (`macos-x86_64`, `linux-x86_64`, `windows-x86_64`) are - unaffected in architecture but now also come from the Java bundle for - consistency. +- **The extracted macOS arm64 `hyperd` is a native arm64 binary.** Earlier + releases of Tableau's C++ `macos-arm64` zip shipped an **x86_64** `hyperd` + (an upstream packaging defect), so on Apple Silicon the extracted binary + only ran under Rosetta; this crate switched to the Java bundle to get a + native one. Tableau fixed the C++ packaging in `0.0.26225`, and from that + release the C++ and Java binaries are byte-identical, so the bundle choice + stopped mattering — and the move to PyPI wheels under **Changed** supersedes + it entirely. The wheels carry the same native arm64 build. ## [0.1.1] - 2026-05-13 diff --git a/hyperdb-bootstrap/Cargo.toml b/hyperdb-bootstrap/Cargo.toml index 3cfa5ced..1119bc38 100644 --- a/hyperdb-bootstrap/Cargo.toml +++ b/hyperdb-bootstrap/Cargo.toml @@ -3,7 +3,7 @@ name = "hyperdb-bootstrap" version.workspace = true edition.workspace = true rust-version.workspace = true -description = "Download and install the hyperd executable from Tableau's Hyper C++ API release packages" +description = "Download and install the hyperd executable from the PyPI tableauhyperapi wheels" license.workspace = true repository.workspace = true homepage.workspace = true @@ -24,16 +24,14 @@ default = ["cli"] cli = ["dep:clap", "dep:anyhow", "dep:tracing-subscriber"] [dependencies] -reqwest = { version = "0.13", default-features = false, features = ["blocking", "rustls-no-provider"] } -# `rustls-no-provider` above ships no crypto provider, so we install `ring` -# ourselves before building a client. See `scrape::ensure_crypto_provider`. -rustls = { workspace = true } +# No in-process HTTP client: `download.rs` and `verify.rs` shell out to `curl`, +# so this crate needs no TLS stack (and no rustls crypto-provider dance). zip = { workspace = true } sha2 = { workspace = true } serde = { workspace = true } +serde_json = { workspace = true } toml = { workspace = true } thiserror = { workspace = true } -regex = { workspace = true } tempfile = { workspace = true } tracing = { workspace = true } clap = { workspace = true, optional = true } diff --git a/hyperdb-bootstrap/README.md b/hyperdb-bootstrap/README.md index 9bd9bd29..1bfc0c94 100644 --- a/hyperdb-bootstrap/README.md +++ b/hyperdb-bootstrap/README.md @@ -1,22 +1,42 @@ # hyperdb-bootstrap -Download and install the `hyperd` executable from Tableau's Hyper Java API -release packages. Ships as both a CLI binary and a library. - -The `hyperd` server isn't on crates.io — it's a prebuilt binary distributed -inside Tableau's Hyper API zips at -. This crate automates -the "download the right zip for your platform, extract `hyperd` out of -`lib/hyper/`, put it somewhere useful" step so contributors and CI can -bootstrap with a single command. - -> **Why the Java bundle, not C++?** Tableau publishes `hyperd` inside both -> the C++ and Java API zips. The C++ `macos-arm64` zip currently ships an -> **x86_64** `hyperd` (an upstream packaging defect), so on Apple Silicon it -> would only run under Rosetta. The Java `macos-arm64` zip carries a native -> arm64 `hyperd`. The two bundles are otherwise identical for our purposes -> (same URL template, same `lib/hyper/hyperd` layout), so this crate pulls -> from the Java bundle on every platform for consistency. +Download and install the `hyperd` executable from the PyPI `tableauhyperapi` +wheels. Ships as both a CLI binary and a library. + +The `hyperd` server isn't on crates.io — it's a prebuilt binary that Tableau +distributes inside its Hyper API packages. This crate takes the PyPI ones: +each `tableauhyperapi` wheel carries `hyperd` at +`tableauhyperapi/bin/hyper/hyperd` (`hyperd.exe` plus `crashdumper.exe` on +Windows). This crate automates the "download the right wheel for your +platform, extract `hyperd` out of it, put it somewhere useful" step so +contributors and CI can bootstrap with a single command. + +> **Why the PyPI wheels?** Two properties that no other distribution +> channel offers together. First, **the download URL is constructible** from +> nothing but the version and the platform's wheel tag: +> +> ```text +> https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-{version}-py3-none-{wheel_tag}.whl +> ``` +> +> Tableau's own download filenames embed an opaque build id (`r07abb490`) +> that cannot be derived from the version, so every bump previously required +> scraping an HTML page to discover it. Second, **PyPI publishes a sha256 per +> file**, so the pinned digests are now *read off an API* rather than produced +> by downloading four ~80 MB archives and hashing them by hand. +> +> The bytes are the same either way: the `hyperd` inside the +> `macosx_13_0_arm64` wheel is bit-identical (sha256 +> `aef5c819…6450e478`, 277,836,448 bytes) to the one this crate used to pull +> from Tableau. Same build, different envelope — and the wheels are ~3.6–4.5% +> smaller. Both report `minos 13.0`, so the `macosx_13_0` wheel tag is not a +> raised support floor and no contributor loses support. +> +> *Historical note, since older revisions of this file argued it at length:* +> Tableau's C++ `macos-arm64` zip did once ship an **x86_64** `hyperd`, which +> is why this crate used to take the Java bundle specifically. Tableau fixed +> that in `0.0.26225`; from that release the C++ and Java binaries are +> byte-identical, so the distinction no longer explains anything. ## Install @@ -36,18 +56,14 @@ hyperdb-bootstrap download --dest /opt/hyperd # Force a re-download even if the version is already cached hyperdb-bootstrap download --force -# Scrape the latest release off the public releases page (best-effort, -# skips sha256 verification). -hyperdb-bootstrap download --latest - -# Install a specific release by version + build id -hyperdb-bootstrap download --version 0.0.24457 --build-id rc36858b6 +# Install a specific release ad-hoc — just the version, no build id needed +hyperdb-bootstrap download --version 0.0.26359 # Use an external pinned-version TOML instead of the baked-in default hyperdb-bootstrap download --version-file ./my-hyperd.toml -# HEAD each supported platform's URL for the pinned release — used by CI -# to catch Tableau yanks/renames early. Exits non-zero on any failure. +# Check the pinned release for every supported platform — used by CI to +# catch yanks/renames early. Exits non-zero on any failure. hyperdb-bootstrap verify # Print the installed binary's path @@ -57,13 +73,18 @@ hyperdb-bootstrap which hyperdb-bootstrap version ``` +`--version X` on its own inherits the builtin pin's `[wheel_tag]` values and +carries **no** digests, so the download is unverified and a WARN is logged. The +four wheel tags are unchanged from `0.0.19484` through `0.0.26479`, so this works +for any realistic ad-hoc pin or benchmark baseline. For a release whose wheel +tags differ, write a full pin file and pass `--version-file`. + **Version-source precedence (highest → lowest):** -1. `--version X --build-id Y` -2. `--latest` -3. `--version-file PATH` -4. `./hyperd-version.toml` (auto-discovered in current dir) -5. Compiled-in default shipped with this crate +1. `--version X` +2. `--version-file PATH` +3. `./hyperd-version.toml` (auto-discovered in current dir) +4. Compiled-in default shipped with this crate ## Library @@ -88,39 +109,54 @@ synchronous applications. ## Build-time guarantees - **Compile-time pin validation.** `build.rs` parses `hyperd-version.toml` - on every build and fails fast if `version`/`build_id` are missing or - malformed, if a sha256 isn't 64 hex chars, or if an unknown platform - key appears. Empty sha256 strings are allowed (skip verification for + on every build and fails fast if `version` is missing or malformed, if a + `[wheel_tag]` entry is missing or empty for a supported platform, if a + sha256 isn't 64 hex chars, or if an unknown platform key appears in + either table. Empty sha256 strings are allowed (skip verification for that platform) but surface a `cargo:warning` so nobody ships a release with missing hashes by accident. -- **URL reachability (`verify` subcommand).** `hyperdb-bootstrap verify` - HEADs every supported platform's download URL for the pinned release. - It's wired into CI (see +- **Pin verification (`verify` subcommand).** `hyperdb-bootstrap verify` + does two things for every supported platform: it HEADs the download URL + for the pinned release, and it cross-checks the pinned sha256 against the + digest PyPI publishes for that exact wheel filename (via + `https://pypi.org/pypi/tableauhyperapi//json`). The second check + is what makes this meaningful — it validates the exact pinned bytes rather + than merely that the CDN serves *something* at that path. It's wired into + CI (see [`.github/workflows/verify-hyperd-pin.yml`](../.github/workflows/verify-hyperd-pin.yml)) so yanked or renamed archives fail a PR instead of the next contributor's `make download-hyperd`. ## Supported platforms -| OS | Arch | Slug | -|---------|---------|-------------------| -| macOS | arm64 | `macos-arm64` | -| macOS | x86_64 | `macos-x86_64` | -| Linux | x86_64 | `linux-x86_64` | -| Windows | x86_64 | `windows-x86_64` | +| OS | Arch | Slug | Wheel tag | +|---------|---------|-------------------|--------------------------| +| macOS | arm64 | `macos-arm64` | `macosx_13_0_arm64` | +| macOS | x86_64 | `macos-x86_64` | `macosx_10_11_x86_64` | +| Linux | x86_64 | `linux-x86_64` | `manylinux2014_x86_64` | +| Windows | x86_64 | `windows-x86_64` | `win_amd64` | Any other `(OS, ARCH)` errors out with a clear message. +The wheel tags are pin *data*, not constants in the Rust source: they aren't +guaranteed stable across releases (arm64 wheels only exist from `0.0.19484`, +and a future macOS floor bump would change `macosx_13_0_arm64`), and a wrong +tag produces a **silent 404** rather than a clear error. Keeping them in +`hyperd-version.toml` makes any such change a visible pin edit. + ## Install layout +The versioned cache directory and `current/VERSION` are keyed on the version +alone — there is no build id in the path any more. + ```text / -├── 0.0.24457.rc36858b6/ # versioned cache +├── 0.0.26479/ # versioned cache │ ├── hyperd # hyperd.exe on Windows -│ └── ... # other files shipped under lib/hyper/ +│ └── ... # other files shipped under the wheel's bin/hyper/ └── current/ # fresh copy on each successful run ├── hyperd - └── VERSION # text: "0.0.24457.rc36858b6" + └── VERSION # text: "0.0.26479" ``` `current/` is a file copy, not a symlink — this avoids needing admin diff --git a/hyperdb-bootstrap/build.rs b/hyperdb-bootstrap/build.rs index d6c47ca1..44633e4c 100644 --- a/hyperdb-bootstrap/build.rs +++ b/hyperdb-bootstrap/build.rs @@ -3,9 +3,9 @@ //! Compile-time sanity check for the pinned release metadata. //! -//! If `hyperd-version.toml` fails to parse, is missing `version`/`build_id`, -//! or has a shape the runtime code can't handle, the build fails here — -//! before any contributor ships a broken bump. +//! If `hyperd-version.toml` fails to parse, is missing `version`, lacks a +//! wheel tag for a supported platform, or has a shape the runtime code can't +//! handle, the build fails here — before any contributor ships a broken bump. //! //! Keep this in sync with `release.rs` / `platform.rs`. The check is //! deliberately lightweight (no network, no crates-io deps beyond what @@ -73,22 +73,12 @@ fn main() { "{}: `version` is empty", pin_path.display() ); - assert!( - !pin.build_id.trim().is_empty(), - "{}: `build_id` is empty", - pin_path.display() - ); // Reject stray whitespace / accidental newlines in the URL components. assert!( !pin.version.contains(char::is_whitespace), "{}: `version` contains whitespace", pin_path.display() ); - assert!( - !pin.build_id.contains(char::is_whitespace), - "{}: `build_id` contains whitespace", - pin_path.display() - ); const SUPPORTED: &[&str] = &[ "macos-arm64", @@ -96,13 +86,43 @@ fn main() { "linux-x86_64", "windows-x86_64", ]; - for key in pin.sha256.keys() { + for (table, keys) in [ + ("wheel_tag", pin.wheel_tag.keys()), + ("sha256", pin.sha256.keys()), + ] { + for key in keys { + assert!( + SUPPORTED.contains(&key.as_str()), + "{}: unknown platform key `{}` in [{}]; supported: {:?}", + pin_path.display(), + key, + table, + SUPPORTED + ); + } + } + + // Every supported platform needs a wheel tag: without one the wheel file + // name is not constructible, so `download` would fail at runtime on that + // platform only — exactly the kind of break that should not reach a + // contributor's machine. + for platform in SUPPORTED { + let tag = pin + .wheel_tag + .get(*platform) + .map(|t| t.trim()) + .unwrap_or_default(); assert!( - SUPPORTED.contains(&key.as_str()), - "{}: unknown platform key `{}` in [sha256]; supported: {:?}", + !tag.is_empty(), + "{}: no [wheel_tag] entry for `{}`; every supported platform needs one", pin_path.display(), - key, - SUPPORTED + platform + ); + assert!( + !tag.contains(char::is_whitespace), + "{}: wheel tag for `{}` contains whitespace", + pin_path.display(), + platform ); } for (plat, sha) in &pin.sha256 { @@ -129,7 +149,8 @@ fn main() { #[derive(serde::Deserialize)] struct PinCheck { version: String, - build_id: String, + #[serde(default)] + wheel_tag: HashMap, #[serde(default)] sha256: HashMap, } diff --git a/hyperdb-bootstrap/hyperd-version.toml b/hyperdb-bootstrap/hyperd-version.toml index b6191cb9..3c81ce05 100644 --- a/hyperdb-bootstrap/hyperd-version.toml +++ b/hyperdb-bootstrap/hyperd-version.toml @@ -1,21 +1,33 @@ -# Pinned Hyper **Java** API release used by hyperdb-bootstrap. +# Pinned Hyper API release used by hyperdb-bootstrap. # -# We pull `hyperd` from the Java binding's bundle, NOT the C++ one: the -# C++ macos-arm64 zip ships an x86_64 `hyperd` (upstream packaging defect), -# which only runs under Rosetta on Apple Silicon. The Java macos-arm64 -# bundle carries a native arm64 `hyperd`. Same URL template and same -# internal layout (lib/hyper/hyperd) — only the binding token + sha256s differ. +# `hyperd` is extracted from the PyPI `tableauhyperapi` wheels. The wheel +# carries the same `hyperd` build as Tableau's Java/C++ API bundles, but its +# filename is fully constructible from `version` + the platform's wheel tag, +# so bumping this pin needs no build id and no HTML scraping. # -# Bump these values (and sha256s) when upgrading. Contributors without -# an override get this exact release, so reproducibility depends on it. -version = "0.0.26479" -build_id = "r96880f6a" +# Digests are copied verbatim from the PyPI JSON API — do not compute by hand: +# curl -s https://pypi.org/pypi/tableauhyperapi//json \ +# | jq -r '.urls[]|"\(.filename) \(.digests.sha256)"' +# +# Contributors without an override get this exact release, so reproducibility +# depends on it. +version = "0.0.26479" + +# Wheel platform tag per target. These are NOT derivable from the version and +# are NOT stable across releases: arm64 wheels only exist from 0.0.19484, and a +# future macOS floor bump would change `macosx_13_0_arm64`. A wrong tag yields a +# silent 404, so the tag lives here — where changing it is a visible pin edit — +# rather than hardcoded in Rust. +[wheel_tag] +"macos-arm64" = "macosx_13_0_arm64" +"macos-x86_64" = "macosx_10_11_x86_64" +"linux-x86_64" = "manylinux2014_x86_64" +"windows-x86_64" = "win_amd64" -# sha256 of each platform's .zip. Omit a platform to skip verification -# for that platform (not recommended). Compute with: -# shasum -a 256 tableauhyperapi-java--release-main...zip +# sha256 of each platform's .whl, as published by PyPI. Omit a platform to skip +# verification for that platform (not recommended). [sha256] -"macos-arm64" = "65bd021b3d3470ac74728ec287866a3ee0dfd806daf2589670feb3580955ee95" -"macos-x86_64" = "6690669c8a6a6c7c6794c101beb31b83e1589d76cb14b0c817523069591f694c" -"linux-x86_64" = "c20be5b6874d319c7db01dcec763d7e65ae2483c0becc75f2914a58accc4f932" -"windows-x86_64" = "3a400508e79c67ce9dcd8bd317ab164a61b5aed8ca880031ed4bbe6621514e1d" +"macos-arm64" = "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" +"macos-x86_64" = "960e276028137847a3870695d9c2d5a1392c173b1e119ff1146d24a75deca71a" +"linux-x86_64" = "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc" +"windows-x86_64" = "7a4f96d2a22351e944fea6db5d03ab5272ad4c0577acc987bfcb3739ed639502" diff --git a/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs b/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs index 101c3fae..0e7da3fe 100644 --- a/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs +++ b/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs @@ -6,8 +6,8 @@ //! Subcommands: //! - `download` — install `hyperd` under `.hyperd//` and refresh //! `.hyperd/current/`. -//! - `verify` — HEAD each platform URL for the pinned release to confirm -//! the CDN is still serving it (CI guard against silent yanks). +//! - `verify` — probe each platform's wheel URL and cross-check its pinned +//! digest against PyPI (CI guard against silent yanks and digest drift). //! - `which` — print the path of the currently-installed `hyperd`. //! - `version` — print the pinned release metadata. @@ -36,8 +36,9 @@ struct Cli { enum Command { /// Download and install hyperd into `.hyperd/` (or --dest). Download(DownloadArgs), - /// HEAD every platform URL for the pinned release to check they're - /// still reachable. Useful as a CI guard against silent yanks. + /// Check every platform's wheel URL is still reachable and that its + /// pinned digest still matches what PyPI publishes. Useful as a CI + /// guard against silent yanks and digest drift. Verify(VerifyArgs), /// Print the path of the currently-installed hyperd (if any). Which(WhichArgs), @@ -48,7 +49,7 @@ enum Command { #[derive(Args)] #[command(group( ArgGroup::new("version_src") - .args(["latest", "version", "version_file"]) + .args(["version", "version_file"]) .required(false) .multiple(false) ))] @@ -61,19 +62,11 @@ struct DownloadArgs { #[arg(long)] force: bool, - /// Scrape the latest release from the Tableau releases page. - /// Best-effort; skips sha256 verification. + /// Explicit version to install (e.g. 0.0.26359). Reuses the pinned wheel + /// tags and skips sha256 verification, since digests are version-specific. #[arg(long)] - latest: bool, - - /// Explicit version to install (e.g. 0.0.24457). Requires --build-id. - #[arg(long, requires = "build_id")] version: Option, - /// Build id for --version (e.g. rc36858b6). - #[arg(long, requires = "version")] - build_id: Option, - /// Path to an external pinned-version TOML file. #[arg(long, value_name = "PATH")] version_file: Option, @@ -128,22 +121,22 @@ fn run_download(args: DownloadArgs) -> Result<()> { )] fn pick_version_source(args: &DownloadArgs) -> Result { // Precedence: - // 1. --version + --build-id (Explicit) - // 2. --latest (ScrapeLatest) - // 3. --version-file PATH (TomlFile) - // 4. ./hyperd-version.toml (auto-discovered TomlFile) - // 5. builtin - if let (Some(v), Some(b)) = (&args.version, &args.build_id) { + // 1. --version X (Explicit) + // 2. --version-file PATH (TomlFile) + // 3. ./hyperd-version.toml (auto-discovered TomlFile) + // 4. builtin + if let Some(v) = &args.version { + // Inherit the builtin pin's wheel tags — they are stable across every + // release that publishes an arm64 wheel (0.0.19484 onward), so an + // ad-hoc version override does not also need to restate them. Drop the + // digests: they are specific to the pinned version's files. let release = PinnedRelease { version: v.clone(), - build_id: b.clone(), + wheel_tag: PinnedRelease::builtin().wheel_tag, sha256: std::collections::HashMap::default(), }; return Ok(VersionSource::Explicit(release)); } - if args.latest { - return Ok(VersionSource::ScrapeLatest); - } if let Some(path) = &args.version_file { return Ok(VersionSource::TomlFile(path.clone())); } @@ -178,9 +171,19 @@ fn run_which(args: WhichArgs) -> Result<()> { )] fn run_version() -> Result<()> { let r = PinnedRelease::builtin(); - println!("pinned version: {}", r.version); - println!("pinned build_id: {}", r.build_id); - println!("version tag: {}", r.version_tag()); + println!("pinned version: {}", r.version); + for platform in [ + Platform::MacosArm64, + Platform::MacosX86_64, + Platform::LinuxX86_64, + Platform::WindowsX86_64, + ] { + println!( + " {:<16} {}", + platform.to_string(), + r.wheel_tag_for(platform).unwrap_or("") + ); + } Ok(()) } @@ -190,51 +193,39 @@ fn run_verify(args: VerifyArgs) -> Result<()> { .with_context(|| format!("loading {}", path.display()))?, None => PinnedRelease::builtin(), }; - println!("verifying hyperd {}...", release.version_tag()); - let outcomes = verify_release(&release).context("HEAD requests failed")?; + println!("verifying hyperd {}...", release.version); + let outcomes = verify_release(&release).context("probing platform URLs failed")?; let mut all_ok = true; for o in &outcomes { - match (o.status, &o.error) { - (Some(status), _) if o.ok() => { - println!( - " OK {:<16} [{status}] {}", - o.platform.to_string(), - o.url - ); - } - (Some(status), _) => { - all_ok = false; - println!( - " FAIL {:<16} [{status}] {}", - o.platform.to_string(), - o.url - ); - } - (None, Some(err)) => { - all_ok = false; - println!( - " FAIL {:<16} [network error] {} ({err})", - o.platform.to_string(), - o.url - ); - } - (None, None) => unreachable!("verify_release always sets status or error"), + let label = if o.ok() { "OK " } else { "FAIL" }; + if !o.ok() { + all_ok = false; } + let http = match (o.status, &o.error) { + (Some(status), _) => format!("{status}"), + (None, Some(err)) => format!("network error: {err}"), + (None, None) => unreachable!("verify_release always sets status or error"), + }; + println!( + " {label} {:<16} [{http}] {}", + o.platform.to_string(), + o.url + ); + println!(" {}", o.digest); } if !all_ok { - anyhow::bail!("one or more platform URLs failed to resolve"); + anyhow::bail!("one or more platforms failed URL or digest verification"); } - println!("all platforms reachable."); + println!("all platforms reachable with matching digests."); Ok(()) } fn print_installed(i: &InstalledHyperd) { let status = if i.cache_hit { "cached" } else { "installed" }; println!( - "{status}: hyperd {version}.{build_id} ({platform}) -> {path}", + "{status}: hyperd {version} ({platform}) -> {path}", status = status, version = i.version, - build_id = i.build_id, platform = i.platform, path = i.binary_path.display(), ); diff --git a/hyperdb-bootstrap/src/error.rs b/hyperdb-bootstrap/src/error.rs index 407c3d83..e17c18af 100644 --- a/hyperdb-bootstrap/src/error.rs +++ b/hyperdb-bootstrap/src/error.rs @@ -8,9 +8,9 @@ use thiserror::Error; /// Errors produced while downloading, verifying, and installing `hyperd`. /// /// Every fallible function in this crate returns a `Result`. The -/// variants line up with the phases of bootstrap: platform detection, -/// HTTP/curl fetching, TOML parsing, archive extraction, and checksum -/// verification. +/// variants line up with the phases of bootstrap: platform detection, URL +/// construction, `curl` fetching, TOML parsing, archive extraction, and +/// checksum verification. #[derive(Debug, Error)] pub enum Error { /// The host (`os` / `arch` combination) has no published `hyperd` build. @@ -36,20 +36,17 @@ pub enum Error { source: std::io::Error, }, - /// A `reqwest` HTTP client error (connection failure, TLS issue, etc.). - #[error("HTTP error: {0}")] - Http(#[source] reqwest::Error), - - /// A server returned a non-success HTTP status while fetching `url`. - #[error("HTTP {status} when fetching {url}")] - HttpStatus { - /// URL that was being fetched when the failure occurred. - url: String, - /// HTTP response status code. - status: u16, + /// The pinned release carries no wheel tag for this platform, so the + /// wheel file name cannot be constructed. + #[error( + "no wheel tag pinned for platform {platform}; add it to the [wheel_tag] table in hyperd-version.toml" + )] + MissingWheelTag { + /// Platform whose `[wheel_tag]` entry is missing. + platform: crate::platform::Platform, }, - /// The fallback `curl` subprocess exited with a non-zero status. + /// The `curl` subprocess exited with a non-zero status. #[error("curl exited with code {code} when fetching {url}")] CurlFailed { /// URL passed to `curl`. @@ -78,10 +75,6 @@ pub enum Error { /// The archive did not contain a recognizable `hyperd` executable. #[error("hyperd executable not found in extracted archive")] HyperdNotInArchive, - - /// Scraping the public releases page for the latest version failed. - #[error("failed to scrape latest release: {0}")] - ScrapeFailed(&'static str), } impl Error { @@ -106,12 +99,10 @@ impl Error { } } - /// Constructs an [`Self::HttpStatus`] error. - pub fn http_status(url: impl Into, status: u16) -> Self { - Error::HttpStatus { - url: url.into(), - status, - } + /// Constructs an [`Self::MissingWheelTag`] error. + #[must_use] + pub fn missing_wheel_tag(platform: crate::platform::Platform) -> Self { + Error::MissingWheelTag { platform } } /// Constructs an [`Self::CurlFailed`] error. diff --git a/hyperdb-bootstrap/src/extract.rs b/hyperdb-bootstrap/src/extract.rs index fb349784..16ec9fdd 100644 --- a/hyperdb-bootstrap/src/extract.rs +++ b/hyperdb-bootstrap/src/extract.rs @@ -3,14 +3,18 @@ //! ZIP-archive extraction for the Hyper API release bundle. //! -//! The upstream archive nests `hyperd` plus its shared libraries inside a -//! versioned top-level directory (e.g. -//! `tableauhyperapi-java-macos-arm64-release-main.0.0.24457.rc36858b6/`) and -//! then under `lib/hyper/` on Linux/macOS or `bin/hyper/` on Windows. This -//! module flattens both layers so downstream consumers only see the -//! `hyperd` runtime files. The layout is identical across the Java and C++ -//! bundles, so this extractor is agnostic to which binding we download -//! (we use Java — see `url.rs` for why). +//! The upstream archive nests `hyperd` plus its shared libraries inside one +//! top-level directory and then under `lib/hyper/` or `bin/hyper/` inside it. +//! This module flattens both layers so downstream consumers only see the +//! `hyperd` runtime files. +//! +//! A wheel is a zip, and the same two-layer shape holds: the PyPI +//! `tableauhyperapi` wheels put the executable at +//! `tableauhyperapi/bin/hyper/hyperd` (plus `hyperd.exe` and +//! `crashdumper.exe` on Windows), so the `tableauhyperapi` package directory +//! lands in the "one optional top-level wrapper" slot and needs no special +//! case. The `lib/hyper/` spelling is retained because Tableau's Java and C++ +//! zips use it on Linux/macOS. use std::fs::{self, File}; use std::io; @@ -18,9 +22,9 @@ use std::path::{Path, PathBuf}; use crate::Error; -/// Extract everything under `lib/hyper/` (or `bin/hyper/` on Windows) from -/// the Hyper API zip into `dest_dir`, flattening the wrapper prefixes -/// away. Returns the list of extracted file paths relative to `dest_dir`. +/// Extract everything under `lib/hyper/` or `bin/hyper/` from the Hyper API +/// archive into `dest_dir`, flattening the wrapper prefixes away. Returns the +/// list of extracted file paths relative to `dest_dir`. /// /// # Errors /// @@ -89,14 +93,13 @@ pub fn extract_hyperd(zip_path: &Path, dest_dir: &Path) -> Result, } /// Return the path stripped of a leading `lib/hyper/` or `bin/hyper/` prefix, -/// or `None` if the entry is outside those directories. The Hyper API zip -/// wraps everything in a top-level `tableauhyperapi--...` directory and -/// nests the runtime under `lib/hyper/` (Linux/macOS) or `bin/hyper/` -/// (Windows) inside it. +/// or `None` if the entry is outside those directories. The archive wraps +/// everything in one top-level directory (`tableauhyperapi/` in a wheel) and +/// nests the runtime under `bin/hyper/` — or `lib/hyper/` in Tableau's +/// Linux/macOS zips — inside it. fn strip_lib_hyper_prefix(path: &Path) -> Option { let mut comps = path.components(); - // Skip one optional top-level wrapper component (e.g. - // `tableauhyperapi-java-macos-arm64-release-main.0.0.24457.rc36858b6`) + // Skip one optional top-level wrapper component (e.g. `tableauhyperapi`) // before looking for the `lib/hyper` or `bin/hyper` pair. let first = comps.next()?; let (a, b) = if first.as_os_str() == "lib" || first.as_os_str() == "bin" { @@ -156,6 +159,153 @@ mod tests { assert_eq!(strip_lib_hyper_prefix(Path::new("other/file")), None); } + /// Entry names taken verbatim from the real + /// `tableauhyperapi-0.0.26479-py3-none-macosx_13_0_arm64.whl` (42 entries; + /// this is the shape-representative subset). Guards the claim that the + /// wheel needs no extractor changes: the `tableauhyperapi` package + /// directory occupies the optional-wrapper slot, and `bin/hyper` is + /// already an accepted pair. + #[test] + fn strip_prefix_matches_real_wheel_entries() { + assert_eq!( + strip_lib_hyper_prefix(Path::new("tableauhyperapi/bin/hyper/hyperd")), + Some(PathBuf::from("hyperd")) + ); + // `libtableauhyperapi.dylib` sits under `bin/` but NOT `bin/hyper/`: + // it is the Python binding's own library, not part of the engine. + assert_eq!( + strip_lib_hyper_prefix(Path::new("tableauhyperapi/bin/libtableauhyperapi.dylib")), + None + ); + for outside in [ + "tableauhyperapi/__init__.py", + "tableauhyperapi/impl/dll.py", + "tableauhyperapi-0.0.26479.dist-info/LICENSE", + "tableauhyperapi-0.0.26479.dist-info/RECORD", + ] { + assert_eq!( + strip_lib_hyper_prefix(Path::new(outside)), + None, + "{outside} should not be extracted" + ); + } + } + + /// The Windows wheel ships `crashdumper.exe` beside `hyperd.exe` under the + /// same `bin/hyper/` directory, so both must survive the flattening. + #[test] + fn strip_prefix_matches_windows_wheel_entries() { + assert_eq!( + strip_lib_hyper_prefix(Path::new("tableauhyperapi/bin/hyper/hyperd.exe")), + Some(PathBuf::from("hyperd.exe")) + ); + assert_eq!( + strip_lib_hyper_prefix(Path::new("tableauhyperapi/bin/hyper/crashdumper.exe")), + Some(PathBuf::from("crashdumper.exe")) + ); + } + + /// End-to-end extraction over a fixture that mirrors the real macOS wheel + /// entry list, including the entries that must be skipped. + #[test] + fn extract_wheel_layout() -> Result<(), Box> { + use std::io::Write; + let tmp = tempfile::tempdir()?; + let wheel_path = tmp + .path() + .join("tableauhyperapi-0.0.26479-py3-none-test.whl"); + { + let file = File::create(&wheel_path)?; + let mut zw = zip::ZipWriter::new(file); + let opts = zip::write::SimpleFileOptions::default(); + for (name, body) in [ + ("tableauhyperapi/__init__.py", "python"), + ("tableauhyperapi/impl/dll.py", "python"), + ( + "tableauhyperapi/bin/libtableauhyperapi.dylib", + "binding lib", + ), + ("tableauhyperapi/bin/hyper/hyperd", "fake hyperd"), + ("tableauhyperapi-0.0.26479.dist-info/LICENSE", "license"), + ("tableauhyperapi-0.0.26479.dist-info/RECORD", "record"), + ] { + zw.start_file(name, opts)?; + zw.write_all(body.as_bytes())?; + } + zw.finish()?; + } + let out = tmp.path().join("out"); + let files = extract_hyperd(&wheel_path, &out)?; + + assert_eq!(files, vec![PathBuf::from("hyperd")]); + assert_eq!(std::fs::read_to_string(out.join("hyperd"))?, "fake hyperd"); + // Nothing outside bin/hyper/ leaks into the install dir. + for skipped in [ + "__init__.py", + "libtableauhyperapi.dylib", + "LICENSE", + "RECORD", + ] { + assert!(!out.join(skipped).exists(), "{skipped} should be skipped"); + } + Ok(()) + } + + /// The Windows wheel case, including `crashdumper.exe`. + #[test] + fn extract_windows_wheel_layout() -> Result<(), Box> { + use std::io::Write; + let tmp = tempfile::tempdir()?; + let wheel_path = tmp + .path() + .join("tableauhyperapi-0.0.26479-py3-none-win_amd64.whl"); + { + let file = File::create(&wheel_path)?; + let mut zw = zip::ZipWriter::new(file); + let opts = zip::write::SimpleFileOptions::default(); + for name in [ + "tableauhyperapi/bin/hyper/hyperd.exe", + "tableauhyperapi/bin/hyper/crashdumper.exe", + "tableauhyperapi/bin/tableauhyperapi.dll", + ] { + zw.start_file(name, opts)?; + zw.write_all(b"fake")?; + } + zw.finish()?; + } + let out = tmp.path().join("out"); + let files = extract_hyperd(&wheel_path, &out)?; + + assert!(files.iter().any(|p| p == Path::new("hyperd.exe"))); + assert!(files.iter().any(|p| p == Path::new("crashdumper.exe"))); + assert!(!out.join("tableauhyperapi.dll").exists()); + Ok(()) + } + + /// A wheel with no engine inside must fail loudly rather than install an + /// empty directory. + #[test] + fn extract_errors_when_no_hyperd() -> Result<(), Box> { + use std::io::Write; + let tmp = tempfile::tempdir()?; + let wheel_path = tmp.path().join("no-engine.whl"); + { + let file = File::create(&wheel_path)?; + let mut zw = zip::ZipWriter::new(file); + zw.start_file( + "tableauhyperapi/__init__.py", + zip::write::SimpleFileOptions::default(), + )?; + zw.write_all(b"python")?; + zw.finish()?; + } + assert!(matches!( + extract_hyperd(&wheel_path, &tmp.path().join("out")), + Err(Error::HyperdNotInArchive) + )); + Ok(()) + } + #[test] fn extract_fixture_zip() -> Result<(), Box> { use std::io::Write; diff --git a/hyperdb-bootstrap/src/install.rs b/hyperdb-bootstrap/src/install.rs index 2cf89317..131e367f 100644 --- a/hyperdb-bootstrap/src/install.rs +++ b/hyperdb-bootstrap/src/install.rs @@ -2,8 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 OR MIT //! High-level [`install()`] entry point: resolves a release, downloads and -//! verifies the archive, extracts the executable into -//! `//`, then refreshes the +//! verifies the wheel, extracts the executable into +//! `//`, then refreshes the //! `/current/` pointer so downstream tooling can always find the //! active install at a stable path. @@ -15,12 +15,11 @@ use crate::download::download_and_verify; use crate::extract::extract_hyperd; use crate::platform::Platform; use crate::release::PinnedRelease; -use crate::scrape::scrape_latest; -use crate::url::build_download_url; +use crate::url::{build_download_url, wheel_filename}; /// Default directory (relative to CWD) used when [`InstallOptions::dest_root`] /// is left at its default. Laid out as -/// `.hyperd//hyperd[.exe]` plus a mirror at `.hyperd/current/`. +/// `.hyperd//hyperd[.exe]` plus a mirror at `.hyperd/current/`. pub const DEFAULT_DEST_ROOT: &str = ".hyperd"; /// Which `hyperd` release [`install`] should resolve. @@ -34,10 +33,8 @@ pub enum VersionSource { Builtin, /// Load pinned metadata from a specific TOML file. TomlFile(PathBuf), - /// Caller-supplied release (e.g. CLI `--version`/`--build-id` flags). + /// Caller-supplied release (e.g. the CLI `--version` flag). Explicit(PinnedRelease), - /// Best-effort scrape of the public releases page. - ScrapeLatest, } /// Configuration passed to [`install`]. @@ -71,10 +68,8 @@ pub struct InstalledHyperd { /// Absolute or relative path to the installed `hyperd` executable. /// Always under `/current/`. pub binary_path: PathBuf, - /// Installed release version (for example, `"0.0.24457"`). + /// Installed release version (for example, `"0.0.26479"`). pub version: String, - /// Installed release build id (for example, `"rc36858b6"`). - pub build_id: String, /// Host platform this install targets. pub platform: Platform, /// `true` if the versioned directory already existed and we skipped the @@ -96,7 +91,7 @@ pub struct InstalledHyperd { /// # Errors /// /// Returns any [`Error`] variant produced by the phases it drives — -/// platform detection, release resolution (TOML parsing or scraping), +/// platform detection, release resolution (TOML parsing), URL construction, /// download, checksum verification, ZIP extraction, or filesystem /// operations under `dest_root`. pub fn install(opts: InstallOptions) -> Result { @@ -104,9 +99,8 @@ pub fn install(opts: InstallOptions) -> Result { Some(p) => p, None => Platform::current()?, }; - let release = resolve_release(&opts.version_source, platform)?; - let version_tag = release.version_tag(); - let versioned_dir = opts.dest_root.join(&version_tag); + let release = resolve_release(&opts.version_source)?; + let versioned_dir = opts.dest_root.join(&release.version); let current_dir = opts.dest_root.join("current"); let exe_name = platform.executable_name(); @@ -120,23 +114,21 @@ pub fn install(opts: InstallOptions) -> Result { download_and_extract(&release, platform, &versioned_dir)?; } - refresh_current(¤t_dir, &versioned_dir, &version_tag)?; + refresh_current(¤t_dir, &versioned_dir, &release.version)?; let binary_path = current_dir.join(exe_name); Ok(InstalledHyperd { binary_path, version: release.version, - build_id: release.build_id, platform, cache_hit, }) } -fn resolve_release(source: &VersionSource, platform: Platform) -> Result { +fn resolve_release(source: &VersionSource) -> Result { match source { VersionSource::Builtin => Ok(PinnedRelease::builtin()), VersionSource::TomlFile(path) => PinnedRelease::from_toml_file(path), VersionSource::Explicit(r) => Ok(r.clone()), - VersionSource::ScrapeLatest => scrape_latest(platform), } } @@ -152,15 +144,15 @@ fn download_and_extract( fs::create_dir_all(versioned_dir) .map_err(|source| Error::io(format!("creating {}", versioned_dir.display()), source))?; - let url = build_download_url(release, platform); + let url = build_download_url(release, platform)?; let tmp = tempfile::tempdir().map_err(|source| Error::io("creating temp dir", source))?; - let zip_path = tmp.path().join("hyperapi-java.zip"); - download_and_verify(&url, release.sha256_for(platform), &zip_path)?; - extract_hyperd(&zip_path, versioned_dir)?; + let wheel_path = tmp.path().join(wheel_filename(release, platform)?); + download_and_verify(&url, release.sha256_for(platform), &wheel_path)?; + extract_hyperd(&wheel_path, versioned_dir)?; Ok(()) } -fn refresh_current(current: &Path, source: &Path, version_tag: &str) -> Result<(), Error> { +fn refresh_current(current: &Path, source: &Path, version: &str) -> Result<(), Error> { // current/ is a fresh file copy every run — avoids Windows symlink // privileges and keeps the Makefile auto-discovery path stable. if current.exists() { @@ -170,7 +162,7 @@ fn refresh_current(current: &Path, source: &Path, version_tag: &str) -> Result<( fs::create_dir_all(current) .map_err(|source| Error::io(format!("creating {}", current.display()), source))?; copy_dir_contents(source, current)?; - fs::write(current.join("VERSION"), version_tag) + fs::write(current.join("VERSION"), version) .map_err(|source| Error::io(format!("writing {}/VERSION", current.display()), source))?; Ok(()) } diff --git a/hyperdb-bootstrap/src/lib.rs b/hyperdb-bootstrap/src/lib.rs index 24364501..3f7daaeb 100644 --- a/hyperdb-bootstrap/src/lib.rs +++ b/hyperdb-bootstrap/src/lib.rs @@ -1,10 +1,12 @@ // Copyright (c) 2026, Salesforce, Inc. All rights reserved. // SPDX-License-Identifier: Apache-2.0 OR MIT -//! Download and install the `hyperd` executable from Tableau's Hyper Java -//! API release packages. (The Java bundle is used rather than the C++ one -//! because the C++ `macos-arm64` zip ships an x86_64 `hyperd`; see the -//! `url` module for the full rationale.) +//! Download and install the `hyperd` executable from the PyPI +//! `tableauhyperapi` wheels. The wheel carries the same `hyperd` build as +//! Tableau's Java/C++ API bundles, but its file name is constructible from +//! the release version alone — no opaque build id, no page scraping — and +//! PyPI publishes a sha256 per file. See the `url` module for the full +//! rationale. //! //! The crate ships both a CLI binary (`hyperdb-bootstrap`) and a small //! library. The library is blocking (no async runtime required) and has @@ -21,8 +23,8 @@ //! ``` //! //! See [`InstallOptions`] and [`VersionSource`] for how to override the -//! destination, pin a specific release, load metadata from an external -//! TOML file, or scrape the latest release from the public releases page. +//! destination, pin a specific release, or load metadata from an external +//! TOML file. /// HTTP (via `curl`) download of release archives + SHA-256 verification. pub mod download; @@ -36,16 +38,13 @@ pub mod install; pub mod platform; /// Pinned-release metadata loaded from `hyperd-version.toml`. pub mod release; -/// Best-effort scraping of the public releases page to discover the latest -/// version when no pin is supplied. -pub mod scrape; -/// URL construction for Tableau's public download endpoint. +/// URL construction for the PyPI wheel download endpoint. pub mod url; -/// Reachability probes that HEAD each platform URL of a pinned release. +/// Reachability and digest checks for each platform of a pinned release. pub mod verify; pub use error::Error; pub use install::{DEFAULT_DEST_ROOT, InstallOptions, InstalledHyperd, VersionSource, install}; pub use platform::Platform; pub use release::PinnedRelease; -pub use verify::{VerifyOutcome, verify_release}; +pub use verify::{DigestStatus, VerifyOutcome, verify_release}; diff --git a/hyperdb-bootstrap/src/release.rs b/hyperdb-bootstrap/src/release.rs index e528cba5..aaeec1c3 100644 --- a/hyperdb-bootstrap/src/release.rs +++ b/hyperdb-bootstrap/src/release.rs @@ -3,9 +3,9 @@ //! A pinned `hyperd` release descriptor loaded from `hyperd-version.toml`. //! -//! Each `PinnedRelease` records a specific `version` + `build_id` pair -//! (the two components that make up a Hyper release tag) and the expected -//! SHA-256 checksums for each platform. +//! Each `PinnedRelease` records the release `version`, the wheel platform tag +//! to request for each target, and the expected SHA-256 checksum of each +//! platform's wheel. use serde::{Deserialize, Serialize}; use std::collections::HashMap; @@ -16,8 +16,8 @@ use crate::platform::Platform; const BUILTIN_TOML: &str = include_str!("../hyperd-version.toml"); -/// A concrete `hyperd` release pinned to a specific version and build, with -/// optional per-platform SHA-256 checksums. +/// A concrete `hyperd` release pinned to a specific version, with per-platform +/// wheel tags and optional per-platform SHA-256 checksums. /// /// The "built-in" pin shipped with the crate lives in /// `hyperdb-bootstrap/hyperd-version.toml` and is available via @@ -26,10 +26,13 @@ const BUILTIN_TOML: &str = include_str!("../hyperd-version.toml"); /// a literal TOML string to [`PinnedRelease::from_toml_str`]. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct PinnedRelease { - /// Upstream release version (for example, `"0.0.24457"`). + /// Upstream release version (for example, `"0.0.26479"`). pub version: String, - /// Upstream build identifier suffix (for example, `"rc36858b6"`). - pub build_id: String, + /// Wheel platform tag keyed by platform (for example, + /// `"macosx_13_0_arm64"`). Kept as pin data rather than hardcoded because + /// the tags are not stable across releases and a wrong tag 404s silently. + #[serde(default)] + pub wheel_tag: HashMap, /// Expected SHA-256 digests keyed by platform. Empty strings are treated /// as "no digest" so that partially-filled tables skip verification for /// the missing targets instead of failing outright. @@ -78,17 +81,24 @@ impl PinnedRelease { /// strings (common in pre-release metadata) are treated as absent. #[must_use] pub fn sha256_for(&self, platform: Platform) -> Option<&str> { - self.sha256 - .get(&platform) - .map(|s| s.trim()) - .filter(|s| !s.is_empty()) + Self::lookup(&self.sha256, platform) } - /// Returns the full Hyper release tag — `version.build_id` — used in - /// download URLs and install directory names. + /// Returns the wheel platform tag for `platform` (for example, + /// `"macosx_13_0_arm64"`), or `None` if the pin does not carry one. + /// + /// Without a tag the wheel filename cannot be constructed, so callers + /// treat `None` as [`Error::MissingWheelTag`] rather than guessing. #[must_use] - pub fn version_tag(&self) -> String { - format!("{}.{}", self.version, self.build_id) + pub fn wheel_tag_for(&self, platform: Platform) -> Option<&str> { + Self::lookup(&self.wheel_tag, platform) + } + + fn lookup(table: &HashMap, platform: Platform) -> Option<&str> { + table + .get(&platform) + .map(|s| s.trim()) + .filter(|s| !s.is_empty()) } } @@ -100,14 +110,29 @@ mod tests { fn builtin_parses() { let r = PinnedRelease::builtin(); assert!(!r.version.is_empty()); - assert!(!r.build_id.is_empty()); } #[test] - fn empty_sha_is_ignored() { + fn builtin_pins_a_wheel_tag_and_digest_for_every_platform() { + let r = PinnedRelease::builtin(); + for p in [ + Platform::MacosArm64, + Platform::MacosX86_64, + Platform::LinuxX86_64, + Platform::WindowsX86_64, + ] { + assert!(r.wheel_tag_for(p).is_some(), "{p} has no wheel tag"); + assert!(r.sha256_for(p).is_some(), "{p} has no sha256"); + } + } + + #[test] + fn empty_entries_are_ignored() { let toml_str = r#" version = "0.0.1" -build_id = "rc1" +[wheel_tag] +"macos-arm64" = "" +"linux-x86_64" = "manylinux2014_x86_64" [sha256] "macos-arm64" = "" "linux-x86_64" = "abc" @@ -115,15 +140,17 @@ build_id = "rc1" let r = PinnedRelease::from_toml_str(toml_str).unwrap(); assert!(r.sha256_for(Platform::MacosArm64).is_none()); assert_eq!(r.sha256_for(Platform::LinuxX86_64), Some("abc")); + assert!(r.wheel_tag_for(Platform::MacosArm64).is_none()); + assert_eq!( + r.wheel_tag_for(Platform::LinuxX86_64), + Some("manylinux2014_x86_64") + ); } #[test] - fn version_tag_format() { - let r = PinnedRelease { - version: "0.0.24457".to_string(), - build_id: "rc36858b6".to_string(), - sha256: HashMap::new(), - }; - assert_eq!(r.version_tag(), "0.0.24457.rc36858b6"); + fn tables_default_to_empty_when_absent() { + let r = PinnedRelease::from_toml_str("version = \"0.0.1\"").unwrap(); + assert!(r.wheel_tag_for(Platform::MacosArm64).is_none()); + assert!(r.sha256_for(Platform::MacosArm64).is_none()); } } diff --git a/hyperdb-bootstrap/src/scrape.rs b/hyperdb-bootstrap/src/scrape.rs deleted file mode 100644 index 2defd3f2..00000000 --- a/hyperdb-bootstrap/src/scrape.rs +++ /dev/null @@ -1,154 +0,0 @@ -// Copyright (c) 2026, Salesforce, Inc. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 OR MIT - -//! Best-effort scraper for the public Hyper releases page. -//! -//! When no version pin is supplied, [`crate::scrape::scrape_latest`] fetches -//! `https://tableau.github.io/hyper-db/docs/releases` and parses out the -//! most recent version + build id for the given platform. This bypasses -//! the compile-time pin and can lag or break when the page layout changes -//! — prefer an explicit [`crate::VersionSource::Builtin`] or -//! [`crate::VersionSource::TomlFile`] in production. - -use regex::Regex; -use std::collections::HashMap; - -use crate::Error; -use crate::platform::Platform; -use crate::release::PinnedRelease; - -const RELEASES_URL: &str = "https://tableau.github.io/hyper-db/docs/releases"; - -/// Installs `ring` as the process-wide rustls crypto provider. -/// -/// Our `reqwest` uses the `rustls-no-provider` feature, which links no -/// provider of its own. `reqwest` resolves one via -/// `CryptoProvider::get_default()`, which has no crate-feature fallback, and -/// *panics* while building a `Client` if nothing is installed. So this must -/// run before the first client is built. -/// -/// An `Err` from `install_default` means something else — most likely the host -/// application — installed a provider first. That is deliberately ignored: a -/// library must not override an embedder's choice. -fn ensure_crypto_provider() { - static INIT: std::sync::OnceLock<()> = std::sync::OnceLock::new(); - INIT.get_or_init(|| { - let _ = rustls::crypto::ring::default_provider().install_default(); - }); -} - -/// Fetches the public releases page and returns the newest `PinnedRelease` -/// that has a Java download for `platform`. -/// -/// The returned `PinnedRelease` has an empty SHA-256 map — scraping only -/// recovers the version + build id, never a digest. -/// -/// # Errors -/// -/// - [`Error::Http`] on network / TLS failure. -/// - [`Error::HttpStatus`] on a non-success HTTP response. -/// - [`Error::ScrapeFailed`] when the page layout no longer matches the -/// expected structure. -pub fn scrape_latest(platform: Platform) -> Result { - tracing::info!(url = RELEASES_URL, "scraping latest release"); - ensure_crypto_provider(); - let client = reqwest::blocking::Client::builder() - .user_agent(concat!("hyperd-bootstrap/", env!("CARGO_PKG_VERSION"))) - .build() - .map_err(Error::Http)?; - let resp = client.get(RELEASES_URL).send().map_err(Error::Http)?; - if !resp.status().is_success() { - return Err(Error::http_status(RELEASES_URL, resp.status().as_u16())); - } - let html = resp.text().map_err(Error::Http)?; - parse_latest(&html, platform) -} - -fn parse_latest(html: &str, platform: Platform) -> Result { - // The releases page lists entries in reverse-chronological order as - // `

VERSION [DATE]

`. The first match is the newest. - let h3_re = - Regex::new(r"]*>\s*([0-9]+(?:\.[0-9]+){1,3})\s*\[[^\]]+\]").expect("valid regex"); - let version = h3_re - .captures(html) - .and_then(|c| c.get(1)) - .map(|m| m.as_str().to_string()) - .ok_or(Error::ScrapeFailed( - "no

VERSION [DATE]

heading found", - ))?; - - // For that version, find the Java zip for the requested platform to - // recover the build id. - let href_re = Regex::new(&format!( - r"tableauhyperapi-java-{plat}-release-main\.{ver}\.(rc[a-z0-9]+)\.zip", - plat = regex::escape(platform.slug()), - ver = regex::escape(&version), - )) - .expect("valid regex"); - let build_id = href_re - .captures(html) - .and_then(|c| c.get(1)) - .map(|m| m.as_str().to_string()) - .ok_or(Error::ScrapeFailed( - "no matching java zip href for scraped version", - ))?; - - Ok(PinnedRelease { - version, - build_id, - sha256: HashMap::new(), - }) -} - -#[cfg(test)] -mod tests { - use super::*; - - /// Guards the `rustls-no-provider` feature choice. - /// - /// `reqwest` panics inside `build()` when no crypto provider is installed, - /// so this needs no network to fail. Without - /// [`ensure_crypto_provider`] it aborts the test thread outright. - #[test] - fn client_builds_with_a_crypto_provider_installed() { - ensure_crypto_provider(); - assert!( - reqwest::blocking::Client::builder() - .user_agent("hyperd-bootstrap-test") - .build() - .is_ok() - ); - } - - #[test] - fn parse_real_page_snippet() { - let html = r#" -

0.0.24457 [February 12 2026]

-
  • Release notes
- Java (macOS arm64) - Java (Linux) -

0.0.20000 [January 1 2025]

- "#; - let release = parse_latest(html, Platform::MacosArm64).unwrap(); - assert_eq!(release.version, "0.0.24457"); - assert_eq!(release.build_id, "rc36858b6"); - } - - #[test] - fn parse_errors_when_no_heading() { - let html = "

nothing here

"; - assert!(matches!( - parse_latest(html, Platform::LinuxX86_64), - Err(Error::ScrapeFailed(_)) - )); - } - - #[test] - fn parse_errors_when_no_matching_href() { - let html = r"

0.0.24457 [Feb 12 2026]

"; - assert!(matches!( - parse_latest(html, Platform::WindowsX86_64), - Err(Error::ScrapeFailed(_)) - )); - } -} diff --git a/hyperdb-bootstrap/src/url.rs b/hyperdb-bootstrap/src/url.rs index 60edd126..25a64c8b 100644 --- a/hyperdb-bootstrap/src/url.rs +++ b/hyperdb-bootstrap/src/url.rs @@ -1,36 +1,59 @@ // Copyright (c) 2026, Salesforce, Inc. All rights reserved. // SPDX-License-Identifier: Apache-2.0 OR MIT -//! Builds canonical download URLs for Tableau's public Hyper **Java** API -//! release bundles. +//! Builds download URLs for the PyPI `tableauhyperapi` wheels, which is where +//! this crate sources the `hyperd` executable. //! -//! We deliberately use the Java binding's bundle rather than the C++ one: -//! the C++ `macos-arm64` zip ships an **x86_64** `hyperd` (an upstream -//! packaging defect), so on Apple Silicon it would only run under Rosetta. -//! The Java `macos-arm64` bundle carries a native arm64 `hyperd`. Both -//! bundles share the identical URL template (only the `java`/`cxx` token -//! differs) and the identical internal layout (`lib/hyper/hyperd`), so the -//! switch is confined to this token and the pinned sha256s. +//! We use the wheel rather than Tableau's Java/C++ API zip on +//! `downloads.tableau.com` because the wheel filename is fully constructible +//! from the release version plus a platform tag. The zip filenames embed an +//! opaque build id (e.g. `r96880f6a`) that cannot be derived from the version, +//! which previously forced every bump through an HTML scraper. The `hyperd` +//! inside the wheel is the same build — byte-identical to the one the Java +//! bundle ships — so this is a change of envelope, not of engine. +//! +//! The legacy `/packages/py3/t//` path used here is +//! constructible without querying the PyPI index; it 302-redirects to the +//! content-addressed URL. PyPI also publishes a sha256 per file, which is +//! where the digests in `hyperd-version.toml` come from. +use crate::Error; use crate::platform::Platform; use crate::release::PinnedRelease; -const BASE_URL: &str = "https://downloads.tableau.com/tssoftware"; +const BASE_URL: &str = "https://files.pythonhosted.org/packages/py3/t/tableauhyperapi"; -/// Builds the `downloads.tableau.com` URL for the given release / platform -/// combination. +/// Builds the wheel file name for the given release / platform combination, +/// for example `tableauhyperapi-0.0.26479-py3-none-macosx_13_0_arm64.whl`. +/// +/// # Errors /// -/// The URL template matches -/// `https://downloads.tableau.com/tssoftware/tableauhyperapi-java--release-main...zip`. -#[must_use] -pub fn build_download_url(release: &PinnedRelease, platform: Platform) -> String { - format!( - "{base}/tableauhyperapi-java-{plat}-release-main.{version}.{build_id}.zip", - base = BASE_URL, - plat = platform.slug(), +/// Returns [`Error::MissingWheelTag`] if the release does not pin a wheel tag +/// for `platform`. The tag is pin data (see [`PinnedRelease::wheel_tag_for`]) +/// precisely so that a missing or changed tag fails loudly here instead of +/// producing a 404 at download time. +pub fn wheel_filename(release: &PinnedRelease, platform: Platform) -> Result { + let tag = release + .wheel_tag_for(platform) + .ok_or_else(|| Error::missing_wheel_tag(platform))?; + Ok(format!( + "tableauhyperapi-{version}-py3-none-{tag}.whl", version = release.version, - build_id = release.build_id, - ) + )) +} + +/// Builds the `files.pythonhosted.org` URL for the given release / platform +/// combination. +/// +/// # Errors +/// +/// Returns [`Error::MissingWheelTag`] if the release does not pin a wheel tag +/// for `platform`. +pub fn build_download_url(release: &PinnedRelease, platform: Platform) -> Result { + Ok(format!( + "{BASE_URL}/{file}", + file = wheel_filename(release, platform)? + )) } #[cfg(test)] @@ -38,17 +61,73 @@ mod tests { use super::*; use std::collections::HashMap; + fn release() -> PinnedRelease { + PinnedRelease { + version: "0.0.26479".to_string(), + wheel_tag: HashMap::from([ + (Platform::MacosArm64, "macosx_13_0_arm64".to_string()), + (Platform::MacosX86_64, "macosx_10_11_x86_64".to_string()), + (Platform::LinuxX86_64, "manylinux2014_x86_64".to_string()), + (Platform::WindowsX86_64, "win_amd64".to_string()), + ]), + sha256: HashMap::new(), + } + } + #[test] fn url_matches_expected_template() { + assert_eq!( + build_download_url(&release(), Platform::MacosArm64).unwrap(), + "https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-0.0.26479-py3-none-macosx_13_0_arm64.whl" + ); + } + + #[test] + fn every_platform_gets_its_own_tag() { + let r = release(); + for (platform, tag) in [ + (Platform::MacosArm64, "macosx_13_0_arm64"), + (Platform::MacosX86_64, "macosx_10_11_x86_64"), + (Platform::LinuxX86_64, "manylinux2014_x86_64"), + (Platform::WindowsX86_64, "win_amd64"), + ] { + assert_eq!( + wheel_filename(&r, platform).unwrap(), + format!("tableauhyperapi-0.0.26479-py3-none-{tag}.whl") + ); + } + } + + #[test] + fn missing_wheel_tag_is_an_error_not_a_guess() { let r = PinnedRelease { - version: "0.0.24457".to_string(), - build_id: "rc36858b6".to_string(), + version: "0.0.26479".to_string(), + wheel_tag: HashMap::new(), sha256: HashMap::new(), }; - let url = build_download_url(&r, Platform::MacosArm64); - assert_eq!( - url, - "https://downloads.tableau.com/tssoftware/tableauhyperapi-java-macos-arm64-release-main.0.0.24457.rc36858b6.zip" - ); + assert!(matches!( + build_download_url(&r, Platform::LinuxX86_64), + Err(Error::MissingWheelTag { .. }) + )); + } + + #[test] + fn builtin_pin_builds_a_url_for_every_platform() { + let r = PinnedRelease::builtin(); + for p in [ + Platform::MacosArm64, + Platform::MacosX86_64, + Platform::LinuxX86_64, + Platform::WindowsX86_64, + ] { + let url = build_download_url(&r, p).expect("builtin pin has every wheel tag"); + assert!(url.starts_with(BASE_URL)); + assert!( + std::path::Path::new(&url) + .extension() + .is_some_and(|ext| ext.eq_ignore_ascii_case("whl")) + ); + assert!(url.contains(&r.version)); + } } } diff --git a/hyperdb-bootstrap/src/verify.rs b/hyperdb-bootstrap/src/verify.rs index 49fc9875..7764a1dd 100644 --- a/hyperdb-bootstrap/src/verify.rs +++ b/hyperdb-bootstrap/src/verify.rs @@ -1,17 +1,32 @@ // Copyright (c) 2026, Salesforce, Inc. All rights reserved. // SPDX-License-Identifier: Apache-2.0 OR MIT -//! HEAD each supported platform's download URL to confirm the pinned -//! release is still reachable on Tableau's CDN. Used by the `verify` -//! CLI subcommand and by CI workflows that guard against silent yanks -//! or URL-scheme changes. +//! Checks that a pinned release is still fetchable *and* still the bytes we +//! pinned. Used by the `verify` CLI subcommand and by CI workflows that guard +//! against silent yanks, URL-scheme changes, and digest drift. +//! +//! Two independent checks run per platform: +//! +//! 1. **Reachability** — HEAD the constructed +//! `files.pythonhosted.org/packages/py3/...` URL. This is the legacy PyPI +//! path, which is constructible without an index query but only reaches the +//! file via a redirect, so it is worth probing directly. +//! 2. **Digest** — cross-check the pinned sha256 against the digest PyPI +//! publishes for that exact wheel file name. This is the stronger check: +//! reachability only proves the CDN serves *something*, whereas the digest +//! proves the pinned bytes are the published bytes. It also catches a stale +//! `[wheel_tag]`, which would otherwise 404 silently on one platform only. +use std::collections::HashMap; +use std::fmt; use std::process::Command; +use serde::Deserialize; + use crate::Error; use crate::platform::Platform; use crate::release::PinnedRelease; -use crate::url::build_download_url; +use crate::url::{build_download_url, wheel_filename}; const PLATFORMS: &[Platform] = &[ Platform::MacosArm64, @@ -20,8 +35,53 @@ const PLATFORMS: &[Platform] = &[ Platform::WindowsX86_64, ]; -/// Result of a single platform reachability probe performed by -/// [`verify_release`]. +/// Outcome of cross-checking one platform's pinned digest against PyPI. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum DigestStatus { + /// The pinned digest matches the one PyPI publishes for the wheel. + Match, + /// PyPI publishes a different digest for the pinned file name. + Mismatch { + /// Digest PyPI publishes for the wheel. + published: String, + }, + /// PyPI's file list has no entry for the pinned wheel file name — + /// typically a stale `[wheel_tag]` or a yanked release. + FileMissing, + /// The pin carries no digest for this platform, so there is nothing to + /// compare. Not a failure: an empty digest is documented as "skip". + NotPinned, + /// The PyPI index could not be consulted. Not treated as a failure, since + /// an index outage says nothing about the pin. + Unknown(String), +} + +impl DigestStatus { + /// Returns `true` unless PyPI actively contradicts the pin. + /// + /// [`Self::Unknown`] and [`Self::NotPinned`] are "no information", not + /// failure; [`Self::Mismatch`] and [`Self::FileMissing`] are hard failures. + #[must_use] + pub fn ok(&self) -> bool { + !matches!(self, Self::Mismatch { .. } | Self::FileMissing) + } +} + +impl fmt::Display for DigestStatus { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Match => f.write_str("digest matches PyPI"), + Self::Mismatch { published } => { + write!(f, "digest MISMATCH — PyPI publishes {published}") + } + Self::FileMissing => f.write_str("PyPI has no file with this name (stale wheel tag?)"), + Self::NotPinned => f.write_str("no digest pinned"), + Self::Unknown(why) => write!(f, "digest unchecked ({why})"), + } + } +} + +/// Result of a single platform's probe performed by [`verify_release`]. #[derive(Debug, Clone)] pub struct VerifyOutcome { /// Platform the probe targeted. @@ -31,47 +91,145 @@ pub struct VerifyOutcome { /// HTTP status returned by the CDN, or `None` if the probe itself failed. pub status: Option, /// Error message when `status` is `None` (spawn failure, parse failure, - /// stderr from a failed `curl` invocation). + /// stderr from a failed `curl` invocation, or URL construction failure). pub error: Option, + /// How the pinned digest compares to the one PyPI publishes. + pub digest: DigestStatus, } impl VerifyOutcome { - /// Returns `true` when the probe observed a 2xx/3xx HTTP status, which - /// is what `curl --head --location` returns when the CDN serves the - /// release. + /// Returns `true` when the probe observed a 2xx/3xx HTTP status (what + /// `curl --head --location` returns when the CDN serves the file) and the + /// digest check did not contradict the pin. #[must_use] pub fn ok(&self) -> bool { - matches!(self.status, Some(s) if (200..400).contains(&s)) + matches!(self.status, Some(s) if (200..400).contains(&s)) && self.digest.ok() } } -/// HEAD every supported platform URL for `release`. Returns one outcome -/// per platform; callers decide how to surface failures. +/// Probe every supported platform for `release`: HEAD its wheel URL and +/// cross-check its pinned digest against PyPI. Returns one outcome per +/// platform; callers decide how to surface failures. /// -/// Uses `curl --head` rather than reqwest so that Akamai's bot-protection -/// layer (which blocks reqwest's TLS fingerprint from GitHub-hosted runner -/// IPs) does not cause false 403 failures. +/// Uses `curl` rather than an in-process HTTP client so that Akamai-style bot +/// protection (which blocks some TLS fingerprints from CI runner IPs) does not +/// cause false failures, and so the crate needs no TLS stack of its own. /// /// # Errors /// /// Currently always returns `Ok(_)` — individual platform failures are -/// surfaced through [`VerifyOutcome::error`]. The `Result` wrapper is -/// kept for forward compatibility if a top-level failure mode is added. +/// surfaced through [`VerifyOutcome::error`] and [`VerifyOutcome::digest`]. +/// The `Result` wrapper is kept for forward compatibility if a top-level +/// failure mode is added. pub fn verify_release(release: &PinnedRelease) -> Result, Error> { + let digests = digest_statuses(release); let mut out = Vec::with_capacity(PLATFORMS.len()); for &platform in PLATFORMS { - let url = build_download_url(release, platform); - let outcome = curl_head(&url); - out.push(VerifyOutcome { - platform, - url, - status: outcome.0, - error: outcome.1, - }); + let digest = digests + .get(&platform) + .cloned() + .unwrap_or_else(|| DigestStatus::Unknown("platform not probed".to_string())); + let outcome = match build_download_url(release, platform) { + Ok(url) => { + let (status, error) = curl_head(&url); + VerifyOutcome { + platform, + url, + status, + error, + digest, + } + } + Err(e) => VerifyOutcome { + platform, + url: String::from(""), + status: None, + error: Some(e.to_string()), + digest, + }, + }; + out.push(outcome); } Ok(out) } +/// Fetch PyPI's file list for the pinned version and compare each platform's +/// pinned digest against it. +fn digest_statuses(release: &PinnedRelease) -> HashMap { + let url = format!( + "https://pypi.org/pypi/tableauhyperapi/{}/json", + release.version + ); + match curl_body(&url) { + Ok(body) => compare_digests(&body, release), + Err(why) => PLATFORMS + .iter() + .map(|&p| (p, DigestStatus::Unknown(why.clone()))) + .collect(), + } +} + +/// The subset of PyPI's per-version JSON payload we care about. Unknown +/// fields are ignored, so this survives additions to the API response. +#[derive(Deserialize)] +struct PypiVersion { + urls: Vec, +} + +#[derive(Deserialize)] +struct PypiFile { + filename: String, + digests: PypiDigests, +} + +#[derive(Deserialize)] +struct PypiDigests { + sha256: String, +} + +/// Pure comparison of a PyPI per-version JSON payload against a pin. Split out +/// from the network fetch so it can be tested against a fixture. +fn compare_digests(index_json: &str, release: &PinnedRelease) -> HashMap { + let parsed: PypiVersion = match serde_json::from_str(index_json) { + Ok(p) => p, + Err(e) => { + let why = format!("could not parse PyPI response: {e}"); + return PLATFORMS + .iter() + .map(|&p| (p, DigestStatus::Unknown(why.clone()))) + .collect(); + } + }; + let published: HashMap<&str, &str> = parsed + .urls + .iter() + .map(|f| (f.filename.as_str(), f.digests.sha256.as_str())) + .collect(); + + PLATFORMS + .iter() + .map(|&platform| { + let status = match wheel_filename(release, platform) { + Err(e) => DigestStatus::Unknown(e.to_string()), + Ok(filename) => match ( + published.get(filename.as_str()), + release.sha256_for(platform), + ) { + (None, _) => DigestStatus::FileMissing, + (Some(_), None) => DigestStatus::NotPinned, + (Some(actual), Some(pinned)) if actual.eq_ignore_ascii_case(pinned) => { + DigestStatus::Match + } + (Some(actual), Some(_)) => DigestStatus::Mismatch { + published: (*actual).to_string(), + }, + }, + }; + (platform, status) + }) + .collect() +} + /// Run `curl --head --silent --show-error --location ` and parse the /// HTTP status from the first status line. Returns `(Some(status), None)` /// on success and `(None, Some(error))` on spawn/parse failure. @@ -88,22 +246,8 @@ fn curl_head(url: &str) -> (Option, Option) { let stderr = String::from_utf8_lossy(&output.stderr).into_owned(); return (None, Some(stderr)); } - // Parse the last "HTTP/x.x NNN" status line (curl --location may - // produce multiple status lines when following redirects). let stdout = String::from_utf8_lossy(&output.stdout); - let status = stdout - .lines() - .filter_map(|line| { - let line = line.trim(); - // Matches "HTTP/1.1 200 OK", "HTTP/2 403", etc. - if line.starts_with("HTTP/") { - line.split_whitespace().nth(1)?.parse::().ok() - } else { - None - } - }) - .next_back(); - match status { + match last_http_status(&stdout) { Some(s) => (Some(s), None), None => ( None, @@ -116,3 +260,184 @@ fn curl_head(url: &str) -> (Option, Option) { } } } + +/// Parse the last `HTTP/x.x NNN` status line. `curl --location` emits one per +/// hop, and the final hop is the one that matters. +fn last_http_status(stdout: &str) -> Option { + stdout + .lines() + .filter_map(|line| { + let line = line.trim(); + // Matches "HTTP/1.1 200 OK", "HTTP/2 403", etc. + if line.starts_with("HTTP/") { + line.split_whitespace().nth(1)?.parse::().ok() + } else { + None + } + }) + .next_back() +} + +/// GET `url` with `curl` and return the response body. +fn curl_body(url: &str) -> Result { + let output = Command::new("curl") + .args(["--silent", "--show-error", "--location", "--fail"]) + .arg(url) + .output() + .map_err(|e| format!("failed to spawn curl: {e}"))?; + if !output.status.success() { + return Err(format!( + "curl exited {}: {}", + output.status.code().unwrap_or(-1), + String::from_utf8_lossy(&output.stderr).trim() + )); + } + Ok(String::from_utf8_lossy(&output.stdout).into_owned()) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Trimmed-down shape of a real `pypi.org/pypi/tableauhyperapi//json` + /// response: only the fields `compare_digests` reads, plus an extra field + /// to prove unknown keys are tolerated. + const INDEX_JSON: &str = r#"{ + "info": {"version": "0.0.26479"}, + "urls": [ + {"filename": "tableauhyperapi-0.0.26479-py3-none-macosx_13_0_arm64.whl", + "digests": {"sha256": "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118", "md5": "x"}, + "size": 80316638}, + {"filename": "tableauhyperapi-0.0.26479-py3-none-manylinux2014_x86_64.whl", + "digests": {"sha256": "9f5ff04c0dc3c17224b7a3f36f297775f2f49aae084da84614003cd6508213bc"}, + "size": 89743672} + ] + }"#; + + fn pin(sha_arm64: &str) -> PinnedRelease { + PinnedRelease { + version: "0.0.26479".to_string(), + wheel_tag: HashMap::from([ + (Platform::MacosArm64, "macosx_13_0_arm64".to_string()), + (Platform::LinuxX86_64, "manylinux2014_x86_64".to_string()), + (Platform::WindowsX86_64, "win_amd64".to_string()), + ]), + sha256: HashMap::from([(Platform::MacosArm64, sha_arm64.to_string())]), + } + } + + #[test] + fn matching_digest_is_reported_as_match() { + let statuses = compare_digests( + INDEX_JSON, + &pin("e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118"), + ); + assert_eq!(statuses[&Platform::MacosArm64], DigestStatus::Match); + assert!(statuses[&Platform::MacosArm64].ok()); + } + + #[test] + fn digest_comparison_is_case_insensitive() { + let statuses = compare_digests( + INDEX_JSON, + &pin("E80E4DAC6D8437AD8C20F36ADD7E523B18BC06D90D4C605A256C57DF8DF2C118"), + ); + assert_eq!(statuses[&Platform::MacosArm64], DigestStatus::Match); + } + + #[test] + fn drifted_digest_is_a_failure() { + let statuses = compare_digests(INDEX_JSON, &pin(&"a".repeat(64))); + assert_eq!( + statuses[&Platform::MacosArm64], + DigestStatus::Mismatch { + published: "e80e4dac6d8437ad8c20f36add7e523b18bc06d90d4c605a256c57df8df2c118" + .to_string() + } + ); + assert!(!statuses[&Platform::MacosArm64].ok()); + } + + /// A wheel tag that PyPI does not publish for this version is the silent-404 + /// vector the `[wheel_tag]` pin exists to make visible. + #[test] + fn unpublished_wheel_tag_is_a_failure() { + let statuses = compare_digests(INDEX_JSON, &pin(&"a".repeat(64))); + assert_eq!( + statuses[&Platform::WindowsX86_64], + DigestStatus::FileMissing + ); + assert!(!statuses[&Platform::WindowsX86_64].ok()); + } + + #[test] + fn published_but_unpinned_digest_is_not_a_failure() { + let statuses = compare_digests(INDEX_JSON, &pin(&"a".repeat(64))); + assert_eq!(statuses[&Platform::LinuxX86_64], DigestStatus::NotPinned); + assert!(statuses[&Platform::LinuxX86_64].ok()); + } + + /// A platform with no pinned wheel tag can't be checked, but must not be + /// reported as a digest failure — `verify_release` surfaces the missing tag + /// through the URL-construction error instead. + #[test] + fn missing_wheel_tag_is_unknown_not_failure() { + let statuses = compare_digests(INDEX_JSON, &pin(&"a".repeat(64))); + assert!(matches!( + statuses[&Platform::MacosX86_64], + DigestStatus::Unknown(_) + )); + assert!(statuses[&Platform::MacosX86_64].ok()); + } + + #[test] + fn unparseable_response_is_unknown_for_every_platform() { + let statuses = compare_digests("503", &pin(&"a".repeat(64))); + assert_eq!(statuses.len(), PLATFORMS.len()); + for status in statuses.values() { + assert!(matches!(status, DigestStatus::Unknown(_))); + assert!(status.ok()); + } + } + + #[test] + fn last_status_line_wins_across_redirects() { + let stdout = "HTTP/2 302\r\nlocation: elsewhere\r\n\r\nHTTP/2 200\r\n"; + assert_eq!(last_http_status(stdout), Some(200)); + assert_eq!(last_http_status("no status here"), None); + } + + #[test] + fn outcome_ok_requires_both_reachability_and_digest() { + let base = VerifyOutcome { + platform: Platform::MacosArm64, + url: "https://example.invalid/x.whl".to_string(), + status: Some(200), + error: None, + digest: DigestStatus::Match, + }; + assert!(base.ok()); + assert!( + !VerifyOutcome { + status: Some(404), + ..base.clone() + } + .ok() + ); + assert!( + !VerifyOutcome { + digest: DigestStatus::FileMissing, + ..base.clone() + } + .ok() + ); + // An index outage must not fail an otherwise-reachable pin. + assert!( + VerifyOutcome { + digest: DigestStatus::Unknown("offline".to_string()), + ..base + } + .ok() + ); + } +} diff --git a/hyperdb-bootstrap/tests/integration.rs b/hyperdb-bootstrap/tests/integration.rs index 2ad4eabd..55e7f08f 100644 --- a/hyperdb-bootstrap/tests/integration.rs +++ b/hyperdb-bootstrap/tests/integration.rs @@ -5,21 +5,67 @@ use hyperdb_bootstrap::{ InstallOptions, PinnedRelease, Platform, VersionSource, install, url::build_download_url, + url::wheel_filename, }; +const PLATFORMS: [Platform; 4] = [ + Platform::MacosArm64, + Platform::MacosX86_64, + Platform::LinuxX86_64, + Platform::WindowsX86_64, +]; + #[test] -fn builtin_release_builds_a_valid_url() { +fn builtin_release_builds_a_valid_wheel_url_for_every_platform() { let r = PinnedRelease::builtin(); - let url = build_download_url(&r, Platform::LinuxX86_64); - assert!(url.starts_with("https://downloads.tableau.com/tssoftware/")); - assert!(url.contains("java-linux-x86_64")); - assert!( - std::path::Path::new(&url) - .extension() - .is_some_and(|ext| ext.eq_ignore_ascii_case("zip")) - ); - assert!(url.contains(&r.version)); - assert!(url.contains(&r.build_id)); + for platform in PLATFORMS { + let url = build_download_url(&r, platform).expect("builtin pin has every wheel tag"); + assert!( + url.starts_with("https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/"), + "unexpected base for {platform}: {url}" + ); + assert!( + std::path::Path::new(&url) + .extension() + .is_some_and(|ext| ext.eq_ignore_ascii_case("whl")) + ); + assert!(url.contains(&r.version)); + assert!(url.contains(r.wheel_tag_for(platform).expect("wheel tag"))); + } +} + +/// The wheel file name is what PyPI publishes a digest against, so it has to +/// match the `tableauhyperapi--py3-none-.whl` convention exactly. +#[test] +fn builtin_wheel_filenames_follow_the_pypi_convention() { + let r = PinnedRelease::builtin(); + for platform in PLATFORMS { + let name = wheel_filename(&r, platform).expect("builtin pin has every wheel tag"); + assert_eq!( + name, + format!( + "tableauhyperapi-{}-py3-none-{}.whl", + r.version, + r.wheel_tag_for(platform).expect("wheel tag") + ) + ); + } +} + +/// Every platform must carry both a wheel tag and a digest, or `download` +/// breaks (or silently skips verification) on that platform alone. +#[test] +fn builtin_release_pins_every_platform() { + let r = PinnedRelease::builtin(); + for platform in PLATFORMS { + assert!( + r.wheel_tag_for(platform).is_some(), + "{platform} has no wheel tag" + ); + let sha = r.sha256_for(platform).expect("digest"); + assert_eq!(sha.len(), 64, "{platform} digest is not 64 hex chars"); + assert!(sha.chars().all(|c| c.is_ascii_hexdigit())); + } } #[test] @@ -32,7 +78,7 @@ fn install_options_defaults_are_sensible() { } #[test] -#[ignore = "hits the public Tableau downloads CDN; run with --ignored"] +#[ignore = "downloads an ~80 MB wheel from PyPI; run with --ignored"] fn install_end_to_end_with_builtin() { let tmp = tempfile::tempdir().unwrap(); let installed = install(InstallOptions { @@ -50,13 +96,34 @@ fn install_end_to_end_with_builtin() { .and_then(|n| n.to_str()) .is_some_and(|n| n.starts_with("hyperd")) ); + // The install dir and VERSION marker are keyed on the version alone. + assert_eq!( + std::fs::read_to_string(tmp.path().join("current").join("VERSION")).unwrap(), + installed.version + ); + assert!(tmp.path().join(&installed.version).is_dir()); } #[test] -#[ignore = "scrapes a live web page; run with --ignored"] -fn scrape_latest_real_page() { - let platform = Platform::current().expect("supported platform"); - let r = hyperdb_bootstrap::scrape::scrape_latest(platform).expect("scrape succeeds"); - assert!(!r.version.is_empty()); - assert!(r.build_id.starts_with("rc")); +#[ignore = "hits the PyPI JSON API and the wheel CDN; run with --ignored"] +fn verify_builtin_release_against_pypi() { + let r = PinnedRelease::builtin(); + let outcomes = hyperdb_bootstrap::verify_release(&r).expect("verify runs"); + assert_eq!(outcomes.len(), PLATFORMS.len()); + for o in &outcomes { + assert!( + o.ok(), + "{} failed: status={:?} error={:?} digest={}", + o.platform, + o.status, + o.error, + o.digest + ); + assert_eq!( + o.digest, + hyperdb_bootstrap::DigestStatus::Match, + "{} digest not confirmed against PyPI", + o.platform + ); + } }