diff --git a/.agents/skills/update-hyperd-release/SKILL.md b/.agents/skills/update-hyperd-release/SKILL.md index 0746fbd6..e19b51fd 100644 --- a/.agents/skills/update-hyperd-release/SKILL.md +++ b/.agents/skills/update-hyperd-release/SKILL.md @@ -1,6 +1,6 @@ --- name: update-hyperd-release -description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version, updating hyperd-version.toml (version + build_id + 4 sha256s), verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. +description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version, updating hyperd-version.toml (version + build_id + 4 sha256s), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. --- # Update the pinned `hyperd` release @@ -12,8 +12,10 @@ gotchas learned in practice. ## Key facts (don't relearn these the hard way) -- **The pin lives in [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml)** — `version`, `build_id`, and four per-platform sha256s. That's the whole source of truth; contributors without an override get exactly this release. -- **We download the Java bundle, NOT the C++ one.** The C++ `macos-arm64` zip ships an **x86_64** `hyperd` (upstream packaging defect) that only runs under Rosetta on Apple Silicon. The Java `macos-arm64` bundle carries a **native arm64** `hyperd`. Same URL template (only the `java`/`cxx` token differs), same internal layout (`lib/hyper/hyperd`). **Verify this invariant every bump** (step 5) — if a future Java bundle regresses to x86_64, the whole reason for using it is gone. +- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 5. + - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, `build_id`, and four per-platform sha256s. This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. + - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin, used for the `hyperd` bundled into the npm packages. +- **We download the Java bundle, NOT the C++ one.** The C++ `macos-arm64` zip ships an **x86_64** `hyperd` (upstream packaging defect) that only runs under Rosetta on Apple Silicon. The Java `macos-arm64` bundle carries a **native arm64** `hyperd`. Same URL template (only the `java`/`cxx` token differs), same internal layout (`lib/hyper/hyperd`). **Verify this invariant every bump** (step 6) — if a future Java bundle regresses to x86_64, the whole reason for using it is gone. - **URL template:** `https://downloads.tableau.com/tssoftware/tableauhyperapi-java--release-main...zip` — platforms: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. - **Crate version is workspace-driven + release-please.** `hyperdb-bootstrap` uses `version.workspace = true`; do **not** hand-edit a crate version. The conventional-commit type drives the release — use `fix(bootstrap): ...` for a routine bump (patch release). - **Never invent `hyperd` flags** (AGENTS.md reminder #9) and **never report tests/benches green without real output** (#10). Tests start a real `hyperd` subprocess; a misconfigured server hangs rather than erroring. @@ -59,9 +61,65 @@ done ### 4. Edit `hyperd-version.toml` Update `version`, `build_id`, and all four `[sha256]` entries. Record the **old** -version/build_id first — you need it for the A/B benchmark (step 7). +version/build_id first — you need it for the A/B benchmark (step 8). -### 5. Verify the pin + the arm64 invariant +### 5. Mirror the pin into the npm release workflow + +**The step that is easy to miss, and it reddens CI every time it is missed.** +[`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) +bundles `hyperd` into the npm packages from its own hardcoded pin, decoupled +from the toml. Update it in the same commit as step 4 — the two files must +never be bumped separately. + +| Key to update | Where in the workflow (line numbers drift — grep) | +|---|---| +| `HYPERD_VERSION` | top-level `env:` block, ~line 26 | +| `HYPERD_BUILD_ID` | top-level `env:` block, ~line 27 | +| `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 99 | +| `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 110 | +| `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 115 | +| `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 105 | + +```bash +grep -nE "HYPERD_VERSION|HYPERD_BUILD_ID|hyperd-slug|hyperd-sha256" \ + .github/workflows/npm-build-publish.yml +``` + +- **The matrix hashes are the same Java-zip sha256s you computed in step 3** — + not hashes of the extracted binary or of some other artifact. The workflow + downloads the identical URL + (`tableauhyperapi-java-${SLUG}-release-main.${HYPERD_VERSION}.${HYPERD_BUILD_ID}.zip`), + and the guard compares each `hyperd-sha256` **directly** against + `[sha256].""` in the toml, so the values are byte-for-byte identical. + Copy them across verbatim. +- **`hyperd-slug` is the join key** and it carries the *toml's* platform names + (`macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`), not npm's + (`darwin-arm64`, `darwin-x64`, …), which live in the sibling `platform:` + field. Don't cross them. +- **The commented-out `darwin-x64` entry is invisible to the guard** — it parses + the YAML, so a commented block simply isn't in the matrix and is never + checked. **Update it anyway.** It is commented out only because those runners + are currently disabled; if it goes stale, whoever re-enables them ships a + mismatched engine or trips the guard on an unrelated PR. + +Then confirm locally before pushing. This is the `verify` check in CI +([`.github/workflows/verify-hyperd-pin.yml`](../../../.github/workflows/verify-hyperd-pin.yml)): + +```bash +python3 .github/scripts/verify-npm-hyperd-pin.py # needs PyYAML +# …or without touching your environment: +uv run --with pyyaml --no-project python3 .github/scripts/verify-npm-hyperd-pin.py +``` + +Expect one `ok:` line per checked key and exit 0. On drift it prints an +`::error::hyperd pin drift — …` line per mismatch and exits 1. + +**Why this guard exists:** the two pins silently diverged once. Only the toml +was bumped, so npm `0.7.1` shipped with bundled engine `0.0.25080` while +crates.io shipped `0.0.26359`. The guard turns that into a red check instead of +a mystery bug report months later. + +### 6. Verify the pin + the arm64 invariant ```bash make verify-hyperd-pin # all four platforms → HTTP 200 at the new pin @@ -73,7 +131,7 @@ file .hyperd/current/hyperd # MUST say "Mach-O 64-bit executable arm6 If `file` reports `x86_64`, **stop** — the Java bundle no longer carries a native arm64 binary and the bundle choice needs re-evaluation. -### 6. Run the full test suite against the NEW engine +### 7. Run the full test suite against the NEW engine Point `HYPERD_PATH` at the freshly downloaded binary — do **not** rely on the workstation default (`~/dev/bin/hyperd`), which may be an old or unversioned build. @@ -88,7 +146,7 @@ Require `failed=0`. Then the pre-commit gate: `cargo fmt --all -- --check` and `cargo clippy --workspace --all-targets --all-features -- -D warnings` (CI's exact clippy command). -### 7. A/B benchmark vs the previous pin +### 8. A/B benchmark vs the previous pin The canonical harness is the **unified suite** ([`hyperdb-api/benches/benchmark_suite.rs`](../../../hyperdb-api/benches/benchmark_suite.rs)). @@ -116,21 +174,21 @@ rm -rf .hyperd-old # clean up the scratch baseline (also add to .gitignore if - **Distrust `× 4` / parallel numbers on a laptop.** They throttle thermally — throughput declines monotonically across sequential runs because the machine is hotter for the second engine. Report single-connection deltas as the reliable signal; withhold multi-connection deltas unless run on a cooled/pinned host. - Report throughput as **M rows/s**, not wall time. -### 8. Log the release in the benchmark tracker +### 9. Log the release in the benchmark tracker Append a row per engine to [`docs/hyperd-release-benchmarks.md`](../../../docs/hyperd-release-benchmarks.md) (median single-connection numbers + the machine + the caveat). This builds the per-release history the BENCHMARK_GUIDE's by-platform tables don't capture. -### 9. Changelog +### 10. Changelog Add a `### Changed` bullet under `## [Unreleased]` in [`hyperdb-bootstrap/CHANGELOG.md`](../../../hyperdb-bootstrap/CHANGELOG.md): the new version/build, "verified native arm64", and the headline performance A/B (with the thermal caveat on multi-connection numbers). -### 10. Commit + PR +### 11. Commit + PR - Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to ()`. - **gh account:** the EMU account (`ssteiner_sfemu`) is Unauthorized on upstream. `gh auth switch --hostname github.com --user StefanSteiner`, then target upstream (it has the CI runners): `gh pr create --repo tableau/hyper-api-rust --base main --head StefanSteiner:`. @@ -139,6 +197,7 @@ thermal caveat on multi-connection numbers). ## Verification checklist (what "done" means) - [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 +- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, `HYPERD_BUILD_ID`, three matrix `hyperd-sha256`s, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 - [ ] `.hyperd/current/hyperd --version` reports the new version/build - [ ] `file` confirms macos-arm64 binary is native arm64 - [ ] `cargo test --workspace` → `failed=0` against the new engine diff --git a/.claude/skills/update-hyperd-release/SKILL.md b/.claude/skills/update-hyperd-release/SKILL.md index 0746fbd6..e19b51fd 100644 --- a/.claude/skills/update-hyperd-release/SKILL.md +++ b/.claude/skills/update-hyperd-release/SKILL.md @@ -1,6 +1,6 @@ --- name: update-hyperd-release -description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version, updating hyperd-version.toml (version + build_id + 4 sha256s), verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. +description: Use when bumping the pinned hyperd release for hyperdb-bootstrap — finding the latest Tableau Hyper API version, updating hyperd-version.toml (version + build_id + 4 sha256s), mirroring the same pin into the npm-build-publish workflow so the drift guard stays green, verifying the pin, running the full test suite, A/B benchmarking against the previous pin, logging the result per release, and opening the PR. --- # Update the pinned `hyperd` release @@ -12,8 +12,10 @@ gotchas learned in practice. ## Key facts (don't relearn these the hard way) -- **The pin lives in [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml)** — `version`, `build_id`, and four per-platform sha256s. That's the whole source of truth; contributors without an override get exactly this release. -- **We download the Java bundle, NOT the C++ one.** The C++ `macos-arm64` zip ships an **x86_64** `hyperd` (upstream packaging defect) that only runs under Rosetta on Apple Silicon. The Java `macos-arm64` bundle carries a **native arm64** `hyperd`. Same URL template (only the `java`/`cxx` token differs), same internal layout (`lib/hyper/hyperd`). **Verify this invariant every bump** (step 5) — if a future Java bundle regresses to x86_64, the whole reason for using it is gone. +- **The pin lives in TWO files and they must move together.** Any drift between them fails CI via [`.github/scripts/verify-npm-hyperd-pin.py`](../../../.github/scripts/verify-npm-hyperd-pin.py) — see step 5. + - [`hyperdb-bootstrap/hyperd-version.toml`](../../../hyperdb-bootstrap/hyperd-version.toml) — `version`, `build_id`, and four per-platform sha256s. This is what `make download-hyperd` and the crates.io path use; contributors without an override get exactly this release. + - [`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) — its **own hardcoded copy** of the same pin, used for the `hyperd` bundled into the npm packages. +- **We download the Java bundle, NOT the C++ one.** The C++ `macos-arm64` zip ships an **x86_64** `hyperd` (upstream packaging defect) that only runs under Rosetta on Apple Silicon. The Java `macos-arm64` bundle carries a **native arm64** `hyperd`. Same URL template (only the `java`/`cxx` token differs), same internal layout (`lib/hyper/hyperd`). **Verify this invariant every bump** (step 6) — if a future Java bundle regresses to x86_64, the whole reason for using it is gone. - **URL template:** `https://downloads.tableau.com/tssoftware/tableauhyperapi-java--release-main...zip` — platforms: `macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`. - **Crate version is workspace-driven + release-please.** `hyperdb-bootstrap` uses `version.workspace = true`; do **not** hand-edit a crate version. The conventional-commit type drives the release — use `fix(bootstrap): ...` for a routine bump (patch release). - **Never invent `hyperd` flags** (AGENTS.md reminder #9) and **never report tests/benches green without real output** (#10). Tests start a real `hyperd` subprocess; a misconfigured server hangs rather than erroring. @@ -59,9 +61,65 @@ done ### 4. Edit `hyperd-version.toml` Update `version`, `build_id`, and all four `[sha256]` entries. Record the **old** -version/build_id first — you need it for the A/B benchmark (step 7). +version/build_id first — you need it for the A/B benchmark (step 8). -### 5. Verify the pin + the arm64 invariant +### 5. Mirror the pin into the npm release workflow + +**The step that is easy to miss, and it reddens CI every time it is missed.** +[`.github/workflows/npm-build-publish.yml`](../../../.github/workflows/npm-build-publish.yml) +bundles `hyperd` into the npm packages from its own hardcoded pin, decoupled +from the toml. Update it in the same commit as step 4 — the two files must +never be bumped separately. + +| Key to update | Where in the workflow (line numbers drift — grep) | +|---|---| +| `HYPERD_VERSION` | top-level `env:` block, ~line 26 | +| `HYPERD_BUILD_ID` | top-level `env:` block, ~line 27 | +| `hyperd-sha256` for `hyperd-slug: macos-arm64` | `jobs.build-npm.strategy.matrix.include`, ~line 99 | +| `hyperd-sha256` for `hyperd-slug: linux-x86_64` | same matrix, ~line 110 | +| `hyperd-sha256` for `hyperd-slug: windows-x86_64` | same matrix, ~line 115 | +| `hyperd-sha256` in the commented-out `darwin-x64` block (`hyperd-slug: macos-x86_64`) | same matrix, ~line 105 | + +```bash +grep -nE "HYPERD_VERSION|HYPERD_BUILD_ID|hyperd-slug|hyperd-sha256" \ + .github/workflows/npm-build-publish.yml +``` + +- **The matrix hashes are the same Java-zip sha256s you computed in step 3** — + not hashes of the extracted binary or of some other artifact. The workflow + downloads the identical URL + (`tableauhyperapi-java-${SLUG}-release-main.${HYPERD_VERSION}.${HYPERD_BUILD_ID}.zip`), + and the guard compares each `hyperd-sha256` **directly** against + `[sha256].""` in the toml, so the values are byte-for-byte identical. + Copy them across verbatim. +- **`hyperd-slug` is the join key** and it carries the *toml's* platform names + (`macos-arm64`, `macos-x86_64`, `linux-x86_64`, `windows-x86_64`), not npm's + (`darwin-arm64`, `darwin-x64`, …), which live in the sibling `platform:` + field. Don't cross them. +- **The commented-out `darwin-x64` entry is invisible to the guard** — it parses + the YAML, so a commented block simply isn't in the matrix and is never + checked. **Update it anyway.** It is commented out only because those runners + are currently disabled; if it goes stale, whoever re-enables them ships a + mismatched engine or trips the guard on an unrelated PR. + +Then confirm locally before pushing. This is the `verify` check in CI +([`.github/workflows/verify-hyperd-pin.yml`](../../../.github/workflows/verify-hyperd-pin.yml)): + +```bash +python3 .github/scripts/verify-npm-hyperd-pin.py # needs PyYAML +# …or without touching your environment: +uv run --with pyyaml --no-project python3 .github/scripts/verify-npm-hyperd-pin.py +``` + +Expect one `ok:` line per checked key and exit 0. On drift it prints an +`::error::hyperd pin drift — …` line per mismatch and exits 1. + +**Why this guard exists:** the two pins silently diverged once. Only the toml +was bumped, so npm `0.7.1` shipped with bundled engine `0.0.25080` while +crates.io shipped `0.0.26359`. The guard turns that into a red check instead of +a mystery bug report months later. + +### 6. Verify the pin + the arm64 invariant ```bash make verify-hyperd-pin # all four platforms → HTTP 200 at the new pin @@ -73,7 +131,7 @@ file .hyperd/current/hyperd # MUST say "Mach-O 64-bit executable arm6 If `file` reports `x86_64`, **stop** — the Java bundle no longer carries a native arm64 binary and the bundle choice needs re-evaluation. -### 6. Run the full test suite against the NEW engine +### 7. Run the full test suite against the NEW engine Point `HYPERD_PATH` at the freshly downloaded binary — do **not** rely on the workstation default (`~/dev/bin/hyperd`), which may be an old or unversioned build. @@ -88,7 +146,7 @@ Require `failed=0`. Then the pre-commit gate: `cargo fmt --all -- --check` and `cargo clippy --workspace --all-targets --all-features -- -D warnings` (CI's exact clippy command). -### 7. A/B benchmark vs the previous pin +### 8. A/B benchmark vs the previous pin The canonical harness is the **unified suite** ([`hyperdb-api/benches/benchmark_suite.rs`](../../../hyperdb-api/benches/benchmark_suite.rs)). @@ -116,21 +174,21 @@ rm -rf .hyperd-old # clean up the scratch baseline (also add to .gitignore if - **Distrust `× 4` / parallel numbers on a laptop.** They throttle thermally — throughput declines monotonically across sequential runs because the machine is hotter for the second engine. Report single-connection deltas as the reliable signal; withhold multi-connection deltas unless run on a cooled/pinned host. - Report throughput as **M rows/s**, not wall time. -### 8. Log the release in the benchmark tracker +### 9. Log the release in the benchmark tracker Append a row per engine to [`docs/hyperd-release-benchmarks.md`](../../../docs/hyperd-release-benchmarks.md) (median single-connection numbers + the machine + the caveat). This builds the per-release history the BENCHMARK_GUIDE's by-platform tables don't capture. -### 9. Changelog +### 10. Changelog Add a `### Changed` bullet under `## [Unreleased]` in [`hyperdb-bootstrap/CHANGELOG.md`](../../../hyperdb-bootstrap/CHANGELOG.md): the new version/build, "verified native arm64", and the headline performance A/B (with the thermal caveat on multi-connection numbers). -### 10. Commit + PR +### 11. Commit + PR - Commit with `git add ` (never `-A`), type `fix(bootstrap): bump pinned hyperd to ()`. - **gh account:** the EMU account (`ssteiner_sfemu`) is Unauthorized on upstream. `gh auth switch --hostname github.com --user StefanSteiner`, then target upstream (it has the CI runners): `gh pr create --repo tableau/hyper-api-rust --base main --head StefanSteiner:`. @@ -139,6 +197,7 @@ thermal caveat on multi-connection numbers). ## Verification checklist (what "done" means) - [ ] `make verify-hyperd-pin` → all four platforms HTTP 200 +- [ ] `npm-build-publish.yml` pin mirrored (`HYPERD_VERSION`, `HYPERD_BUILD_ID`, three matrix `hyperd-sha256`s, plus the commented-out `darwin-x64` one) and `verify-npm-hyperd-pin.py` exits 0 - [ ] `.hyperd/current/hyperd --version` reports the new version/build - [ ] `file` confirms macos-arm64 binary is native arm64 - [ ] `cargo test --workspace` → `failed=0` against the new engine diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5b96f156..9062bede 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -63,20 +63,36 @@ jobs: - run: cargo fmt --all --check clippy: - # Clippy lints are platform-independent, so a single runner is enough. - # If a lint ever diverges by target (rare), broaden the matrix. - name: clippy - runs-on: ubuntu-latest + # Most lints are platform-independent, but `cfg`-gated code is stripped + # before lints run, so a single runner leaves every `#[cfg(windows)]` block + # unlinted. That is not hypothetical: the edition-2024 sweep flattened 127 + # `collapsible_if` sites and missed one inside a `#[cfg(windows)]` block in + # `process.rs`, because ubuntu never compiled it. Windows is the leg worth + # paying for — it carries a whole distinct Named Pipe transport path. + # + # macOS is deliberately omitted: it shares `cfg(unix)` with the Linux leg, + # so only `#[cfg(target_os = "macos")]` blocks remain unlinted, a much + # smaller surface than the Windows one. + name: clippy (${{ matrix.os }}) + runs-on: ${{ matrix.os }} timeout-minutes: 30 + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, windows-latest] steps: - uses: actions/checkout@v7 - - name: Install system libraries (fontconfig for plotters, mold for fast linking, protobuf) + - name: Install system libraries (Linux) + if: runner.os == 'Linux' run: sudo apt-get update -q && sudo apt-get install -y libfontconfig1-dev mold protobuf-compiler + - name: Install protobuf (Windows) + if: runner.os == 'Windows' + run: choco install protoc -y - uses: actions-rust-lang/setup-rust-toolchain@v1 with: toolchain: stable components: clippy - cache-key: clippy + cache-key: clippy-${{ matrix.os }} # See fmt job for rationale; clippy enforces -D warnings via # the explicit `-- -D warnings` arg below, not via env. rustflags: "" @@ -240,6 +256,15 @@ jobs: # its own workflow when wired up. hyperdb-bootstrap has its own # coverage (next step) and doesn't need hyperd running. + - name: hyperdb-compile-check tests + # Declares its own [workspace], so the run above cannot see it. Nothing + # else ran this published crate's tests — the msrv and RHEL jobs only + # `cargo check` it. + shell: bash + env: + HYPERD_PATH: ${{ github.workspace }}/.hyperd/current + run: cargo test --manifest-path hyperdb-compile-check/Cargo.toml + - name: hyperdb-bootstrap tests run: cargo test -p hyperdb-bootstrap diff --git a/.github/workflows/npm-build-publish.yml b/.github/workflows/npm-build-publish.yml index ed8fffbc..15897370 100644 --- a/.github/workflows/npm-build-publish.yml +++ b/.github/workflows/npm-build-publish.yml @@ -23,8 +23,8 @@ permissions: env: CARGO_TERM_COLOR: always - HYPERD_VERSION: "0.0.26359" - HYPERD_BUILD_ID: "r07abb490" + HYPERD_VERSION: "0.0.26479" + HYPERD_BUILD_ID: "r96880f6a" jobs: verify-ci: @@ -96,23 +96,23 @@ jobs: os: macos-14 target: aarch64-apple-darwin hyperd-slug: macos-arm64 - hyperd-sha256: "434a5e7f95d914a7b328ac333b872fa044fabc6e23ad2aac9d0c9b762032b5ab" + hyperd-sha256: "65bd021b3d3470ac74728ec287866a3ee0dfd806daf2589670feb3580955ee95" # TODO: re-enable when macos-13 runners are more available # - platform: darwin-x64 # os: macos-13 # target: x86_64-apple-darwin # hyperd-slug: macos-x86_64 - # hyperd-sha256: "d5d3dae60ce071aed45e534fb6a4fc6c7edce47e31d664d131af9f76d9b3a2aa" + # hyperd-sha256: "6690669c8a6a6c7c6794c101beb31b83e1589d76cb14b0c817523069591f694c" - platform: linux-x64-gnu os: ubuntu-latest target: x86_64-unknown-linux-gnu hyperd-slug: linux-x86_64 - hyperd-sha256: "40e488c01ddc1ecaa53123a88fcf4150161a5828e1c22b475dd73e1cd9cbbbed" + hyperd-sha256: "c20be5b6874d319c7db01dcec763d7e65ae2483c0becc75f2914a58accc4f932" - platform: win32-x64-msvc os: windows-latest target: x86_64-pc-windows-msvc hyperd-slug: windows-x86_64 - hyperd-sha256: "8546e67501ed3f15e97c9a0ed6a0dfc56fa9f070878bac435a7e5f7d1bfb6c99" + hyperd-sha256: "3a400508e79c67ce9dcd8bd317ab164a61b5aed8ca880031ed4bbe6621514e1d" runs-on: ${{ matrix.os }} defaults: run: diff --git a/.github/workflows/rhel-compatibility.yml b/.github/workflows/rhel-compatibility.yml index b49dcd3b..728c1d6a 100644 --- a/.github/workflows/rhel-compatibility.yml +++ b/.github/workflows/rhel-compatibility.yml @@ -30,6 +30,14 @@ on: - '.github/workflows/rhel-compatibility.yml' workflow_dispatch: +# Serialize per ref: this job builds a container and the whole workspace, so +# successive pushes to a PR would otherwise stack multi-minute runs. Matches +# the pattern used by ci.yml, release.yml, release-please.yml and +# npm-build-publish.yml. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + permissions: contents: read @@ -38,6 +46,12 @@ env: # prerequisites" step. Pinned to the version this workspace builds against # locally (libprotoc 35.1). PROTOC_VERSION: '35.1' + # sha256 of protoc-${PROTOC_VERSION}-linux-x86_64.zip. Upstream publishes no + # checksum file for its releases, so this is computed from the asset and + # pinned here. Refresh it alongside PROTOC_VERSION: + # curl -fsSLO https://github.com/protocolbuffers/protobuf/releases/download/vX.Y/protoc-X.Y-linux-x86_64.zip + # sha256sum protoc-X.Y-linux-x86_64.zip + PROTOC_SHA256: '6930ebf62bd4ea607b98fff052596c6ee564b9835b4ce172c75a3f53ae9d91b7' jobs: rhel-native: @@ -88,10 +102,17 @@ jobs: # require a tool beyond cargo and rustc. The durable fix is to generate # the .rs files at publish time and vendor them into the crate; until # then CI supplies protoc explicitly. + # + # The archive is unpacked into /usr/local as root, so its integrity is + # verified first. Version-pinning alone does not help here: a retagged + # or compromised release would still be accepted. run: | - curl -fsSLO "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-x86_64.zip" - unzip -q "protoc-${PROTOC_VERSION}-linux-x86_64.zip" -d /usr/local - rm "protoc-${PROTOC_VERSION}-linux-x86_64.zip" + set -euo pipefail + ZIP="protoc-${PROTOC_VERSION}-linux-x86_64.zip" + curl -fsSLO "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/${ZIP}" + echo "${PROTOC_SHA256} ${ZIP}" | sha256sum -c - + unzip -q "${ZIP}" -d /usr/local + rm "${ZIP}" protoc --version - uses: actions/checkout@v7 @@ -123,4 +144,12 @@ jobs: env: RUSTFLAGS: '' CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER: cc - run: cargo check --workspace --locked --all-targets + run: | + set -euo pipefail + cargo check --workspace --locked --all-targets + # hyperdb-compile-check declares its own [workspace], so --workspace + # skips it -- yet release.yml publishes it, meaning the gate that + # proves "builds on Red Hat's toolchain with no rustup" would + # otherwise never cover a crate enterprise consumers can depend on. + cargo check --locked --all-targets \ + --manifest-path hyperdb-compile-check/Cargo.toml diff --git a/AGENTS.md b/AGENTS.md index d6d8c046..644ced28 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,7 @@ This is a **pure-Rust implementation** of the Hyper database API, using the Post **Key characteristics:** - 100% pure Rust (no FFI, no C dependencies) -- High performance (25M rows/sec inserts, 31M rows/sec queries single-connection; 48M / 73M across 4 connections) +- High performance on a single connection (100M-row benchmark, Apple M3 Max): 68.9M rows/sec inserts via the async `AsyncArrowInserter`, 25.0M rows/sec via the sync `Inserter`, 31.1M rows/sec full-scan queries via the sync path. See [docs/BENCHMARK_GUIDE.md](docs/BENCHMARK_GUIDE.md) for the multi-connection and per-platform figures. - Independent library (can be extracted from this repository) - Zero build system dependencies (uses standard Cargo) - **No feature flags on `hyperdb-api`** — every capability of the flagship crate (TLS, pooling, geography, transactions, chrono) is always available. A few companion crates do carry optional features; see [Feature Flags](#feature-flags) for the complete list. @@ -147,25 +147,21 @@ HYPERD_PATH=~/dev/bin/hyperd cargo test -p hyperdb-mcp --test attach_tests ### Editor Setup (VS Code / Windsurf / Cursor) -A few of our transitive deps (`rmcp`, `rmcp-macros`, `base64ct`, `clap_lex`) use `edition = "2024"`. Older copies of the rust-analyzer binary bundled with the VS Code extension reject that with: +**The entire workspace is `edition = "2024"`** as of 1.0.0 — not just a few transitive deps. Older copies of the rust-analyzer binary bundled with the VS Code extension reject that outright: -``` +```text failed to interpret `cargo metadata`'s json: unknown variant `2024` ``` -If you hit this, install rust-analyzer via rustup and point the extension at it from your **user** settings (not workspace settings — we intentionally don't commit this so contributors on newer extensions aren't forced to change anything): - -```bash -rustup component add rust-analyzer -``` - -Then in your user `settings.json`: +`rust-toolchain.toml` lists `rust-analyzer` in its `components`, so rustup installs a matching binary for you when the toolchain is provisioned — no manual `rustup component add` step. What you may still need is to point the extension at that binary rather than its bundled one, from your **user** settings (not workspace settings — we intentionally don't commit this, so contributors on newer extensions aren't forced to change anything): ```json "rust-analyzer.server.path": "rust-analyzer" ``` -The rustup-shipped binary tracks the active toolchain (1.85+ supports edition 2024) so it stays in lockstep with `cargo`. `"rust-analyzer"` with no path resolves via `$PATH` — the rustup shim under `~/.cargo/bin` on Unix or `%USERPROFILE%\.cargo\bin` on Windows. +The rustup-shipped binary tracks the active toolchain, so it stays in lockstep with `cargo`. `"rust-analyzer"` with no path resolves via `$PATH` — the rustup shim under `~/.cargo/bin` on Unix or `%USERPROFILE%\.cargo\bin` on Windows. + +One rust-analyzer quirk worth knowing, unrelated to the edition: with the `compile-time` feature enabled, `query_as!` validation depends on `derive(Table)` having registered the type in the same proc-macro process. rust-analyzer expands macros lazily and from cache, so it can expand a `query_as!` in a process where no derive ran. Validation detects that and skips rather than reporting a false "not registered" error, so the editor should stay quiet on code that `cargo check` accepts. ### Common Commands @@ -397,7 +393,7 @@ fail loudly when stale. - **Inserter API uses binary COPY protocol** - 10-100x faster than INSERT statements - **Streaming results** - Always process in chunks, never load all rows -- **Arrow batching** - Use `ArrowInserter` for maximum throughput (30M rows/sec single-connection, 48M+ across 4) +- **Arrow batching** - Use the async `AsyncArrowInserter` for maximum insert throughput: 68.9M rows/sec on a single connection, versus 25.0M rows/sec for the sync `Inserter`. Only the async variant is benchmarked at that rate — the sync `ArrowInserter` is not measured by the suite. Spending extra connections on an Arrow insert buys nothing on the benchmarked host. - **Release builds** - Use `--release` for benchmarks (debug is 10x+ slower) - **Connection pooling** - Use `pool` module for async high-concurrency scenarios @@ -474,9 +470,9 @@ All commit messages **must** follow the format `(): ` — 8. **Update the *per-crate* `CHANGELOG.md` for user-visible crate-level changes.** When a PR adds, changes, or removes any public API surface in a - publishable crate (`hyperdb-api`, `hyperdb-api-core`, `hyperdb-api-derive`, - `hyperdb-api-node`, `hyperdb-api-salesforce`, `hyperdb-bootstrap`, - `hyperdb-mcp`, `sea-query-hyperdb`), append a bullet to the + publishable crate (`hyperdb-api`, `hyperdb-api-core`, `hyperdb-compile-check`, + `hyperdb-api-derive`, `hyperdb-api-node`, `hyperdb-api-salesforce`, + `hyperdb-bootstrap`, `hyperdb-mcp`, `sea-query-hyperdb`), append a bullet to the `## [Unreleased]` section of that crate's `CHANGELOG.md` under the appropriate [Keep a Changelog](https://keepachangelog.com/) heading (`### Added`, `### Changed`, `### Deprecated`, `### Removed`, `### Fixed`, @@ -486,7 +482,7 @@ All commit messages **must** follow the format `(): ` — **Which changelog files you may edit** — this is the part that trips people up, because [CONTRIBUTING.md](CONTRIBUTING.md#what-contributors-do) says contributors do *not* hand-edit changelogs. Both rules are correct; they govern different files: - **Root [`CHANGELOG.md`](CHANGELOG.md) — never hand-edit.** It is release-please-generated, has no `## [Unreleased]` section, and is the only `changelog-path` in `release-please-config.json`. -- **The eight per-crate `CHANGELOG.md` files — hand-maintained.** Each carries exactly one `## [Unreleased]` section and none appear in release-please's `packages` or `extra-files`. This reminder applies to these. +- **The nine per-crate `CHANGELOG.md` files — hand-maintained.** Each carries exactly one `## [Unreleased]` section and none appear in release-please's `packages` or `extra-files`. This reminder applies to these. `hyperdb-compile-check` is the ninth: it is published (`release.yml` does so explicitly, since it declares its own `[workspace]` and `--workspace` cannot see it) but was missing from this list, which is why it had no changelog until 1.0.0. - **The npm sub-package changelogs** under `hyperdb-api-node/npm/*/` and `hyperdb-mcp/npm/*/` — leave alone; they have no `## [Unreleased]` section. 1. **Never invent `hyperd` flags or engine parameters.** Obtain `hyperd` via @@ -500,3 +496,41 @@ All commit messages **must** follow the format `(): ` — appearing to "run." 2. **Never report a test/build as passing without seeing real output.** Check exit codes. If a command produces no output for ~30s, treat it as **hanging/failed**, not passing, and say so explicitly. A green claim backed by no captured output is a defect, not a result — tests here start a real `hyperd` subprocess (`HyperProcess::drop()` stops it), so a misconfigured server hangs rather than erroring cleanly. + +3. **Run markdownlint on any Markdown you touch, before committing.** It is + **not** a CI gate, so nothing catches these for you — the only feedback is the + editor extension, and an agent working headless gets none at all. + + ```bash + npx markdownlint-cli2 + ``` + + No arguments: `.markdownlint-cli2.jsonc` supplies the globs and the path + exclusions. Rules live in `.markdownlint.json` (shared with the editor + extension); `.markdownlintignore` exists only because the extension reads it + and `markdownlint-cli2` does not. + + **There is a pre-existing backlog**, so a nonzero count is not automatically + yours. Judge new findings against the file's prior state — `git show + upstream/main:` and re-lint — rather than assuming, or you will "fix" + things that were never broken and miss the ones you introduced. + + Three traps that have actually bitten: + +- **Duplicate `### Fixed` / `### Added` siblings under one `## [Unreleased]`** + (MD024). Changelogs here often already have the section further down. Merge + your bullet into the existing one instead of adding a second heading — that + also keeps [Keep a Changelog](https://keepachangelog.com/) ordering. +- **Bare ``` fences** (MD040) need a language. Use `text` for command output, + ASCII diagrams, error messages, and templates. +- **Never bulk-auto-fix fences with a naive script.** A language-tagged + opening fence does not match a bare-fence test, so the *closing* fence gets + mistaken for an opening one and tagged — silently turning a terminator into a + new block. This corrupted 176 fences across 22 files once. Any such pass must + track fence state; prefer `markdownlint-cli2 --fix`, which is safe, and note + that it cannot fix MD040 because choosing a language needs judgement. + + Beware format-on-save: a Markdown formatter reformatting tables to satisfy + MD060 once stripped the README's badge links (`[![CI](img)](target)` became + `![CI](img)`) and split an inline link across a newline into two links. MD060 + is disabled in `.markdownlint.json` for exactly this reason. diff --git a/Cargo.toml b/Cargo.toml index 233b80cd..60ce97ce 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -67,7 +67,7 @@ serde_json = "1.0" url = "2.5" chrono = { version = "0.4", default-features = false, features = ["std", "clock"] } parking_lot = "0.12" -# hyperd-bootstrap dependencies +# hyperdb-bootstrap dependencies zip = { version = "8", default-features = false, features = ["deflate"] } toml = "1.1" clap = { version = "4", features = ["derive"] } diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 9bab1b1a..0f83bc84 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -664,25 +664,33 @@ in `hyperdb-api-core/src/types/special.rs`. ### Benchmark Results -**100M rows, 4 columns, optimized hyperd:** +[docs/BENCHMARK_GUIDE.md](docs/BENCHMARK_GUIDE.md) is the source of truth for +throughput numbers, across every platform and both transports. The +single-connection figures below are the headline subset, reproduced here so +this document stands on its own — **100M rows, 4 columns, Apple M3 Max, the +`hyperd` pin in `hyperd-version.toml`:** | Operation | Throughput | Notes | |-----------|-----------|-------| -| Insert (single-threaded) | 22M rows/sec | `Inserter`, HyperBinary format | -| Insert (multi-threaded) | 24M rows/sec | `ChunkSender`, 14 workers | -| Full table scan | 18M rows/sec | Streaming, 64K row chunks | -| Filtered query (10M rows) | 19M rows/sec | Single sensor_id filter | -| Aggregation | 0.04s | Server-side GROUP BY | - -**Insert Comparison (100M rows):** - -| Metric | Single-Threaded (`Inserter`) | Multi-Threaded (`ChunkSender`) | -|--------|------------------------------|--------------------------------| -| Time | ~4.5s | ~4.2s | -| Throughput | ~22M rows/sec | ~24M rows/sec | -| MB/sec | ~505 MB/sec | ~545 MB/sec | -| Memory | ~23 MB | ~1.3 GB (queued chunks) | -| Speedup | baseline | 1.08-1.10x | +| Insert, async | 68.9M rows/sec | `AsyncArrowInserter`, Arrow batches | +| Insert, sync | 25.0M rows/sec | `Inserter`, HyperBinary format | +| Full table scan, sync | 31.1M rows/sec | Streaming, 64K row chunks | +| Filtered query (10M rows), sync | 33.2M rows/sec | Single sensor_id filter | +| Aggregation | 0.05s | Server-side GROUP BY | + +Two things worth internalizing before you optimize against these: + +- **The async Arrow path is the fast insert path**, at roughly 2.7× the sync + `Inserter`. That gap is specific to `AsyncArrowInserter`; the sync + `ArrowInserter` is not measured by the suite, so do not assume it inherits + the number. +- **Spending more workers on a sync insert no longer buys throughput.** + `ChunkSender` across 4 workers now lands at or just below the + single-connection `Inserter`, so the historical ~1.1× multi-threaded speedup + no longer reproduces. It still changes the memory profile substantially — see + below. Multi-connection figures in general carry a wide run-to-run spread on + laptop-class hardware; read them from the guide, which documents that spread, + rather than treating them as precise. **Memory Behavior:** @@ -714,6 +722,14 @@ cargo test --release --test grpc_benchmark_tests benchmark_100m_complex -- --noc ### Performance Comparison: Rust vs C++ +> **Historical measurement — do not read the Rust column as current.** This +> table was taken on an older `hyperd` pin and toolchain, and its Rust figures +> no longer match the suite (see the table above). The C++ side has not been +> re-measured since, so the Rust column is deliberately left as it was rather +> than refreshed in place: mixing a current Rust number against a stale C++ +> number would make the comparison meaningless. Treat the *direction* as +> indicative and the magnitudes as needing a fresh head-to-head run. + **100M rows:** | Metric | C++ | Rust (single) | Rust (multi) | Winner | diff --git a/Makefile b/Makefile index 7afd1ffa..dc547842 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: clean clean-test-files clean-doc build build-api build-release build-api-release test test-api test-release test-api-release doc examples download-hyperd verify-hyperd-pin npm-pack check-rhel +.PHONY: clean clean-test-files clean-doc build build-api build-release build-api-release test test-api test-release test-api-release doc examples download-hyperd verify-hyperd-pin npm-pack check-rhel help # Environment variables for runtime # HYPERD_PATH points to the Hyper server executable. @@ -99,11 +99,22 @@ build-api-release: cargo build --release -p hyperdb-api-core -p hyperdb-api # Run tests (debug) with proper environment +# Mirrors the CI `test` job so a local pass means a CI pass. It previously +# covered only 3 of the 8 crates, which made its pass count look like whole- +# workspace coverage when it was a subset -- use `test-api` for the narrow run. +# +# Excludes match CI: hyperdb-api-node needs napi-rs plus a Node toolchain and +# has its own workflow, and hyperdb-bootstrap runs separately below because it +# does not need hyperd. hyperdb-compile-check declares its own [workspace], so +# `--workspace` cannot see it -- and until this target included it, nothing in +# CI or the Makefile ran that published crate's tests at all. test: @echo "Environment:" @echo " HYPERD_PATH=$(HYPERD_PATH)" @echo "" - cargo test -p hyperdb-api-core -p hyperdb-api -p hyperdb-mcp + cargo test --workspace --exclude hyperdb-api-node --exclude hyperdb-bootstrap + cargo test -p hyperdb-bootstrap + cargo test --manifest-path hyperdb-compile-check/Cargo.toml # Run tests (debug) - API only (no MCP/Node) test-api: diff --git a/README.md b/README.md index b19792ea..097324ae 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ executable, obtained with `make download-hyperd`. See ## Key Features - **Pure Rust** — no C library dependencies, standard `cargo build` -- **High Performance** — 25M rows/sec inserts, 31M rows/sec queries on a single connection; 48M / 73M across 4 (100M row benchmark, Apple M3 Max — see [benchmarks](docs/BENCHMARK_GUIDE.md)) +- **High Performance** — on a single connection: 68.9M rows/sec inserts with the async `AsyncArrowInserter`, 25.0M rows/sec with the sync `Inserter`, and 31.1M rows/sec full-scan queries (100M row benchmark, Apple M3 Max — see [benchmarks](docs/BENCHMARK_GUIDE.md) for multi-connection and per-platform figures) - **Memory Safe** — streaming by default, constant memory for billion-row results - **Dual Architecture** — sync (`Connection`) and async (`AsyncConnection`) APIs - **Typed Row Mapping** — `#[derive(FromRow)]` structs, including streaming `stream_as` for constant-memory typed queries @@ -174,7 +174,7 @@ fn main() -> Result<()> { .add_required_column("name", SqlType::text()); Catalog::new(&conn).create_table(&table_def)?; - // Insert data (COPY protocol, 25M+ rows/sec) + // Insert data (sync Inserter, COPY protocol, ~25M rows/sec) { let mut inserter = Inserter::new(&conn, &table_def)?; inserter.add_row(&[&1i32, &"Alice"])?; diff --git a/docs/BENCHMARK_GUIDE.md b/docs/BENCHMARK_GUIDE.md index 542134f6..d9b55ca2 100644 --- a/docs/BENCHMARK_GUIDE.md +++ b/docs/BENCHMARK_GUIDE.md @@ -146,14 +146,26 @@ block from the suite's stdout. - **Rust:** rustc 1.98.0 (88d9e12ae 2026-08-18) - **Node.js:** v24.18.0 (for the hyperdb-api-node bench) - **hyperdb-api version:** 1.0.0 -- **hyperd:** `0.0.26359.r07abb490` (the pin in `hyperd-version.toml`, arm64 native) +- **hyperd:** `0.0.26479.r96880f6a` (the pin in `hyperd-version.toml`, arm64 native) - **Date:** 2026-09-05 (Rust suite: median of 5 runs; Node bench: median of 15) +> **Partial re-measure at the `0.0.26479` pin.** The Rust tables below were +> first collected at `0.0.26359`, then re-run as an interleaved A/B against +> `0.0.26479` (median of 5 runs per engine, at both 100M and 10M). Exactly one +> workload moved: **`AsyncArrowInserter`, single connection**, whose rows are +> restated below. Every other single-connection figure reproduced within ±2%, +> and the `× 4` figures within their own much wider spread, so those rows are +> carried forward from the `0.0.26359` session rather than replaced by a fresh +> sample differing only by noise — re-rolling them would have published, for +> instance, a spurious −18% on `query.full_scan × 4`. Per-release history lives +> in [hyperd-release-benchmarks.md](hyperd-release-benchmarks.md). The Node.js +> figures further down were **not** re-measured and remain at `0.0.26359`. + #### Rust suite — 100M rows per workload, 4 parallel workers | Workload | Variant | Flavor | Rows | Time (s) | Rows/sec | MB/sec | |---|---|---|---:|---:|---:|---:| -| insert.bulk | AsyncArrowInserter | async | 100.00M | 3.360 | 29.76 M/s | 714.3 | +| insert.bulk | AsyncArrowInserter | async | 100.00M | 1.451 | 68.90 M/s | 1653.6 | | insert.bulk | AsyncArrowInserter × 4 | async | 100.00M | 2.063 | 48.47 M/s | 1163.4 | | insert.bulk | ChunkSender × 4 | sync | 100.00M | 4.049 | 24.70 M/s | 592.8 | | insert.bulk | Inserter (HyperBinary) | sync | 100.00M | 3.998 | 25.01 M/s | 600.3 | @@ -171,14 +183,17 @@ block from the suite's stdout. > **Read the `× 4` rows as order-of-magnitude only.** Measured over 5 runs on > this host, the multi-connection variants have a run-to-run spread of > **±20–61%**, because 4 workers contend on a 14-core laptop. The -> single-connection rows are stable to within ±2.4% and are the ones to -> compare across releases. +> single-connection rows are the ones to compare across releases: all but one +> are stable to within ±2.4%. The exception is `AsyncArrowInserter`, which +> swings ±25–35% run to run even on one connection — compare it only via +> medians of several runs, and only against another median. **Headline takeaways (Rust, macOS / M3 Max):** -- **Parallel reads are the standout** — `query.full_scan × 4` reaches **73 M rows/s / 1763 MB/s**, a 2.4× wall-clock speedup over the single-connection sync scan. Parallel inserts lead too, with `AsyncArrowInserter × 4` at **48 M rows/s / 1163 MB/s**. +- **Parallel reads are the standout** — `query.full_scan × 4` reaches roughly **73 M rows/s / 1763 MB/s**, very approximately 2× the single-connection sync scan. Per the note above these are order-of-magnitude figures, so do not read a precise speedup ratio out of them; the single-connection rows are the ones with a tight enough spread to compare. +- **Parallelism no longer helps Arrow inserts.** Since the `0.0.26479` engine, single-connection `AsyncArrowInserter` (68.9 M rows/s) outruns `AsyncArrowInserter × 4`, so spending connections on an Arrow insert buys nothing on this host. - **Sync beats async on single-connection reads.** `query.full_scan` sync runs 31.1 M rows/s against async's 24.9 M rows/s, and `query.filtered` 33.2 vs 26.9 M rows/s. Async wins only once it can use multiple connections, so prefer the sync path for a single streaming consumer and reach for async when you have concurrency to exploit. -- **Async still wins single-connection *inserts*** — `AsyncArrowInserter` at 29.8 M rows/s versus sync `Inserter` at 25.0 M rows/s. +- **Async dominates single-connection *inserts*** — `AsyncArrowInserter` at 68.9 M rows/s versus sync `Inserter` at 25.5 M rows/s, a 2.7× gap. This is the one figure the `0.0.26479` engine bump moved: **+127%** (30.4 → 68.9 M rows/s), reproduced as **+75%** at 10M. Both are medians of 5 interleaved runs whose old and new ranges do not overlap, so the gain survives this workload's wide ±25–35% spread. Sync inserts were unaffected. - **Single-connection scans are much faster than the previous entry** (18.8 → 31.1 M rows/s sync full-scan). Note this is *not* a controlled comparison: the prior numbers were taken on a different `hyperd`, rustc 1.94, and macOS 26.4, so the gain cannot be attributed to any single change. #### Node.js bench — 10M rows (same schema) @@ -216,6 +231,30 @@ eager scan exhausts the heap. For large reads through > unreliable. The sub-10 ms measurements in particular are dominated by one > GC pause or JIT decision. +#### Rust suite — 10M rows per workload, 4 parallel workers + +Same host and `hyperd` as the 100M table above, collected separately on +2026-09-05 (median of 5 runs). This exists so the Rust-vs-Node comparison +below is checkable: that comparison runs both harnesses at 10M, and quoting +Rust figures with only a 100M table published made them impossible to verify. + +| Workload | Variant | Flavor | Rows | Time (s) | Rows/sec | MB/sec | +|---|---|---|---:|---:|---:|---:| +| insert.bulk | AsyncArrowInserter | async | 10.00M | 0.202 | 49.39 M/s | 1185.4 | +| insert.bulk | AsyncArrowInserter × 4 | async | 10.00M | 0.263 | 37.97 M/s | 911.2 | +| insert.bulk | ChunkSender × 4 | sync | 10.00M | 0.435 | 23.00 M/s | 551.9 | +| insert.bulk | Inserter (HyperBinary) | sync | 10.00M | 0.423 | 23.64 M/s | 567.4 | +| insert.bulk | spawn_blocking+ChunkSender × 4 | async | 10.00M | 0.271 | 36.86 M/s | 884.7 | +| query.aggregation | 4 parallel connections | async | 40 | 0.035 | 1 K/s | 0.0 | +| query.aggregation | single connection | async | 10 | 0.007 | 1 K/s | 0.0 | +| query.aggregation | single connection | sync | 10 | 0.007 | 1 K/s | 0.0 | +| query.filtered | 4 parallel connections | async | 1.00M | 0.042 | 23.84 M/s | 286.0 | +| query.filtered | single connection | async | 1.00M | 0.040 | 25.29 M/s | 303.5 | +| query.filtered | single connection | sync | 1.00M | 0.032 | 31.42 M/s | 377.1 | +| query.full_scan | 4 parallel connections | async | 10.00M | 0.180 | 55.43 M/s | 1330.4 | +| query.full_scan | single connection | async | 10.00M | 0.405 | 24.69 M/s | 592.6 | +| query.full_scan | single connection | sync | 10.00M | 0.322 | 31.03 M/s | 744.6 | + #### Rust vs Node.js — 10M apples-to-apples Same schema, same dataset shape, **both harnesses run at 10M rows** so the @@ -225,18 +264,26 @@ building the Arrow table, and IPC-serializing it, all inside the measurement. | Workload | Rust (best) | Node (best) | Rust factor | |---|---|---|---:| -| insert.bulk | AsyncArrowInserter × 4 — 37.8 M/s / 907.9 MB/s | **ArrowInserter — 41.3 M/s / 991.7 MB/s** | **0.9× (Node ahead)** | -| insert.bulk (row API) | sync Inserter — **23.1 M/s / 553.1 MB/s** | RowInserter — 2.15 M/s / 51.5 MB/s | ~11× (CPU-bound JS encode) | -| query.full_scan | async × 4 — **54.3 M/s / 1302 MB/s** | executeQueryToArrow — 28.6 M/s / 685.7 MB/s | 1.9× | -| query.filtered | sync — **31.0 M/s / 372.2 MB/s** | executeQueryToArrow — 20.0 M/s / 480.0 MB/s | 1.6× | +| insert.bulk | **AsyncArrowInserter (1 conn) — 49.39 M/s / 1185.4 MB/s** | ArrowInserter — 41.3 M/s / 991.7 MB/s | **1.2×** | +| insert.bulk (row API) | sync Inserter — **23.64 M/s / 567.4 MB/s** | RowInserter — 2.15 M/s / 51.5 MB/s | ~11× (CPU-bound JS encode) | +| query.full_scan | async × 4 — **55.43 M/s / 1330.4 MB/s** | executeQueryToArrow — 28.6 M/s / 685.7 MB/s | 1.9× | +| query.filtered | sync — **31.42 M/s / 377.1 MB/s** | executeQueryToArrow — 20.0 M/s / 480.0 MB/s | 1.6× | | query.aggregation | sync — ~1 K/s | GROUP BY — 167 M/s | — (server-side; both latency-bound) | -Reading: on the **Arrow-IPC ingest path Node is at parity with Rust, and at -this scale slightly ahead.** That is a scale artifact rather than JS beating -native — Rust's `× 4` variant pays a fixed cost to spin up 4 workers and -connections, which it only amortizes on larger inputs (the same variant -reaches 48.5 M rows/s at 100M rows, comfortably ahead of Node). Read the row -as "the Arrow path costs you nothing at 10M," not as a language ranking. +Reading: on the **Arrow-IPC ingest path the two are within striking distance, +with Rust now ~1.2× ahead.** Note *which* Rust variant wins that row: the +single-connection `AsyncArrowInserter`, not the `× 4` one. At 10M rows the +parallel variant still pays a fixed cost to spin up 4 workers and connections +that it cannot amortize, and since the `0.0.26479` engine roughly doubled the +single-connection async Arrow path, that path is now the fastest Rust insert +at this scale outright. The honest reading of this row is "the Arrow path is +competitive from either language," not a language ranking. + +> **This row mixes engine versions.** The Rust figures are at `0.0.26479`; the +> Node figures were measured at `0.0.26359` and have not been re-run. Node's +> `ArrowInserter` goes through the same `hyperd` ingest path that got faster, +> so it plausibly gains too and the `1.2×` should be read as provisional until +> the Node bench is re-run at the current pin. On **reads** Rust keeps a genuine ~1.6–1.9× lead, since it never materializes an Arrow table in a JS heap. And the **row-by-row API remains the one to diff --git a/docs/hyperd-release-benchmarks.md b/docs/hyperd-release-benchmarks.md index 2dd1642c..6b20d53f 100644 --- a/docs/hyperd-release-benchmarks.md +++ b/docs/hyperd-release-benchmarks.md @@ -49,6 +49,8 @@ win is attributable to whichever side introduced it. | 0.0.25080 | r2bfd835b | 0.7.x | (baseline) | M-series (thermal, laptop) | 26.87 | 26.10 | 30.01 | — | Prior pin; measured as A/B baseline during the 0.0.26225 bump. | | 0.0.26225 | rbf04a855 | 0.7.x | 2026-08-07 | M-series (thermal, laptop) | 24.94 | 24.67 | 29.95 | sync insert −5–7%; async ~flat | See PR #219 (never shipped — held by the macOS-14 deadlock). | | 0.0.26359 | r07abb490 | 0.7.x | 2026-08-24 | M-series (thermal, laptop) | 24.12 | 24.62 | 30.07 | vs live 0.0.25080: Inserter −11%, ChunkSender −6%, async −4% | **Shipped fix for the macOS-14 deadlock that held 0.0.26225** (PR #237). Same-session 0.0.25080 A/B baseline: 27.17 / 26.12 / 31.33. Insert path carries cold-start variance and the new engine ran second, so treat the small insert deltas as soft. | +| 0.0.26359 | r07abb490 | 1.0.0-rc.1 | 2026-09-05 | M3 Max 14-core (thermal, laptop) | 25.43 | 24.52 | 30.35 | ~flat vs the `0.7.x` row above | A/B baseline leg for the 0.0.26479 bump, re-measured on the *current* API so the engine delta below is not confounded by the 0.7.x → 1.0.0-rc.1 API change. It reproduces the 2026-08-24 row within ~1–5%, which is the evidence that the API change did not move these numbers. | +| 0.0.26479 | r96880f6a | 1.0.0-rc.1 | 2026-09-05 | M3 Max 14-core (thermal, laptop) | 25.46 | 24.86 | **68.90** | **AsyncArrowInserter +127%**; Inserter +0.1%, ChunkSender +1.4% | Engine leg. Median of 5 runs per engine, **interleaved** (old, new, old, new, …) so thermal drift loads both sides equally. The async Arrow gain is not a noise artifact despite a 23–35% per-workload spread: the two sample ranges are **disjoint** (old 24.51–31.43, new 45.56–69.95), and the same jump reproduces at 10M rows (28.22 → 49.39, +75%, also disjoint). Sync insert paths are untouched. Multi-connection (`× 4`) deltas are withheld per Methodology — measured spread 17–41% on this host with deltas of *both* signs at different scales (`AsyncArrowInserter × 4` read +23% at 100M and −20% at 10M in the same session), so they carry no signal. | ## Query (single-connection, M rows/s) @@ -57,6 +59,8 @@ win is attributable to whichever side introduced it. | 0.0.25080 | r2bfd835b | 0.7.x | (baseline) | M-series (thermal, laptop) | 18.79 | 18.73 | 33.23 | 27.05 | — | Prior pin. | | 0.0.26225 | rbf04a855 | 0.7.x | 2026-08-07 | M-series (thermal, laptop) | 31.23 | 25.10 | 32.89 | 27.18 | **full_scan +66% sync / +34% async**; filtered ~flat | Large win on the dominant query path. All 1485 workspace tests pass; identical query results. Never shipped (macOS-14 deadlock). | | 0.0.26359 | r07abb490 | 0.7.x | 2026-08-24 | M-series (thermal, laptop) | 31.40 | 24.91 | 33.14 | 27.04 | **full_scan +67% sync / +33% async** vs live 0.0.25080; filtered ~flat | The full_scan win from the 0.262xx engine line survives into the deadlock-fixed build (PR #237). Same-session 0.0.25080 A/B baseline: 18.82 / 18.68 / 33.56 / 26.24. All 1485 tests pass; macOS-14 CI green (no deadlock). | +| 0.0.26359 | r07abb490 | 1.0.0-rc.1 | 2026-09-05 | M3 Max 14-core (thermal, laptop) | 31.29 | 24.82 | 33.54 | 26.57 | ~flat vs the `0.7.x` row above | A/B baseline leg for the 0.0.26479 bump, re-measured on the current API. Reproduces the 2026-08-24 row within ~1.5% on every column. | +| 0.0.26479 | r96880f6a | 1.0.0-rc.1 | 2026-09-05 | M3 Max 14-core (thermal, laptop) | 31.08 | 24.88 | 33.75 | 26.73 | **flat: −0.7% / +0.2% / +0.6% / +0.6%** | Engine leg, median of 5 interleaved runs. No query path moved: every delta is inside the ≤1.9% run-to-run spread these four workloads showed in the same session, so the query side of this bump is a no-op. The bump's one real change is on the insert side (see the table above). All 1586 workspace tests pass against this engine. | ## How to add a row diff --git a/docs/superpowers/plans/2026-09-05-post-rc-cleanup.md b/docs/superpowers/plans/2026-09-05-post-rc-cleanup.md new file mode 100644 index 00000000..1dae6608 --- /dev/null +++ b/docs/superpowers/plans/2026-09-05-post-rc-cleanup.md @@ -0,0 +1,378 @@ +# Post-`1.0.0-rc.1` Cleanup Implementation Plan + +Closes the findings left open when `1.0.0-rc.1` shipped. Nothing here blocks +the RC; the point is to land it before promoting to `1.0.0`, so the final +release carries no known defects. + +Sources: the three adversarial reviews run against PR #250 (data-path, +edition-2024 mechanics, CI/release claims), plus items discovered while cutting +the release. Every item below was re-verified against `main` at `033b2da` on +2026-09-05 — none are stale. + +**Scope decision: non-Node work first.** Tasks 1–9 touch Rust core, CI, and +docs. The `hyperdb-api-node` findings are deliberately deferred to Tasks 10–13 +so the N-API surface changes land as one reviewable group rather than being +interleaved with unrelated work. + +## Global constraints + +- **Branch:** `chore/post-rc-cleanup`, already open with the rust-analyzer + registry fix (`22b0a7a`). Base is `upstream/main`, not `origin/main` — the + fork's `main` lags. +- **Reminder 7 applies.** Any narrowing integer `as` cast encountered must + become a `TryFrom`, even incidentally. +- **Reminder 8 applies.** Public-API changes need a per-crate `CHANGELOG.md` + bullet under `## [Unreleased]`. Internal refactors do not. +- **Reminder 10 applies.** No task is complete without captured command output + and a checked exit code. +- **`hyperdb-compile-check` is outside the workspace.** `cargo clippy + --workspace` and `cargo test --workspace` skip it. Verify it explicitly with + `--manifest-path hyperdb-compile-check/Cargo.toml`. +- **`make test` covers only 3 of 8 crates** (`hyperdb-api-core`, `hyperdb-api`, + `hyperdb-mcp`). Use `cargo test --workspace` to match CI, which yields 1568 + rather than 1519. +- Do not touch `.agents/` or `.codex/` — untracked local config, deliberately + left alone. + +## Verification gate + +Every task ends with the subset relevant to it; the plan-completion gate runs +all of it: + +```sh +cargo fmt --all -- --check +cargo clippy --workspace --all-targets --all-features -- -D warnings +cargo clippy --manifest-path hyperdb-compile-check/Cargo.toml --all-targets -- -D warnings +RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p hyperdb-api -p hyperdb-api-core \ + -p hyperdb-api-derive -p hyperdb-api-node -p hyperdb-api-salesforce \ + -p hyperdb-bootstrap -p hyperdb-mcp -p sea-query-hyperdb +cargo deny check +cargo audit --deny warnings +cargo test --workspace +cargo test --manifest-path hyperdb-compile-check/Cargo.toml +cargo +1.88 check --workspace --locked --all-targets --all-features +cargo +1.88 check --locked --all-targets --manifest-path hyperdb-compile-check/Cargo.toml +``` + +## File map + +| Area | Files | +|---|---| +| Discarded Arrow errors | `hyperdb-api-core/src/client/grpc/authenticated_client.rs` | +| Dead crate name | `hyperdb-api/src/process.rs`, `hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs`, `hyperdb-bootstrap/src/lib.rs`, `hyperdb-bootstrap/src/error.rs`, `hyperdb-bootstrap/src/release.rs`, `hyperdb-bootstrap/tests/integration.rs` | +| Windows-gated lint | `hyperdb-api/src/process.rs` | +| Lint drift | `hyperdb-compile-check/Cargo.toml` | +| Stale comment | `hyperdb-api-core/src/protocol/types.rs` | +| CI hardening | `.github/workflows/rhel-compatibility.yml`, `.github/workflows/ci.yml`, `Makefile` | +| Docs | `AGENTS.md`, `docs/BENCHMARK_GUIDE.md`, `CONTRIBUTING.md` | +| Node (deferred) | `hyperdb-api-node/index.d.ts`, `hyperdb-api-node/src/{inserter,columnar,result}.rs` | + +--- + +## Task 1: Stop silently discarding Arrow decode errors + +**Highest correctness weight in the plan.** `authenticated_client.rs:976` and +`:1064` both do `if let Ok(batch) = batch_result`, dropping per-batch decode +failures on the floor. A corrupt batch mid-stream therefore yields a *silently +partial* label map from `get_table_labels` / `get_column_labels` — the caller +cannot distinguish "this table has no labels" from "decoding failed halfway". + +The `#[expect(clippy::manual_flatten)]` waivers that previously documented this +as deliberate were removed during the edition-2024 let-chain sweep, which took +the recorded intent with them. That is why this looks like an oversight now. + +**Decide first, then implement.** Two defensible outcomes: + +1. **Propagate** — return the decode error. Correct, but changes behavior for + any caller currently tolerating partial results, so it needs a + `hyperdb-api-core` changelog bullet and a look at both call sites' callers. +2. **Keep discarding, but record why** — restore the intent in a comment and + log at `warn` so a partial map is at least observable. + +Option 1 is preferred unless a caller depends on partial success. Establish +that by reading the callers of both functions before choosing. + +**Verify:** `cargo test --workspace`; add a unit test that feeds a truncated +Arrow IPC stream and asserts the chosen behavior (error propagated, or warning +emitted and result marked partial). + +## Task 2: Fix the dead `hyperd-bootstrap` command in user-facing errors + +`hyperdb-api/src/process.rs:286` instructs users to run: + +```text +cargo run -p hyperd-bootstrap -- download +``` + +That package has not existed since the rename to `hyperdb-bootstrap`; the +command fails with *"package ID specification `hyperd-bootstrap` did not match +any packages"*. This is the **first thing a new user sees** when `HYPERD_PATH` +is unset, so it is the highest annoyance-per-minute item here. + +Also stale, in descending user visibility: + +- `hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs:169` — runtime error text + `run \`hyperd-bootstrap download\` first` +- `hyperdb-api/src/process.rs:259`, `:269` — doc comments +- `hyperdb-bootstrap/src/lib.rs:9`, `error.rs:4`, `release.rs:23`, + `tests/integration.rs:4` — doc comments + +Fix the two runtime strings first; the doc comments are cosmetic but should go +in the same pass so the name is consistent. + +**Verify:** `grep -rn 'hyperd-bootstrap' --include='*.rs' .` returns nothing +outside `hyperd-version.toml` paths (where `hyperdb-bootstrap/hyperd-version.toml` +is a legitimate path, not the crate name). Then confirm the suggested command +actually runs: `cargo run -p hyperdb-bootstrap --bin hyperdb-bootstrap -- --help`. + +## Task 3: Fix the `collapsible_if` the let-chain sweep could not see + +`hyperdb-api/src/process.rs:667-671`: + +```rust +if pipe_name.is_some() { + if let Some(ref pname) = pipe_name { + return ConnectionEndpoint::named_pipe(".", pname); + } +} +``` + +This is the exact shape the edition-2024 sweep flattened 127 of, but it sits +inside `#[cfg(windows)]`. CI's clippy job runs `ubuntu-latest` only, and +`cfg`-stripped code is removed before lints run, so it was never linted. The +claim "all 127 fixed" is therefore Linux-scoped. + +The `is_some()` guard is dead weight — reduce to `if let Some(ref pname) = +pipe_name`. This is the only surviving instance found across roughly 75 +platform-gated regions. + +Blast radius is currently limited because neither `Makefile` nor `build.ps1` +exposes a clippy target, so a Windows contributor only hits it by running +clippy by hand. Consider adding `windows-latest` to the clippy matrix — +`ci.yml`'s own comment already anticipates this ("If a lint ever diverges by +target (rare), broaden the matrix"). + +**Verify:** `cargo clippy --target x86_64-pc-windows-msvc` if a Windows target +is installable locally; otherwise rely on the broadened CI matrix. + +## Task 4: Close the `hyperdb-compile-check` lint drift + +Root `Cargo.toml:147-148` promotes `missing_errors_doc` and +`missing_panics_doc` to `deny`. `hyperdb-compile-check/Cargo.toml:75-76` still +says `warn`. That crate declares its own `[workspace]`, so it cannot inherit +the workspace lint table and instead duplicates it verbatim — and the commit +that promoted the levels updated only the root copy. + +It is also excluded from `cargo clippy --workspace`, so the "measured zero +violations" claim behind the promotion never covered it. + +Mirror the two `deny` levels, then confirm the crate actually satisfies them. +If it does not, that is a finding rather than a reason to skip: fix the missing +sections. + +**Verify:** `cargo clippy --manifest-path hyperdb-compile-check/Cargo.toml +--all-targets -- -D warnings`. + +## Task 5: Retarget the stale `split_at_checked` comment + +`hyperdb-api-core/src/protocol/types.rs:358` says *"`split_at_checked` plus +`slice::get` cannot overflow"*, but the shipped implementation uses +`split_first_chunk::<4>()` (lines 304 and 340). `split_at_checked` was the +earlier attempt that commit `d7157c3` describes discarding. Retarget the +comment so the next reader is not sent looking for a call that is not there. + +**Verify:** `cargo doc` clean; grep confirms no remaining `split_at_checked` +reference. + +## Task 6: Harden the RHEL workflow + +Three separate issues in `.github/workflows/rhel-compatibility.yml`: + +1. **`protoc` is fetched with no integrity check.** It is downloaded over + HTTPS and unzipped into `/usr/local` as root. Version-pinning is already + correct (`PROTOC_VERSION: '35.1'`, matching `Makefile:169`), so drift is not + the risk — a retagged or compromised release is. Add a `sha256sum -c` + against a pinned digest, or `gh attestation verify`. +2. **No `concurrency` group** (verified: zero `concurrency` keys). Every other + workflow in the repo has one. Successive pushes to a PR stack multi-minute + container jobs. Mirror `ci.yml`: `group: ${{ github.workflow }}-${{ github.ref }}` + with `cancel-in-progress` on `pull_request`. +3. **The gate skips `hyperdb-compile-check`.** Its `cargo check --workspace` + cannot see that crate, yet `release.yml` publishes it — so the one gate that + proves "builds on Red Hat's toolchain with no rustup" never covers a crate + enterprise consumers can depend on. The new `msrv` job already checks it + separately; do the same here. + +Also fold in the trivial `Makefile` fix: `help` is missing from `.PHONY` +(verified), which is pre-existing. + +**Verify:** `make check-rhel` locally; then confirm the workflow is green on +the PR. Note the job is path-filtered, so a docs-only commit will not run it — +touch a `.rs` or `Cargo.toml` file to exercise it. + +## Task 7: Correct the stale `AGENTS.md` Editor Setup section + +`AGENTS.md:148-166` is now wrong on both of its points: + +- Line 150 frames edition 2024 as something *"a few of our transitive deps + (`rmcp`, `rmcp-macros`, `base64ct`, `clap_lex`)"* use. The entire workspace + is edition 2024 as of `1.0.0-rc.1`. +- It instructs contributors to run `rustup component add rust-analyzer` by + hand, which `rust-toolchain.toml` now does automatically via its `components` + entry. Commit `091327c` claimed to obviate that instruction; the batched docs + pass did not follow through. + +Keep the `rust-analyzer.server.path` guidance — that part is still useful and +deliberately not committed to workspace settings. + +**Verify:** read-through; no automated gate covers this. + +## Task 8: Make the benchmark comparison traceable + +`docs/BENCHMARK_GUIDE.md`'s "Rust vs Node.js — 10M apples-to-apples" table +cites Rust-at-10M figures, but that table is not in the document (verified: +zero "Rust suite — 10M" sections), so not one number in the comparison can be +checked against a source. The 100M table sits directly above it, which invites +the reader to assume — wrongly — that the comparison came from there. + +Two fixes: + +1. Add the Rust 10M table, or state the exact invocation that produced the + column. The data exists: `bench_ab/final/rust10m-r*.json`, five runs, median + taken. +2. `:232` still reads `~1 K/s` for the aggregation row. Replace with the + measured figure. + +While there, re-check the callout added in the macOS refresh: it warns that the +`× 4` rows carry ±20–61% spread and that single-connection rows are the ones to +compare, yet the very next takeaway leads with a `× 4` number and derives a +two-significant-figure "2.4× speedup" from it. + +**Verify:** every figure in the comparison table appears in a table in the same +document, or has its invocation stated. + +## Task 9: Reconcile `make test` with CI, and give `hyperdb-compile-check` a changelog + +Two loose ends that mislead rather than break: + +- **`make test` covers 3 of 8 crates**, so the "1519 passed" figure quoted + throughout the 1.88 uplift is a subset; CI's `cargo test --workspace` gives + 1568. Either broaden the target to match CI, or rename it and document what + it covers, so a contributor running it locally is not misled about coverage. +- **`hyperdb-compile-check` has no `CHANGELOG.md`** despite being published to + crates.io by `release.yml:264`. It is absent from AGENTS.md reminder 8's + eight-crate list, which is why it was missed. Either add one and extend the + list to nine, or state explicitly why it is exempt. + +**Verify:** `cargo test --workspace` count matches whatever the Makefile target +now claims. + +--- + +## Deferred: `hyperdb-api-node` (Tasks 10–13) + +Grouped so the N-API surface lands as one reviewable change. Ordered by weight. + +### Task 10: Guard the `expect()` at the N-API boundary + +`columnar.rs:107` is sound today — the bounds scan directly above makes the +`expect()` unreachable, verified by inspection: `LO..=HI` is inclusive, both +bounds are widening `i32 → i64` casts, and both passes iterate the same `v` +binding under one `&self` borrow with no interior mutability. + +The reason to act anyway: **napi 3.10 does not wrap `#[napi]` bodies in +`catch_unwind`** — no such call exists in its `src/`. A panic here unwinds into +V8's C++ frames, which is undefined behavior, not a JS exception. The +correctness rests entirely on the two passes staying coupled, and nothing +enforces that against a future edit. + +Add a `debug_assert!` in the narrowing pass, or move scan and map into one +function commented as a unit. Also add the missing `# Panics` section — +the workspace denies `missing_panics_doc` and this function has none (verified), +which pulls against commit `d7157c3`, where `protocol/types.rs` was +restructured specifically to avoid an `expect()` for that lint. + +### Task 11: Name the row and column in insert rejection errors + +A caller who buffers a million rows and calls `execute()` currently gets: + +```text +row encoding error: value 70000 does not fit the destination SMALLINT column (valid range -32768..=32767) +``` + +The whole batch fails with no way to locate the offending datum. This blunts +the Task 2.6 fix: the old behavior silently wrote a wrong number, and the new +behavior says a wrong number exists *somewhere* in your data. + +`encode_rows` already has both coordinates — `col_idx` from the `enumerate()` +at `inserter.rs:300`, and the row index one `.enumerate()` away on the `for row +in rows` loop at `:299`. Thread them into the `map_err` at `:305`. + +### Task 12: Add the missing `@throws` to `index.d.ts` write paths + +Both *read* getters were updated (`getInt32` at `:616`, `getInt32Column` at +`:230`). The three *write* paths were not — and by the change's own framing +those are the more serious half, since that is where corruption reached the +`.hyper` file. `RowInserter.addRow`, `addRows`, and `execute` carry no +`@throws`, so a TypeScript consumer gets no signal that a previously-succeeding +insert can now reject. + +Put it on `execute()` — that is where the error actually surfaces, not +`addRow()`. `addColumnar` has the same gap: a value routed through the +`int64Columns` bucket into an `INT` column now goes through `narrow_i32` +(`inserter.rs:392`) and throws, which its doc block at `:433` does not mention. + +### Task 13: Narrow the `cast_precision_loss` allow to its stated reason + +`columnar.rs:4-7` allows `clippy::cast_precision_loss` crate-wide with the +reason *"diagnostic metric output; bounded chunk sizes"*. That reason does not +describe what the annotation actually permits: it also covers +`get_float64_column`'s `x as f64` on an `Int64` column (`:131`) and +`result.rs:310`, which are data-path conversions losing precision above 2^53 on +real user values. + +The behavior is fine and matches `getInt64Column`'s documented caveat. Only the +justification is wrong. Narrow the allow to the specific diagnostic sites, or +widen the reason to name the data-path conversions honestly. + +--- + +## Not in scope: separate investigations + +- **macOS IPC is broken.** `BENCH_TRANSPORT=ipc` fails because `hyperd` never + creates the Unix socket the client dials + (`…/hyper-/domain/hyper`, ENOENT). It reproduces at the pre-migration + baseline, so it predates the 1.88 uplift, and macOS IPC has never been + captured in `BENCHMARK_GUIDE.md` — there is no recorded state it regressed + from. Needs its own investigation, not a cleanup task. +- **The `qs` advisory cannot be fixed here.** Patched in 6.16.0, but the + registry enforces a release-age cutoff making anything newer than 2026-08-29 + uninstallable, and every installable version (≤ 6.15.3) is affected. An + `overrides` pin was tried and reverted because it broke `npm install` + outright. Affects only the hyper-explorer example, which ships in no + published package. +- **Float→integer saturation** still hands JS a plausible wrong number: + `getInt32()` on a `DOUBLE` holding `5e9` returns `2147483647`. Deliberate and + disclosed in rustdoc, `index.d.ts` and the changelog, but it is the same + failure mode the narrowing work set out to eliminate. Worth an issue, not a + fix in this plan — it is not a regression. +- **~134 markdown lint items** — 69 untagged code fences, 28 over-long lines, + and heading-structure rules. Needs per-block judgment: an automated pass + corrupted 176 fences by mistaking closing fences for opening ones, because a + language-tagged opening fence does not match a bare-fence test. Any retry + must track fence state. +- **`scrape_dc_sql_reference.py`** references `scripts/n.md` and `scrape_n.py`, + which look like a sed/rename mishap in that script. Cosmetic, but it makes + the documented refresh command wrong. + +## Plan completion gate + +1. Tasks 1–9 landed, each with captured output. +2. The full verification gate above exits 0 on every command. +3. CI green on all 17 checks, including `msrv (1.88)`, `doc`, and the RHEL job + — with a `.rs` or `Cargo.toml` file touched so the path-filtered RHEL job + actually runs. +4. Per-crate changelog bullets added for any public-API change (Task 1 if it + propagates; Tasks 11–12 when the Node group lands). +5. `EXECUTION-LOG.md`-style record appended to this document with measurements, + matching the practice established during the 1.88 uplift. diff --git a/hyperdb-api-core/CHANGELOG.md b/hyperdb-api-core/CHANGELOG.md index cc97b042..177021dd 100644 --- a/hyperdb-api-core/CHANGELOG.md +++ b/hyperdb-api-core/CHANGELOG.md @@ -26,6 +26,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Fixed +- **`AuthenticatedGrpcClient::get_table_labels` and `get_column_labels` now + report Arrow failures instead of returning a partial map.** Both iterated + record batches with `if let Ok(batch) = batch_result`, so a decode failure + mid-stream was discarded and the caller received a map covering only the + batches that happened to decode — indistinguishable from "this table defines + no labels", which is the worst failure mode for metadata used to render UI. + A schema mismatch was swallowed the same way, by the `downcast_ref` tuple in + the same `if let` chain. + + Both now return `Err`. A batch projecting fewer than two columns is also + reported rather than panicking: `RecordBatch::column(1)` panics out of + bounds, which the previous code never guarded. + + Callers that prefer the old behavior can keep it explicitly with + `.unwrap_or_default()`. The parsing itself is unchanged — JSON + `displayName` extraction, verbatim passthrough for plain descriptions, and + skipping NULL rows all behave as before, now covered by unit tests. + - `text_from_hyper_binary` and `bytea_from_hyper_binary` no longer risk a `usize` overflow on 32-bit targets. Both read a `u32` length prefix, widened it to `usize`, and bounds-checked with `buf.len() < 4 + len`; where `usize` diff --git a/hyperdb-api-core/src/client/grpc/authenticated_client.rs b/hyperdb-api-core/src/client/grpc/authenticated_client.rs index 67c1b9fa..62fcc0c2 100644 --- a/hyperdb-api-core/src/client/grpc/authenticated_client.rs +++ b/hyperdb-api-core/src/client/grpc/authenticated_client.rs @@ -953,8 +953,6 @@ impl AuthenticatedGrpcClient { &mut self, schema: &str, ) -> Result> { - let mut labels = std::collections::HashMap::new(); - let query = format!( r"SELECT c.relname as table_name, COALESCE(d.description, c.relname) as label @@ -966,56 +964,7 @@ impl AuthenticatedGrpcClient { ); let result = self.execute_query(&query).await?; - let reader = arrow::ipc::reader::StreamReader::try_new( - std::io::Cursor::new(result.arrow_data()), - None, - ) - .map_err(|e| crate::client::Error::other(format!("Failed to parse Arrow data: {e}")))?; - - for batch_result in reader { - if let Ok(batch) = batch_result - && let (Some(name_arr), Some(label_arr)) = ( - batch - .column(0) - .as_any() - .downcast_ref::(), - batch - .column(1) - .as_any() - .downcast_ref::(), - ) - { - for i in 0..batch.num_rows() { - use arrow::array::Array; - if !name_arr.is_null(i) && !label_arr.is_null(i) { - let table_name = name_arr.value(i).to_string(); - let label_raw = label_arr.value(i); - - // Parse JSON to extract displayName: {"displayName":"value"} - let label = if label_raw.starts_with('{') { - if let Ok(value) = serde_json::from_str::(label_raw) - { - value - .get("displayName") - .and_then(|v| v.as_str()) - .map_or_else( - || label_raw.to_string(), - std::string::ToString::to_string, - ) - } else { - label_raw.to_string() - } - } else { - label_raw.to_string() - }; - - labels.insert(table_name, label); - } - } - } - } - - Ok(labels) + parse_label_pairs(&result.arrow_data()) } /// Returns a map of column names to their display labels for a given table. @@ -1040,8 +989,6 @@ impl AuthenticatedGrpcClient { schema: &str, table: &str, ) -> Result> { - let mut labels = std::collections::HashMap::new(); - let query = format!( r"SELECT a.attname as column_name, COALESCE(d.description, a.attname) as label @@ -1054,56 +1001,7 @@ impl AuthenticatedGrpcClient { ); let result = self.execute_query(&query).await?; - let reader = arrow::ipc::reader::StreamReader::try_new( - std::io::Cursor::new(result.arrow_data()), - None, - ) - .map_err(|e| crate::client::Error::other(format!("Failed to parse Arrow data: {e}")))?; - - for batch_result in reader { - if let Ok(batch) = batch_result - && let (Some(name_arr), Some(label_arr)) = ( - batch - .column(0) - .as_any() - .downcast_ref::(), - batch - .column(1) - .as_any() - .downcast_ref::(), - ) - { - for i in 0..batch.num_rows() { - use arrow::array::Array; - if !name_arr.is_null(i) && !label_arr.is_null(i) { - let col_name = name_arr.value(i).to_string(); - let label_raw = label_arr.value(i); - - // Parse JSON to extract displayName: {"displayName":"value"} - let label = if label_raw.starts_with('{') { - if let Ok(value) = serde_json::from_str::(label_raw) - { - value - .get("displayName") - .and_then(|v| v.as_str()) - .map_or_else( - || label_raw.to_string(), - std::string::ToString::to_string, - ) - } else { - label_raw.to_string() - } - } else { - label_raw.to_string() - }; - - labels.insert(col_name, label); - } - } - } - } - - Ok(labels) + parse_label_pairs(&result.arrow_data()) } /// Ensures we have a valid DC JWT, refreshing if necessary. @@ -1620,3 +1518,234 @@ impl AuthenticatedGrpcClientSync { .block_on(self.inner.get_column_labels(schema, table)) } } + +/// Parses an Arrow IPC stream of `(name, label)` text pairs into a map. +/// +/// Shared by [`AuthenticatedGrpcClient::get_table_labels`] and +/// [`AuthenticatedGrpcClient::get_column_labels`], whose queries project the +/// same two `TEXT` columns and differ only in the catalog they read. +/// +/// Data Cloud stores labels as JSON in `pg_description` +/// (`{"displayName":"Label"}`). A label that is not a JSON object, or whose +/// JSON lacks `displayName`, passes through verbatim — that fallback is +/// intentional, since plain-text descriptions are valid. +/// +/// Rows where either column is NULL are skipped: the queries `COALESCE` the +/// description to the identifier, so a NULL means the catalog row itself is +/// unusable rather than that the label is absent. +/// +/// # Errors +/// +/// Returns [`crate::client::Error`] if the stream cannot be opened, a record +/// batch fails to decode, a batch projects fewer than two columns, or either +/// of the first two columns is not a `StringArray`. +/// +/// Each of those used to be swallowed — the decode error and the type +/// mismatch by an `if let` chain, the short batch by an outright panic in +/// `RecordBatch::column`. Swallowing them produced a partial map that a caller +/// could not distinguish from "this table defines no labels", which is the +/// worst possible failure for metadata used to render UI. +fn parse_label_pairs(arrow_data: &[u8]) -> Result> { + use arrow::array::{Array, StringArray}; + + let mut labels = std::collections::HashMap::new(); + + let reader = arrow::ipc::reader::StreamReader::try_new(std::io::Cursor::new(arrow_data), None) + .map_err(|e| crate::client::Error::other(format!("Failed to parse Arrow data: {e}")))?; + + for batch_result in reader { + let batch = batch_result.map_err(|e| { + crate::client::Error::other(format!("Failed to decode Arrow record batch: {e}")) + })?; + + // `RecordBatch::column` panics out of bounds, so check before indexing. + if batch.num_columns() < 2 { + return Err(crate::client::Error::other(format!( + "label query must project 2 columns, got {}", + batch.num_columns() + ))); + } + + let downcast = |idx: usize| -> Result<&StringArray> { + batch + .column(idx) + .as_any() + .downcast_ref::() + .ok_or_else(|| { + crate::client::Error::other(format!( + "label query column {idx} must be TEXT, got {:?}", + batch.column(idx).data_type() + )) + }) + }; + let name_arr = downcast(0)?; + let label_arr = downcast(1)?; + + for i in 0..batch.num_rows() { + if name_arr.is_null(i) || label_arr.is_null(i) { + continue; + } + let label_raw = label_arr.value(i); + + // Only attempt JSON when it looks like an object; a bare + // description is the common case and need not go through serde. + let label = if label_raw.starts_with('{') { + serde_json::from_str::(label_raw) + .ok() + .as_ref() + .and_then(|v| v.get("displayName")) + .and_then(serde_json::Value::as_str) + .map_or_else(|| label_raw.to_string(), std::string::ToString::to_string) + } else { + label_raw.to_string() + }; + + labels.insert(name_arr.value(i).to_string(), label); + } + } + + Ok(labels) +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use arrow::array::{ArrayRef, Int32Array, StringArray}; + use arrow::datatypes::{DataType, Field, Schema}; + use arrow::record_batch::RecordBatch; + + use super::parse_label_pairs; + + /// Encodes one record batch as an Arrow IPC stream, as the server would. + fn ipc_stream(batch: &RecordBatch) -> Vec { + let mut buf = Vec::new(); + { + let mut writer = arrow::ipc::writer::StreamWriter::try_new(&mut buf, &batch.schema()) + .expect("create IPC writer"); + writer.write(batch).expect("write batch"); + writer.finish().expect("finish stream"); + } + buf + } + + fn two_text_batch(names: Vec<&str>, labels: Vec<&str>) -> RecordBatch { + let schema = Schema::new(vec![ + Field::new("name", DataType::Utf8, true), + Field::new("label", DataType::Utf8, true), + ]); + RecordBatch::try_new( + Arc::new(schema), + vec![ + Arc::new(StringArray::from(names)) as ArrayRef, + Arc::new(StringArray::from(labels)) as ArrayRef, + ], + ) + .expect("build batch") + } + + #[test] + fn extracts_display_name_from_json_and_passes_plain_text_through() { + let batch = two_text_batch( + vec!["accounts", "orders", "leads"], + vec![ + r#"{"displayName":"Accounts"}"#, + "Plain Description", + // JSON object without displayName falls back to the raw text. + r#"{"other":"x"}"#, + ], + ); + let labels = parse_label_pairs(&ipc_stream(&batch)).expect("valid stream parses"); + + assert_eq!(labels.get("accounts").map(String::as_str), Some("Accounts")); + assert_eq!( + labels.get("orders").map(String::as_str), + Some("Plain Description") + ); + assert_eq!( + labels.get("leads").map(String::as_str), + Some(r#"{"other":"x"}"#), + "a JSON object without displayName must pass through verbatim" + ); + } + + #[test] + fn truncated_stream_is_an_error_not_a_partial_map() { + // The regression this guards: a decode failure used to be swallowed by + // `if let Ok(batch)`, yielding an empty map that a caller could not + // tell apart from "this table defines no labels". + let full = ipc_stream(&two_text_batch(vec!["accounts"], vec!["Accounts"])); + let truncated = &full[..full.len() / 2]; + + let err = parse_label_pairs(truncated) + .expect_err("a truncated Arrow stream must not silently yield a partial map"); + let msg = format!("{err}"); + assert!( + msg.contains("Arrow"), + "error should name the Arrow failure, got: {msg}" + ); + } + + #[test] + fn non_text_column_is_an_error() { + // Previously the `downcast_ref` tuple silently skipped the whole batch. + let schema = Schema::new(vec![ + Field::new("name", DataType::Utf8, true), + Field::new("label", DataType::Int32, true), + ]); + let batch = RecordBatch::try_new( + Arc::new(schema), + vec![ + Arc::new(StringArray::from(vec!["accounts"])) as ArrayRef, + Arc::new(Int32Array::from(vec![1])) as ArrayRef, + ], + ) + .expect("build batch"); + + let err = parse_label_pairs(&ipc_stream(&batch)) + .expect_err("a non-TEXT label column must be reported, not skipped"); + assert!( + format!("{err}").contains("must be TEXT"), + "error should explain the type mismatch, got: {err}" + ); + } + + #[test] + fn single_column_batch_errors_instead_of_panicking() { + // `RecordBatch::column(1)` panics out of bounds; the guard turns that + // into a diagnosable error. + let schema = Schema::new(vec![Field::new("name", DataType::Utf8, true)]); + let batch = RecordBatch::try_new( + Arc::new(schema), + vec![Arc::new(StringArray::from(vec!["accounts"])) as ArrayRef], + ) + .expect("build batch"); + + let err = parse_label_pairs(&ipc_stream(&batch)) + .expect_err("a one-column batch must error rather than panic"); + assert!( + format!("{err}").contains("must project 2 columns"), + "error should name the column-count problem, got: {err}" + ); + } + + #[test] + fn null_rows_are_skipped_without_failing() { + let schema = Schema::new(vec![ + Field::new("name", DataType::Utf8, true), + Field::new("label", DataType::Utf8, true), + ]); + let batch = RecordBatch::try_new( + Arc::new(schema), + vec![ + Arc::new(StringArray::from(vec![Some("accounts"), None])) as ArrayRef, + Arc::new(StringArray::from(vec![Some("Accounts"), Some("Orphan")])) as ArrayRef, + ], + ) + .expect("build batch"); + + let labels = parse_label_pairs(&ipc_stream(&batch)).expect("nulls are not an error"); + assert_eq!(labels.len(), 1, "the NULL-name row should be skipped"); + assert_eq!(labels.get("accounts").map(String::as_str), Some("Accounts")); + } +} diff --git a/hyperdb-api-core/src/protocol/types.rs b/hyperdb-api-core/src/protocol/types.rs index 662c822d..b9f62ce4 100644 --- a/hyperdb-api-core/src/protocol/types.rs +++ b/hyperdb-api-core/src/protocol/types.rs @@ -355,8 +355,9 @@ mod tests { /// trusted. `u32::MAX` is the interesting value: the previous /// implementation computed `4 + len`, which overflows `usize` on a 32-bit /// target and wraps to a small number, making the bounds check pass and - /// the subsequent slice index panic. `split_at_checked` plus - /// `slice::get` cannot overflow. + /// the subsequent slice index panic. `split_first_chunk::<4>` followed by + /// `slice::get` removes the arithmetic entirely, so there is nothing left + /// to overflow. #[test] fn oversized_declared_length_is_rejected() { let mut buf = BytesMut::new(); diff --git a/hyperdb-api-derive/CHANGELOG.md b/hyperdb-api-derive/CHANGELOG.md index 459f91dd..3cab5955 100644 --- a/hyperdb-api-derive/CHANGELOG.md +++ b/hyperdb-api-derive/CHANGELOG.md @@ -18,6 +18,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Fixed +- `query_as!` and `query_scalar!` no longer report false "not registered" + errors in rust-analyzer. Registration is a process-global side effect of + expanding `derive(Table)`, which under `cargo` always happens before + function-body macros in the same host process. rust-analyzer's + `proc-macro-srv` is long-lived and expands lazily, out of order, and from + cache, so a `query_as!` could be re-expanded in a process where no derive had + run — yielding a red squiggle on code that `cargo check` compiles cleanly. + Validation now treats a completely empty registry as "no information" and + skips, rather than concluding the type is unregistered. Genuine diagnostics + are unaffected: once anything is registered, a miss is still a real miss. + - Intra-doc links on `query_as!` and `query_scalar!` now resolve. They referenced `hyperdb_api` types, which this crate deliberately does not depend on in order to break the `hyperdb-api` -> derive -> `hyperdb-compile-check` diff --git a/hyperdb-api-derive/src/lib.rs b/hyperdb-api-derive/src/lib.rs index 1e601be5..de82d583 100644 --- a/hyperdb-api-derive/src/lib.rs +++ b/hyperdb-api-derive/src/lib.rs @@ -115,7 +115,14 @@ pub fn table_derive(input: TokenStream) -> TokenStream { /// is emitted. /// /// Within a single file, struct-level derives always expand before -/// function-body macros, so ordering within a file is not a concern. +/// function-body macros, so ordering within a file is not a concern under +/// `cargo`. +/// +/// This ordering guarantee does **not** hold in rust-analyzer, whose +/// `proc-macro-srv` is long-lived and expands lazily, out of order, and from +/// cache — a `query_as!` can be re-expanded in a process where no derive has +/// run. Validation therefore skips when the registry is entirely empty, so the +/// editor does not report errors on code that compiles. #[proc_macro] pub fn query_as(input: TokenStream) -> TokenStream { match expand_query_as(&input.into()) { diff --git a/hyperdb-api/CHANGELOG.md b/hyperdb-api/CHANGELOG.md index 81b78a59..25d17a27 100644 --- a/hyperdb-api/CHANGELOG.md +++ b/hyperdb-api/CHANGELOG.md @@ -18,16 +18,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/). that holds `&self` — use the `*_unguarded` methods added below. Migration recipe in [docs/TRANSACTIONS.md](../docs/TRANSACTIONS.md#unguarded-transaction-control). -### Added +### Fixed -- `Connection::begin_transaction_unguarded`, `commit_unguarded` and - `rollback_unguarded`, plus the `AsyncConnection` equivalents. These are the - supported replacement for the removed deprecated methods and were previously - `pub(crate)` as `*_raw`. They are not deprecated, but they are not the - default path either: the caller owns pairing a begin with a commit or - rollback on **every** path, including panics and cancelled futures, since an - unmatched begin wedges the session in a way reconnect logic cannot clear. - Reach for them only when the guard's `&mut self` borrow is impossible. +- The `HYPERD_PATH is not set` error suggested + `cargo run -p hyperd-bootstrap -- download`, a package that has not existed + since the rename to `hyperdb-bootstrap`. The command failed with + "package(s) `hyperd-bootstrap` not found in workspace", which is the first + thing a new user saw when `HYPERD_PATH` was unset. ### Changed @@ -68,6 +65,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Added +- `Connection::begin_transaction_unguarded`, `commit_unguarded` and + `rollback_unguarded`, plus the `AsyncConnection` equivalents. These are the + supported replacement for the removed deprecated methods and were previously + `pub(crate)` as `*_raw`. They are not deprecated, but they are not the + default path either: the caller owns pairing a begin with a commit or + rollback on **every** path, including panics and cancelled futures, since an + unmatched begin wedges the session in a way reconnect logic cannot clear. + Reach for them only when the guard's `&mut self` borrow is impossible. + - `KvStore::set_if_absent` / `AsyncKvStore::set_if_absent` — guarded write that inserts only if the key is absent (no check-then-write race; single `INSERT ... WHERE NOT EXISTS`). Returns `true` if written, `false` if the key already existed (nothing written). - `KvStore::set_batch_if_absent` / `AsyncKvStore::set_batch_if_absent` — atomic batch variant of `set_if_absent`, returning `BatchGuardOutcome { written, skipped }`. All keys are validated before the transaction opens; an invalid key aborts the whole batch. - `KvStore::byte_size` / `AsyncKvStore::byte_size` — returns the total byte length of all values in the store (`SUM(OCTET_LENGTH(value))`); 0 for an empty store. diff --git a/hyperdb-api/README.md b/hyperdb-api/README.md index 7de3b9e0..1c3a1084 100644 --- a/hyperdb-api/README.md +++ b/hyperdb-api/README.md @@ -3,7 +3,10 @@ A **pure-Rust** implementation of the Hyper database API. Create, read, and manipulate Hyper database files (`.hyper`) without any C library dependencies. -- 22-24M rows/sec inserts, 18M rows/sec queries (100M row benchmark) +- Single-connection throughput on a 100M row benchmark: 68.9M rows/sec inserts + with the async `AsyncArrowInserter`, 25.0M rows/sec with the sync `Inserter`, + 31.1M rows/sec full-scan queries (see + [docs/BENCHMARK_GUIDE.md](../docs/BENCHMARK_GUIDE.md)) - Streaming by default — constant memory for billion-row results - Both sync (`Connection`) and async (`AsyncConnection`) APIs - Built-in string-native key-value store (`KvStore` / `AsyncKvStore`) @@ -279,7 +282,10 @@ from positional `row.get(0)` to streaming `stream_as`. ### Inserter (COPY Protocol) -The high-performance `Inserter` uses HyperBinary COPY protocol for 22M+ rows/sec: +The high-performance `Inserter` uses HyperBinary COPY protocol for roughly 25M +rows/sec on a single connection. For the fastest insert path, use the async +`AsyncArrowInserter` instead — see +[docs/BENCHMARK_GUIDE.md](../docs/BENCHMARK_GUIDE.md). ```rust let mut inserter = Inserter::new(&conn, &table_def)?; diff --git a/hyperdb-api/src/process.rs b/hyperdb-api/src/process.rs index c55ea680..a3c1cb88 100644 --- a/hyperdb-api/src/process.rs +++ b/hyperdb-api/src/process.rs @@ -256,7 +256,7 @@ impl HyperProcess { /// directory containing it. /// /// If `HYPERD_PATH` is unset, returns an error instructing the caller - /// to either set it or run the `hyperd-bootstrap` downloader to + /// to either set it or run the `hyperdb-bootstrap` downloader to /// install a pinned release at `.hyperd/current/hyperd`. fn find_hyperd() -> Result { #[cfg(windows)] @@ -266,7 +266,7 @@ impl HyperProcess { let Ok(path_str) = std::env::var("HYPERD_PATH") else { // Walk up from CWD looking for .hyperd/current/ written by - // `hyperd-bootstrap download`. This lets `node examples/foo.mjs` + // `hyperdb-bootstrap download`. This lets `node examples/foo.mjs` // run from any subdirectory of the repo without exporting HYPERD_PATH. if let Ok(cwd) = std::env::current_dir() { let mut dir = cwd.as_path(); @@ -283,7 +283,7 @@ impl HyperProcess { } return Err(Error::config( "HYPERD_PATH is not set. Point it at a hyperd executable, \ - or run `make download-hyperd` (or `cargo run -p hyperd-bootstrap -- download`) \ + or run `make download-hyperd` (or `cargo run -p hyperdb-bootstrap -- download`) \ to install a pinned release at `.hyperd/current/hyperd`.", )); }; @@ -664,10 +664,8 @@ impl HyperProcess { #[cfg(windows)] { // Check if it's a Named Pipe endpoint - if pipe_name.is_some() { - if let Some(ref pname) = pipe_name { - return ConnectionEndpoint::named_pipe(".", pname); - } + if let Some(ref pname) = pipe_name { + return ConnectionEndpoint::named_pipe(".", pname); } } // TCP endpoint (host:port format) @@ -967,6 +965,7 @@ impl HyperProcess { /// /// Returns `None` if the process is using TCP. #[cfg(windows)] + #[must_use] pub fn pipe_name(&self) -> Option<&str> { self.pipe_name.as_deref() } diff --git a/hyperdb-bootstrap/CHANGELOG.md b/hyperdb-bootstrap/CHANGELOG.md index d39bb26a..61cb6d64 100644 --- a/hyperdb-bootstrap/CHANGELOG.md +++ b/hyperdb-bootstrap/CHANGELOG.md @@ -43,8 +43,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/). thermally on a laptop. Verified native arm64 and all 1485 workspace tests pass; the macOS-14 CI runner is green (the deadlock is gone). +- **Bump the pinned `hyperd` release to `0.0.26479` (`r96880f6a`).** Updates the + version, build id, and all four per-platform sha256s in `hyperd-version.toml`. + **Verified native arm64** — `file` reports `Mach-O 64-bit executable arm64` + for the `macos-arm64` bundle's `lib/hyper/hyperd`, so the reason this crate + pulls the Java bundle rather than the C++ one still holds. All four platform + URLs return HTTP 200 at the new pin, and all **1586** workspace tests pass + against the new engine. Performance (interleaved same-session A/B vs the + previous `0.0.26359` pin, medians of 5 runs per engine at 100M rows, + single-connection): the async Arrow insert path **more than doubles** + (`AsyncArrowInserter` **+127%**, 30.4 → 68.9 M rows/s), reproduced as **+75%** + at 10M rows. Everything else is flat — sync `Inserter` +0.1%, `ChunkSender` + +1.4%, and all four query paths within ±0.7%. The async insert gain survives + its own wide (±25–35%) run-to-run spread because the old and new sample + ranges do not overlap at either scale. Multi-connection (`× 4`) deltas are + **not** reported: on this 14-core laptop they throttle thermally and spread + 17–41% run to run, enough to have read **+23% at 100M and −20% at 10M for the + same workload in the same session**. + ### Fixed +- The "no hyperd installed" error suggested `hyperd-bootstrap download`, but + the binary is `hyperdb-bootstrap` — the suggested command did not exist. + Doc comments across the crate carried the same pre-rename name, as did the + `hyperdb-bootstrap/hyperd-version.toml` path in `release.rs`. + - **Download `hyperd` from the Java API bundle instead of the C++ bundle.** Tableau's C++ `macos-arm64` zip ships an **x86_64** `hyperd` (an upstream packaging defect), so on Apple Silicon the extracted `hyperd` only ran diff --git a/hyperdb-bootstrap/hyperd-version.toml b/hyperdb-bootstrap/hyperd-version.toml index 2b0f0de1..b6191cb9 100644 --- a/hyperdb-bootstrap/hyperd-version.toml +++ b/hyperdb-bootstrap/hyperd-version.toml @@ -1,4 +1,4 @@ -# Pinned Hyper **Java** API release used by hyperd-bootstrap. +# Pinned Hyper **Java** API release used by hyperdb-bootstrap. # # We pull `hyperd` from the Java binding's bundle, NOT the C++ one: the # C++ macos-arm64 zip ships an x86_64 `hyperd` (upstream packaging defect), @@ -8,14 +8,14 @@ # # Bump these values (and sha256s) when upgrading. Contributors without # an override get this exact release, so reproducibility depends on it. -version = "0.0.26359" -build_id = "r07abb490" +version = "0.0.26479" +build_id = "r96880f6a" # sha256 of each platform's .zip. Omit a platform to skip verification # for that platform (not recommended). Compute with: # shasum -a 256 tableauhyperapi-java--release-main...zip [sha256] -"macos-arm64" = "434a5e7f95d914a7b328ac333b872fa044fabc6e23ad2aac9d0c9b762032b5ab" -"macos-x86_64" = "cbf01253f3b2b4288085ecb01a6415219338573fb8b5e1f101c457e2a26d5672" -"linux-x86_64" = "40e488c01ddc1ecaa53123a88fcf4150161a5828e1c22b475dd73e1cd9cbbbed" -"windows-x86_64" = "8546e67501ed3f15e97c9a0ed6a0dfc56fa9f070878bac435a7e5f7d1bfb6c99" +"macos-arm64" = "65bd021b3d3470ac74728ec287866a3ee0dfd806daf2589670feb3580955ee95" +"macos-x86_64" = "6690669c8a6a6c7c6794c101beb31b83e1589d76cb14b0c817523069591f694c" +"linux-x86_64" = "c20be5b6874d319c7db01dcec763d7e65ae2483c0becc75f2914a58accc4f932" +"windows-x86_64" = "3a400508e79c67ce9dcd8bd317ab164a61b5aed8ca880031ed4bbe6621514e1d" diff --git a/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs b/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs index b61ba8f5..101c3fae 100644 --- a/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs +++ b/hyperdb-bootstrap/src/bin/hyperdb-bootstrap.rs @@ -1,7 +1,7 @@ // Copyright (c) 2026, Salesforce, Inc. All rights reserved. // SPDX-License-Identifier: Apache-2.0 OR MIT -//! `hyperd-bootstrap` — CLI front-end for the library of the same name. +//! `hyperdb-bootstrap` — CLI front-end for the library of the same name. //! //! Subcommands: //! - `download` — install `hyperd` under `.hyperd//` and refresh @@ -166,7 +166,7 @@ fn run_which(args: WhichArgs) -> Result<()> { Ok(()) } else { anyhow::bail!( - "no hyperd installed at {} (run `hyperd-bootstrap download` first)", + "no hyperd installed at {} (run `hyperdb-bootstrap download` first)", binary.display() ); } diff --git a/hyperdb-bootstrap/src/error.rs b/hyperdb-bootstrap/src/error.rs index d8a5055b..407c3d83 100644 --- a/hyperdb-bootstrap/src/error.rs +++ b/hyperdb-bootstrap/src/error.rs @@ -1,7 +1,7 @@ // Copyright (c) 2026, Salesforce, Inc. All rights reserved. // SPDX-License-Identifier: Apache-2.0 OR MIT -//! Error types returned by the `hyperd-bootstrap` crate. +//! Error types returned by the `hyperdb-bootstrap` crate. use thiserror::Error; diff --git a/hyperdb-bootstrap/src/lib.rs b/hyperdb-bootstrap/src/lib.rs index e4b04387..24364501 100644 --- a/hyperdb-bootstrap/src/lib.rs +++ b/hyperdb-bootstrap/src/lib.rs @@ -6,7 +6,7 @@ //! because the C++ `macos-arm64` zip ships an x86_64 `hyperd`; see the //! `url` module for the full rationale.) //! -//! The crate ships both a CLI binary (`hyperd-bootstrap`) and a small +//! The crate ships both a CLI binary (`hyperdb-bootstrap`) and a small //! library. The library is blocking (no async runtime required) and has //! no dependency on `tokio`, so it can be called from build scripts, //! `postinstall` hooks, or any sync Rust code. diff --git a/hyperdb-bootstrap/src/release.rs b/hyperdb-bootstrap/src/release.rs index 8f0a3422..e528cba5 100644 --- a/hyperdb-bootstrap/src/release.rs +++ b/hyperdb-bootstrap/src/release.rs @@ -20,7 +20,7 @@ const BUILTIN_TOML: &str = include_str!("../hyperd-version.toml"); /// optional per-platform SHA-256 checksums. /// /// The "built-in" pin shipped with the crate lives in -/// `hyperd-bootstrap/hyperd-version.toml` and is available via +/// `hyperdb-bootstrap/hyperd-version.toml` and is available via /// [`PinnedRelease::builtin`]. Callers can override it by loading an /// external TOML file (see [`PinnedRelease::from_toml_file`]) or by passing /// a literal TOML string to [`PinnedRelease::from_toml_str`]. diff --git a/hyperdb-bootstrap/tests/integration.rs b/hyperdb-bootstrap/tests/integration.rs index 8d5e8fef..2ad4eabd 100644 --- a/hyperdb-bootstrap/tests/integration.rs +++ b/hyperdb-bootstrap/tests/integration.rs @@ -1,7 +1,7 @@ // Copyright (c) 2026, Salesforce, Inc. All rights reserved. // SPDX-License-Identifier: Apache-2.0 OR MIT -//! Integration tests for hyperd-bootstrap URL and install flows. +//! Integration tests for hyperdb-bootstrap URL and install flows. use hyperdb_bootstrap::{ InstallOptions, PinnedRelease, Platform, VersionSource, install, url::build_download_url, diff --git a/hyperdb-compile-check/CHANGELOG.md b/hyperdb-compile-check/CHANGELOG.md new file mode 100644 index 00000000..5c2a1368 --- /dev/null +++ b/hyperdb-compile-check/CHANGELOG.md @@ -0,0 +1,37 @@ +# Changelog + +All notable changes to the `hyperdb-compile-check` crate will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/), +and this project adheres to [Semantic Versioning](https://semver.org/). + +This crate is the compile-time SQL validation engine behind +`hyperdb-api-derive`'s `compile-time` feature. It is published because Cargo +requires it — `hyperdb-api-derive` depends on it — but it is not a public API +surface, and consumers should depend on `hyperdb-api-derive` instead. + +## [Unreleased] + +### Fixed + +- **Validation no longer reports false "not registered" errors when nothing has + been registered in the current process.** Registration is a process-global + side effect of expanding `derive(Table)`. Under `cargo` that ordering is + guaranteed — rustc expands a crate's macros in one host process, and + struct-level derives expand before function-body macros. rust-analyzer's + `proc-macro-srv` is long-lived and expands lazily, out of order, and from + cache, so a `query_as!` could be re-expanded in a process where no derive had + run. The result was a red squiggle in the editor on code that `cargo check` + compiled cleanly. + + An empty registry is now treated as "no information available" and validation + is skipped. Genuine diagnostics are unaffected: once anything is registered, + a lookup miss is still a real miss. + +### Changed + +- **BREAKING:** the minimum supported Rust version is now **1.88**, up from + 1.81, and the crate is compiled with **edition 2024**. 1.88 is the version + Red Hat Enterprise Linux 9.7 ships as `rust-toolset`. +- The `arrow` dependency moved from **58** to **59**, in lockstep with + `hyperdb-api`. diff --git a/hyperdb-compile-check/Cargo.toml b/hyperdb-compile-check/Cargo.toml index aa8c98af..2c7f9675 100644 --- a/hyperdb-compile-check/Cargo.toml +++ b/hyperdb-compile-check/Cargo.toml @@ -33,6 +33,17 @@ tempfile = "3.20" [dev-dependencies] tempfile = "3.20" +# Mirrors the root `[workspace.lints]` tables. This crate declares its own +# `[workspace]` so it can build standalone, which means it *cannot* inherit +# them with `lints.workspace = true` — the duplication is forced. +# +# Keep the two in sync by hand. They drifted once already: the root promoted +# `missing_errors_doc` and `missing_panics_doc` to `deny` while this copy stayed +# at `warn`, and nothing caught it because `cargo clippy --workspace` skips this +# crate. Lint it explicitly: +# +# cargo clippy --manifest-path hyperdb-compile-check/Cargo.toml \ +# --all-targets -- -D warnings [lints.rust] unexpected_cfgs = { level = "warn", check-cfg = ['cfg(kani)'] } unsafe_op_in_unsafe_fn = "deny" @@ -72,6 +83,6 @@ unreadable_literal = "allow" items_after_statements = "allow" match_same_arms = "allow" match_wildcard_for_single_variants = "allow" -missing_errors_doc = "warn" -missing_panics_doc = "warn" +missing_errors_doc = "deny" +missing_panics_doc = "deny" multiple_crate_versions = { level = "allow", priority = 1 } diff --git a/hyperdb-compile-check/src/registry.rs b/hyperdb-compile-check/src/registry.rs index ce9c17c6..10b6940d 100644 --- a/hyperdb-compile-check/src/registry.rs +++ b/hyperdb-compile-check/src/registry.rs @@ -96,6 +96,21 @@ pub fn registered_names() -> Vec { registry().lock().keys().cloned().collect() } +/// Returns true if nothing has been registered in this process. +/// +/// This distinguishes "the type is not registered" from "no `derive(Table)` +/// has run here yet", which are very different situations. Under `cargo` the +/// second cannot happen for a crate that has any registered struct: rustc +/// expands a crate's macros in one host process, and struct-level derives +/// expand before function-body macros. Under rust-analyzer it happens +/// routinely, because `proc-macro-srv` is long-lived and expands lazily, +/// out of order, and from cache — so a `query_as!` can be re-expanded in a +/// process where no derive ever ran. Callers treat an empty registry as +/// "cannot validate" and skip, rather than reporting a false error in the IDE. +pub fn is_empty() -> bool { + registry().lock().is_empty() +} + /// The public `Registry` type — a thin newtype that provides the seeding /// interface against a live `CompileTimeDb`. Created from a lock guard by /// `validate_query_as`. diff --git a/hyperdb-compile-check/src/validate.rs b/hyperdb-compile-check/src/validate.rs index fa87cf06..51360ce2 100644 --- a/hyperdb-compile-check/src/validate.rs +++ b/hyperdb-compile-check/src/validate.rs @@ -43,19 +43,29 @@ use crate::registry::{self, Registry}; /// the result schema is missing columns the struct requires. pub fn validate_query_as(struct_name: &str, sql: &str) -> Result<(), ValidationError> { // Step 1: look up by struct ident (not SQL table name — they differ). - let (_table_name, entry) = registry::get_by_struct(struct_name).ok_or_else(|| { - ValidationError::StructNotRegistered { - struct_name: struct_name.to_owned(), + let Some((_table_name, entry)) = registry::get_by_struct(struct_name) else { + // An empty registry means no `derive(Table)` ran in this process, so we + // have no basis to claim the struct is unregistered. See + // `registry::is_empty`. + if registry::is_empty() { + return Ok(()); } - })?; + return Err(ValidationError::StructNotRegistered { + struct_name: struct_name.to_owned(), + }); + }; let mut db = get_or_init().lock(); // Step 2+4: dry-run with bounded seed-and-retry on 42P01. - let schema = run_dry_run_with_seed(sql, &mut db)?; + let outcome = run_dry_run_with_seed(sql, &mut db)?; drop(db); + let Some(schema) = outcome else { + return Ok(()); + }; + // Step 3: name-subset diff. finish_name_check(struct_name, &entry.fields, &schema) } @@ -76,10 +86,14 @@ pub fn validate_query_as(struct_name: &str, sql: &str) -> Result<(), ValidationE pub fn validate_scalar_sql(sql: &str) -> Result<(), ValidationError> { let mut db = get_or_init().lock(); - let schema = run_dry_run_with_seed(sql, &mut db)?; + let outcome = run_dry_run_with_seed(sql, &mut db)?; drop(db); + let Some(schema) = outcome else { + return Ok(()); + }; + let col_count = schema.column_count(); if col_count != 1 { return Err(ValidationError::HyperError { @@ -101,21 +115,28 @@ pub fn validate_scalar_sql(sql: &str) -> Result<(), ValidationError> { /// only the first missing table would be seeded per call. /// /// Stops early on syntax errors, missing-column errors, or unregistered tables. +/// +/// Returns `Ok(None)` when validation cannot be performed because nothing is +/// registered in this process — see [`registry::is_empty`]. Callers skip +/// validation in that case instead of reporting a false diagnostic. fn run_dry_run_with_seed( sql: &str, db: &mut crate::db::CompileTimeDb, -) -> Result { +) -> Result, ValidationError> { // Bound to prevent infinite loops on pathological SQL (e.g., a self-join // that repeatedly 42P01s on the same unregistered table after seeding). const MAX_SEED_ROUNDS: usize = 8; for _ in 0..MAX_SEED_ROUNDS { match dry_run(db, sql) { - Ok(schema) => return Ok(schema), + Ok(schema) => return Ok(Some(schema)), Err(e) => match classify(&e) { ErrorClass::MissingTable(t) => match Registry::seed_if_known(&t, db) { Ok(true) => {} // seeded successfully; loop iterates to retry the dry-run Ok(false) => { + if registry::is_empty() { + return Ok(None); + } return Err(ValidationError::TablesNotRegistered { tables: vec![t] }); } Err(seed_err) => { @@ -188,6 +209,12 @@ mod tests { #[test] fn struct_not_registered_error() { + // Register first, deliberately. The diagnostic only fires when the + // registry is non-empty — an empty one means no `derive(Table)` ran in + // this process, which is not evidence that `Ghost` is unregistered. + // Without this the test would depend on which other test happened to + // register first, since tests share a process and run in parallel. + setup_users(); let err = validate_query_as("Ghost", "SELECT 1").unwrap_err(); assert!( matches!(err, ValidationError::StructNotRegistered { .. }), @@ -195,6 +222,14 @@ mod tests { ); } + // The empty-registry skip is deliberately not unit-tested here. Every test + // in this binary shares one process-global registry and they run in + // parallel, so any such test races: another test can register between the + // emptiness check and the call under test. `registry::is_empty` documents + // the behaviour, and the case it exists for — rust-analyzer expanding a + // `query_as!` in a process where no derive ran — is observable directly in + // the editor rather than from a test. + #[test] #[ignore = "requires HYPERD_PATH; run manually"] fn valid_query_passes() { diff --git a/hyperdb-mcp/README.md b/hyperdb-mcp/README.md index e1e29472..535086c8 100644 --- a/hyperdb-mcp/README.md +++ b/hyperdb-mcp/README.md @@ -4,7 +4,7 @@ An MCP (Model Context Protocol) server that turns the Hyper columnar database into an instant SQL analytics engine. Data flows in from other MCP plugins or files, lands in Hyper automatically, and becomes queryable with SQL — no setup, no schema files, no database management. -Built on the pure-Rust [`hyperdb-api`](../hyperdb-api/) crate for maximum performance: 22M+ rows/sec inserts, 18M+ rows/sec queries, constant memory for billion-row results. +Built on the pure-Rust [`hyperdb-api`](../hyperdb-api/) crate for maximum performance. On a single connection that crate benchmarks at 68.9M rows/sec inserts with the async `AsyncArrowInserter`, 25.0M rows/sec with the sync `Inserter`, and 31.1M rows/sec full-scan queries, with constant memory for billion-row results — see [docs/BENCHMARK_GUIDE.md](../docs/BENCHMARK_GUIDE.md). --- diff --git a/hyperdb-mcp/src/daemon/discovery.rs b/hyperdb-mcp/src/daemon/discovery.rs index 82d6bce9..04c7d707 100644 --- a/hyperdb-mcp/src/daemon/discovery.rs +++ b/hyperdb-mcp/src/daemon/discovery.rs @@ -468,7 +468,7 @@ mod tests { use tempfile::TempDir; use crate::daemon::health::{DaemonState, HealthListener}; - use crate::diagnostics::{PathEncoding, ReportedPath}; + use crate::diagnostics::ReportedPath; use super::*; @@ -757,6 +757,8 @@ mod tests { { use std::os::unix::ffi::OsStringExt; + use crate::diagnostics::PathEncoding; + let non_utf8_path = tmp .path() .join(OsString::from_vec(b"missing-\xff.json".to_vec())); diff --git a/hyperdb-mcp/src/diagnostics.rs b/hyperdb-mcp/src/diagnostics.rs index e126a132..464917f2 100644 --- a/hyperdb-mcp/src/diagnostics.rs +++ b/hyperdb-mcp/src/diagnostics.rs @@ -990,8 +990,12 @@ fn resolve_doctor_hyperd() -> DoctorHyperdResolution { fn resolve_configured_hyperd( configured: &Path, - _configured_text: &str, + configured_text: &str, ) -> (Option, Option) { + // Only the Windows `.exe` fallback below reads the raw text form. + #[cfg(not(windows))] + let _ = configured_text; // silence the unused-variable lint off Windows + #[cfg(windows)] const HYPERD_EXE: &str = "hyperd.exe"; #[cfg(not(windows))] @@ -1024,7 +1028,7 @@ fn resolve_configured_hyperd( } #[cfg(windows)] { - let executable = PathBuf::from(format!("{_configured_text}.exe")); + let executable = PathBuf::from(format!("{configured_text}.exe")); if executable.exists() { return (Some(executable), None); } diff --git a/hyperdb-mcp/tests/doctor_tests.rs b/hyperdb-mcp/tests/doctor_tests.rs index 7471c20b..9ce15f76 100644 --- a/hyperdb-mcp/tests/doctor_tests.rs +++ b/hyperdb-mcp/tests/doctor_tests.rs @@ -847,7 +847,7 @@ fn non_utf8_overlong_path(root: &Path) -> OsString { let mut wide: Vec = root.as_os_str().encode_wide().collect(); wide.extend([u16::from(b'\\'), 0xD800, u16::from(b'-')]); - wide.extend(std::iter::repeat(u16::from(b'x')).take(5 * 1024)); + wide.extend(std::iter::repeat_n(u16::from(b'x'), 5 * 1024)); OsString::from_wide(&wide) }