Hyper API for Rust

A pure-Rust implementation of the Hyper database API, using the PostgreSQL wire protocol with Hyper-specific extensions. Create, read, and manipulate Hyper database files (.hyper) without any C library dependencies.

Project Status — 1.0.0, AI-Assisted

This crate is AI-assisted but human-directed: much of the code was written by AI coding assistants under close review, with the design, architecture, and engineering trade-offs decided by an experienced developer.

As of 1.0.0 the public API is stable and follows semantic versioning: breaking changes require a major release, so the frequent churn of the 0.x line is behind us.

Contributors and reviewers should, at a minimum, run an AI code reviewer over any changes, following the conventions, layering rules, and patterns captured in AGENTS.md (and the subdirectory hyperdb-api-node/AGENTS.md). Those files are the authoritative guidance for AI assistants working in this repository.

Requirements

• Rust 1.88 or newer. 1.88 is the MSRV (minimum supported Rust version) and a floor, not a pin — newer toolchains are fine, and the workspace itself is built on edition 2024.
• RHEL 9.7 and binary-compatible distributions are supported using Red Hat's system-native rust-toolset, with no rustup required. This is why the MSRV is 1.88: it is the version RHEL 9.7 documents. The AppStream module is a rolling stream and is currently ahead of that floor, and CI verifies the workspace against it — see Enterprise Compatibility for the package list and caveats.

Also required at build time: protoc (for the gRPC bindings) and the hyperd executable, obtained with make download-hyperd. See Platform Support for the full OS and architecture matrix.

Key Features

• Pure Rust — no C library dependencies, standard cargo build
• High Performance — on a single connection: 68.9M rows/sec inserts with the async AsyncArrowInserter, 25.0M rows/sec with the sync Inserter, and 31.1M rows/sec full-scan queries (100M row benchmark, Apple M3 Max — see benchmarks for multi-connection and per-platform figures)
• Memory Safe — streaming by default, constant memory for billion-row results
• Dual Architecture — sync (Connection) and async (AsyncConnection) APIs
• Typed Row Mapping — #[derive(FromRow)] structs, including streaming stream_as for constant-memory typed queries
• Compile-time SQL Validation — opt-in query_as! macro checks SQL against your schema at build time (red squigglies in VS Code)
• Connection Pooling — async pooling via deadpool for high-concurrency applications
• Key-Value Store — string-native KvStore / AsyncKvStore backed by a single fixed table
• Arrow Integration — insert and read data in Arrow IPC stream format
• gRPC Transport — read-only access with Arrow IPC and load balancing support
• Full Type Support — all Hyper types including Numeric, Geography, Intervals
• Salesforce Auth — OAuth 2.0 and JWT Bearer Token flows for Data Cloud
• TLS — via rustls (always-on, pure Rust)
• Formal Verification — Kani proof harnesses for model-checked correctness

Quick Start

Build from Source

Install Rust via rustup.rs, install protoc for your platform, download the hyperd executable with make download-hyperd (bundled helper — see hyperdb-bootstrap), then build:

Platform	Install protoc	Build
macOS	brew install protobuf	make build
Linux (Debian/Ubuntu)	sudo apt-get install -y protobuf-compiler build-essential	make build
Linux (Fedora/RHEL)	sudo dnf install protobuf-compiler	make build
Windows	choco install protoc (also install VS Build Tools with the "Desktop development with C++" workload for the MSVC linker)	.\build.ps1 build

# Linux / macOS
make download-hyperd  # downloads hyperd into .hyperd/current/ (first time only)
make build            # or `make build-release` for optimized builds
make test             # runs unit + integration tests, test-release for release build
make doc              # builds the Hyper Rust documentation

# Windows (PowerShell)
.\build.ps1 download-hyperd
.\build.ps1 build     # or `.\build.ps1 build-release`
.\build.ps1 test      # or test-release for release build
.\build.ps1 doc       # builds the Hyper Rust documentation

The Makefile and build.ps1 wrappers auto-discover the downloaded hyperd at .hyperd/current/hyperd, and — if nothing is found on disk — auto-run download-hyperd the first time you invoke a target that actually needs hyperd (build, test, examples, doc). So make test from a clean checkout Just Works; subsequent runs are cache hits. If you already have a hyperd elsewhere, set HYPERD_PATH=/path/to/hyperd and the downloader stays inert — nothing is fetched, and no build step touches the network. Plain cargo build / cargo test also work as long as one of those is true.

See DEVELOPMENT.md for the full build guide including WSL, cross-compilation, benchmarks, and per-platform troubleshooting.

Installation

Add to your Cargo.toml:

[dependencies]
hyperdb-api = { path = "hyperdb-api" }

Installing the CLIs

hyperdb-mcp and hyperdb-bootstrap ship two ways:

Via npm (recommended for hyperdb-mcp; bundles a matching hyperd):

npm install -g hyperdb-mcp

Supported platforms: macOS ARM64 (Apple Silicon), Linux x64 (glibc), Windows x64. Intel macOS is built-from-source only at the moment — see the platform table in hyperdb-mcp/README.md.

Via crates.io (compiles from source; no bundled hyperd):

cargo install hyperdb-mcp
cargo install hyperdb-bootstrap

hyperdb-bootstrap will then download a compatible hyperd for you:

hyperdb-bootstrap download

Environment Setup

The hyperd executable (Hyper database server) must be available. The simplest path is:

make download-hyperd   # or `.\build.ps1 download-hyperd` on Windows

This installs hyperd under .hyperd/current/ in the repo and is auto-discovered by the Makefile / build.ps1. If you already have a hyperd elsewhere, export HYPERD_PATH instead:

export HYPERD_PATH=/path/to/hyperd

Sync Example

use hyperdb_api::{
    Catalog, Connection, CreateMode, HyperProcess, Inserter,
    Result, SqlType, TableDefinition,
};

fn main() -> Result<()> {
    let hyper = HyperProcess::new(None, None)?;
    let conn = Connection::new(&hyper, "example.hyper", CreateMode::CreateIfNotExists)?;

    // Create a table
    let table_def = TableDefinition::from("users")
        .add_required_column("id", SqlType::int())
        .add_required_column("name", SqlType::text());
    Catalog::new(&conn).create_table(&table_def)?;

    // Insert data (sync Inserter, COPY protocol, ~25M rows/sec)
    {
        let mut inserter = Inserter::new(&conn, &table_def)?;
        inserter.add_row(&[&1i32, &"Alice"])?;
        inserter.add_row(&[&2i32, &"Bob"])?;
        inserter.execute()?;
    }

    // Query data
    let result = conn.execute_query("SELECT * FROM users")?;
    for row in result.rows() {
        let row = row?;
        let id: Option<i32> = row.get(0);
        let name: Option<String> = row.get(1);
        println!("{:?} - {:?}", id, name);
    }

    Ok(())
}

Async Example

use hyperdb_api::{AsyncConnection, CreateMode, HyperProcess, Result};

#[tokio::main]
async fn main() -> Result<()> {
    let hyper = HyperProcess::new(None, None)?;
    let endpoint = hyper.require_endpoint()?;

    let conn = AsyncConnection::connect(
        endpoint,
        "example_async.hyper",
        CreateMode::CreateIfNotExists,
    ).await?;

    conn.execute_command("CREATE TABLE users (id INT, name TEXT)").await?;
    conn.execute_command("INSERT INTO users VALUES (1, 'Alice')").await?;
    conn.close().await?;
    Ok(())
}

Crate Overview

Crate	Purpose	Published
hyperdb-api	High-level API — connections, inserters, catalog, Arrow, pooling	crates.io
hyperdb-api-core	Internal implementation details (types, protocol, client). Not a public API — depend on hyperdb-api instead.	crates.io
hyperdb-api-salesforce	Salesforce Data Cloud OAuth authentication	crates.io
hyperdb-mcp	MCP server for LLM-driven SQL analytics on .hyper files	crates.io
sea-query-hyperdb	HyperDB dialect backend for sea-query	crates.io
hyperdb-api-node	Node.js/TypeScript bindings via napi-rs	npm
hyperdb-bootstrap	Download the hyperd executable from Tableau's release packages	crates.io

Examples

The API ships 14 examples in hyperdb-api/examples/ plus 2 companion crate examples.

Core Examples

Example	Description
insert_data_into_single_table	Create a table and insert data using Inserter
insert_data_into_multiple_tables	Multiple related tables
create_hyper_file_from_csv	Load CSV data into a Hyper table
delete_data_in_existing_hyper_file	Delete data with SQL DELETE
update_data_in_existing_hyper_file	Update data with SQL UPDATE
read_and_print_data_from_existing_hyper_file	Read table definitions and query data
insert_data_with_expressions	Column mappings with MappedInserter
insert_geospatial_data_to_a_hyper_file	Insert geospatial data

Rust-Specific Examples

Example	Description
arrow	Read/write Arrow RecordBatch data
async_usage	AsyncConnection and Tokio patterns
threaded_inserter	Multi-threaded bulk insertion with InsertChunk/ChunkSender
grpc_query	gRPC transport, Arrow IPC results
connection_pool	Async connection pooling with deadpool
transactions	RAII guards, multi-table rollback, DDL, reconnect semantics

Running Examples

export HYPERD_PATH=/path/to/hyperd

# Run individual examples
cargo run -p hyperdb-api --example insert_data_into_single_table
cargo run -p hyperdb-api --example arrow
cargo run -p hyperdb-api --example connection_pool

# Companion crate examples
cargo run -p sea-query-hyperdb --example basic_usage
cargo run -p hyperdb-api-salesforce --example salesforce_auth_example

# Run all examples
./run_all_examples.sh

Companion Crates

sea-query-hyperdb

HyperDB dialect backend for sea-query — use for window functions, CTEs, complex JOINs, and type-safe query composition:

[dependencies]
sea-query = "0.32"
sea-query-hyperdb = { path = "sea-query-hyperdb" }

use sea_query::{Query, Expr, Iden};
use sea_query_hyperdb::HyperQueryBuilder;

let sql = Query::select()
    .column(Users::Name)
    .from(Users::Table)
    .and_where(Expr::col(Users::Age).gt(18))
    .to_string(HyperQueryBuilder);

let result = conn.fetch_all(&sql)?;

hyperdb-api-salesforce

Salesforce Data Cloud OAuth authentication — JWT Bearer Token, Username-Password, and Refresh Token flows:

[dependencies]
hyperdb-api-salesforce = { path = "hyperdb-api-salesforce" }

use hyperdb_api_salesforce::{SalesforceAuthConfig, AuthMode, SharedTokenProvider};

let auth_config = SalesforceAuthConfig::new(
    "https://login.salesforce.com",
    "your-connected-app-consumer-key",
)?
.auth_mode(AuthMode::private_key("user@example.com", &private_key_pem)?);

let token_provider = SharedTokenProvider::new(auth_config)?;

See hyperdb-api-salesforce/README.md for full setup guide.

Node.js Bindings

The hyperdb-api-node package provides Node.js and TypeScript bindings built with napi-rs:

const { HyperProcess, Connection, CreateMode } = require('hyperdb-api-node');

const hyper = new HyperProcess();
const conn = await Connection.connect(hyper.endpoint, 'my.hyper', CreateMode.CreateAndReplace);

// Tagged template literals — SQL injection safe
const rows = await conn.sql`SELECT * FROM users WHERE age > ${18}`;
await conn.close();
hyper.close();

See hyperdb-api-node/README.md for full documentation.

Platform Support

Platform	Status	Build Tool
Linux (x86_64)	Supported	make build
macOS (ARM & x64)	Supported	make build
Windows	Supported	.\build.ps1 build
WSL	Supported	make build

MSRV: Rust 1.88 (see rust-version in Cargo.toml), chosen to match the rust-toolset version Red Hat Enterprise Linux 9.7 ships. The workspace uses edition 2024.

Enterprise Compatibility

This workspace builds with Red Hat's system-native Rust toolchain and no rustup, which is how enterprise environments typically consume it. RHEL provides rust-toolset in AppStream as a rolling Application Stream:

dnf install -y rust-toolset gcc gcc-c++ fontconfig-devel unzip
cargo build --release

Notes for system-toolchain builds, all verified against ubi9/ubi:

• protoc is required and is not packaged for UBI. hyperdb-api-core generates its gRPC bindings at build time via tonic-prost-build. dnf search protobuf offers only protobuf-c and python3-protobuf, and ubi-9-codeready-builder-rpms is already enabled by default, so install protoc from the upstream release and put it on PATH.
• gcc, gcc-c++ and fontconfig-devel are needed for chart rendering only. They come from hyperdb-mcp's plotters dependency, whose font stack compiles C and C++. Nothing in this workspace's own code requires a C or C++ compiler.
• .cargo/config.toml is a developer convenience, not a build requirement. It selects clang plus the mold linker on x86_64-unknown-linux-gnu for faster local linking; mold is not packaged for UBI. Either remove it or neutralize it with RUSTFLAGS= and CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=cc. Note that --config target.<triple>.rustflags=[] does not work, because Cargo joins rustflags across configuration sources rather than replacing them.
• rust-toolchain.toml can be ignored. It is read only by rustup's proxy shims, so a distro-packaged cargo never consults it.
• RHEL's rust-toolset is a rolling stream and is currently ahead of the 1.88.0 documented in the RHEL 9.7 release notes. The MSRV floor of 1.88 is therefore conservative and safe.

Compatibility is enforced by .github/workflows/rhel-compatibility.yml, which runs cargo check --workspace --locked --all-targets in a ubi9/ubi container using nothing but the distro toolchain. Reproduce it locally with make check-rhel.

Documentation

Resource	Description
hyperdb-api/README.md	Full user guide for the hyperdb-api crate
docs/WHATS_NEW_0.4.md	Highlights of the 0.4.0 release
docs/ROW_MAPPING.md	The five ways to map result rows into Rust values
hyperdb-api-derive/README.md	#[derive(FromRow)], #[derive(Table)], and compile-time SQL validation
DEVELOPMENT.md	Architecture, building, testing, benchmarks — for contributors
CONTRIBUTING.md	How to contribute
docs/GITHUB_OPERATIONS.md	CI/release workflows and how maintainers cut a release
docs/TRANSACTIONS.md	Transaction API design
docs/BENCHMARK_GUIDE.md	How to run benchmarks

Per-crate documentation: each crate has its own README.md (see Crate Overview).

Generate API docs locally:

make doc    # or: cargo doc --no-deps --open

Contributing

See CONTRIBUTING.md for the governance model, contribution checklist, commit message format, and pull request process.

Acknowledgments

This project includes code adapted from sfackler/rust-postgres (the postgres-protocol, tokio-postgres, and postgres-types crates by Steven Fackler, MIT or Apache-2.0). See NOTICE for the full third-party attribution list and the upstream license text.

License

Licensed under either of MIT or Apache-2.0 at your option.