diff --git a/.markdownlint-cli2.jsonc b/.markdownlint-cli2.jsonc index 3b48ce3..8b44128 100644 --- a/.markdownlint-cli2.jsonc +++ b/.markdownlint-cli2.jsonc @@ -24,6 +24,12 @@ // reference, so hand-edits are undone by the next refresh. "hyperdb-mcp/scripts/dc_sql_reference.md", + // Root changelog only. release-please generates it -- it is the sole + // "changelog-path" in release-please-config.json -- so any hand-fix is + // clobbered on the next release. Deliberately NOT "**/CHANGELOG.md": the + // nine per-crate changelogs are hand-maintained and stay in scope. + "CHANGELOG.md", + // npm platform sub-packages: packaging boilerplate, largely generated "hyperdb-api-node/npm/**", "hyperdb-mcp/npm/**" diff --git a/.markdownlintignore b/.markdownlintignore index f5b42c6..78aaf14 100644 --- a/.markdownlintignore +++ b/.markdownlintignore @@ -13,6 +13,14 @@ test_results/ # so hand-editing it is undone by the next refresh. hyperdb-mcp/scripts/dc_sql_reference.md +# Root changelog only -- release-please generates it (the sole changelog-path +# in release-please-config.json), so hand-fixes are clobbered on the next +# release. The leading slash is load-bearing: an unanchored "CHANGELOG.md" +# would also hide the nine hand-maintained per-crate changelogs, which are in +# scope. (markdownlint-cli2 matches its own list as globs, not gitignore +# patterns, so the equivalent entry there needs no slash.) +/CHANGELOG.md + # npm platform sub-packages: packaging boilerplate, largely generated hyperdb-api-node/npm/ hyperdb-mcp/npm/ diff --git a/AGENTS.md b/AGENTS.md index 2d816a2..b6ec3c9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,7 +43,7 @@ This is a **pure-Rust implementation** of the Hyper database API, using the Post The codebase uses a **layered architecture**. The flagship user-facing crate is `hyperdb-api`; its implementation details live in `hyperdb-api-core`, which preserves three internal submodules (`types`, `protocol`, `client`) that contributors navigate independently. Two optional companion crates extend the public surface. -``` +```text ┌─────────────────────────────────────────────────────┐ │ hyperdb-api (High-level API, public) │ │ - Connection, AsyncConnection, HyperProcess │ @@ -251,7 +251,7 @@ Three *other* crates do define features. `hyperdb-api` is flag-free; the workspa Tests are organized by crate: -``` +```text hyperdb-api/tests/ # Integration tests (high-level API) hyperdb-api/tests/common/ # Shared test utilities hyperdb-api-core/tests/ # Client-level integration tests @@ -519,7 +519,7 @@ All commit messages **must** follow the format `(): ` — upstream/main:` and re-lint — rather than assuming, or you will "fix" things that were never broken and miss the ones you introduced. - Three traps that have actually bitten: + Four traps that have actually bitten: - **Duplicate `### Fixed` / `### Added` siblings under one `## [Unreleased]`** (MD024). Changelogs here often already have the section further down. Merge @@ -533,6 +533,11 @@ All commit messages **must** follow the format `(): ` — new block. This corrupted 176 fences across 22 files once. Any such pass must track fence state; prefer `markdownlint-cli2 --fix`, which is safe, and note that it cannot fix MD040 because choosing a language needs judgement. +- **A nested `.markdownlint.json` replaces the root config rather than merging + with it**, so a new one must `extends` the root or every rule there reverts to + default. Dropping that line from `docs/superpowers/.markdownlint.json` turns 5 + findings into 92 — 70 of them the MD060 disabled just below. It hides well: it + makes the *linter* wrong, not the document. Beware format-on-save: a Markdown formatter reformatting tables to satisfy MD060 once stripped the README's badge links (`[![CI](img)](target)` became diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ef03d45..6bd481a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,13 +2,13 @@ This page lists the operational governance model of this project, as well as the recommendations and requirements for how to best contribute to Tableau Hyper Rust API. We strive to obey these as best as possible. As always, thanks for contributing – we hope these guidelines make it easier and shed some light on our approach and processes. -# Governance Model +## Governance Model -## Community Based +### Community Based The intent and goal of open sourcing this project is to increase the contributor and user base. The governance model is one where new project leads (`admins`) will be added to the project based on their contributions and efforts, a so-called "do-acracy" or "meritocracy" similar to that used by all Apache Software Foundation projects. -# Issues, requests & ideas +## Issues, requests & ideas Use GitHub Issues page to submit issues, enhancement requests and discuss ideas. @@ -40,7 +40,7 @@ Use GitHub Issues page to submit issues, enhancement requests and discuss ideas. If you're new to our project and looking for some way to make your first contribution, look for Issues labelled `good first contribution`. -# Code Style & Guidelines +## Code Style & Guidelines This project follows the **[Microsoft Pragmatic Rust Guidelines](https://microsoft.github.io/rust-guidelines/)**. The repo-specific adaptation — what is machine-enforced, what is reviewer-enforced, and the list of documented exceptions — is in [docs/RUST_GUIDELINES.md](docs/RUST_GUIDELINES.md). @@ -71,7 +71,7 @@ path-filtered, so a docs-only PR does not run it. When a lint genuinely cannot be satisfied for a given site, suppress it with `#[expect(lint_name, reason = "")]` rather than bare `#[allow(...)]` — the `reason` is mandatory and `#[expect]` auto-removes itself when the lint would no longer fire. See the [Exceptions](docs/RUST_GUIDELINES.md#exceptions) section of the guidelines page for the current workspace-level waivers. -# Contribution Checklist +## Contribution Checklist - [ ] Clean, simple, well styled code — conforms to [docs/RUST_GUIDELINES.md](docs/RUST_GUIDELINES.md) - [ ] Commits should be atomic and messages must be descriptive. Related issues should be mentioned by Issue number. @@ -92,7 +92,7 @@ When a lint genuinely cannot be satisfied for a given site, suppress it with `#[ - [ ] Reviews - Changes must be approved via peer code review -# Signed Commits +## Signed Commits This repo requires signed commits on `main`. Any PR whose commits are unsigned will be blocked at merge time — the GitHub Actions CI runs fine on unsigned commits, but the merge button won't enable. @@ -129,7 +129,7 @@ Two gotchas to avoid: GPG signing is also supported — see [GitHub's signing-commits guide](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) for the GPG and S/MIME paths. SSH is the recommended default for this repo. -# Creating a Pull Request +## Creating a Pull Request 1. **Ensure the bug/feature was not already reported** by searching on GitHub under Issues. If none exists, create a new issue so that other contributors can keep track of what you are trying to add/fix and offer suggestions (or let you know if there is already an effort in progress). 2. **Fork** the repository on GitHub. @@ -142,20 +142,20 @@ GPG signing is also supported — see [GitHub's signing-commits guide](https://d > **NOTE**: Be sure to [sync your fork](https://help.github.com/articles/syncing-a-fork/) before making a pull request. -# Contributor License Agreement ("CLA") +## Contributor License Agreement ("CLA") In order to accept your pull request, we need you to submit a CLA. You only need to do this once to work on any of Salesforce's open source projects. Complete your CLA here: -# Commit Message Format +## Commit Message Format This project uses [Conventional Commits](https://www.conventionalcommits.org/) to automate versioning and release management. Please format your commit messages accordingly. -## Commit Message Structure +### Commit Message Structure -``` +```text (): @@ -169,7 +169,7 @@ This project uses [Conventional Commits](https://www.conventionalcommits.org/) t - **Body** (optional): Detailed explanation of the change - **Footer** (optional): Issue references -## Commit Types and Version Impact +### Commit Types and Version Impact | Commit Type | Version Bump | Example | |------------|--------------|---------| @@ -182,9 +182,9 @@ This project uses [Conventional Commits](https://www.conventionalcommits.org/) t > `fix:` for changes that end-users of the crate or npm package would notice. > A `fix(ci):` commit will trigger an unintended patch release. -## Examples +### Examples -``` +```text feat: add support for batch query execution fix(hyperdb-api-core): resolve type mismatch @@ -196,13 +196,13 @@ ci: fix chmod step in npm-build-publish workflow chore: update arrow dependency to 56 ``` -# Release Process +## Release Process This repo uses [release-please](https://github.com/googleapis/release-please) to fully automate version bumps, changelog generation, tagging, and the crates.io / npm publish dance. -## What contributors do +### What contributors do **Use [Conventional Commits](https://www.conventionalcommits.org/) for every PR title.** That's it. release-please reads the merged commits to figure out @@ -219,7 +219,7 @@ The **per-crate** `CHANGELOG.md` files are different: each carries a expected to append to them for user-visible API changes. See [AGENTS.md](AGENTS.md) reminder 8 for the policy and the full file list. -## What maintainers do +### What maintainers do The end-to-end flow lives in [`docs/GITHUB_OPERATIONS.md` → Cutting a release](docs/GITHUB_OPERATIONS.md#cutting-a-release). @@ -250,7 +250,7 @@ For pre-releases (`-rc.N`, `-alpha.N`, `-beta.N`), include a `Release-As:` footer in a commit on `main` — see [`docs/GITHUB_OPERATIONS.md`](docs/GITHUB_OPERATIONS.md#pre-releases). -## Published Crates +### Published Crates | Package | Registry | Notes | |---------|----------|-------| @@ -264,10 +264,10 @@ footer in a commit on `main` — see | `hyperdb-compile-check` | crates.io | Compile-time SQL validation backend. Not a workspace member (it declares its own `[workspace]` to break the dependency cycle), but release-please manages its version and it must be published for `hyperdb-api-derive`'s off-by-default `compile-time` feature to resolve. | | `hyperdb-api-node` | npm | Node.js/TypeScript bindings. `publish = false` for crates.io — the only crate in the tree that is not a Cargo publish target. | -# Code of Conduct +## Code of Conduct Please follow our [Code of Conduct](CODE_OF_CONDUCT.md). -# License +## License By contributing your code, you agree to license your contribution under the terms of our project [MIT](LICENSE-MIT) and [Apache-2.0](LICENSE-APACHE) dual license, and to sign the [Salesforce CLA](https://cla.salesforce.com/sign-cla). diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index f1c5938..898c7c5 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -103,7 +103,7 @@ The API uses **lifetime annotations** to provide compile-time guarantees that re are used correctly. All dependent types (`Inserter`, `Catalog`, `Rowset`) carry a `'conn` lifetime tying them to their parent `Connection`: -``` +```text Connection (owns data) ├── Inserter<'conn> │ └── CopyInWriter<'conn> @@ -565,7 +565,7 @@ cargo test -p hyperdb-api --test integration_test ### Test Structure -``` +```text hyperdb-api/tests/ # Integration tests (high-level API) hyperdb-api/tests/common/ # Shared test utilities hyperdb-api-core/tests/ # Client-level integration tests diff --git a/README.md b/README.md index 097324a..65d714a 100644 --- a/README.md +++ b/README.md @@ -364,8 +364,10 @@ This workspace builds with Red Hat's **system-native Rust toolchain and no `rustup`**, which is how enterprise environments typically consume it. RHEL provides `rust-toolset` in AppStream as a rolling Application Stream: - dnf install -y rust-toolset gcc gcc-c++ fontconfig-devel unzip - cargo build --release +```bash +dnf install -y rust-toolset gcc gcc-c++ fontconfig-devel unzip +cargo build --release +``` Notes for system-toolchain builds, all verified against `ubi9/ubi`: diff --git a/SECURITY.md b/SECURITY.md index b69c021..ca9d103 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,4 +1,4 @@ -## Security +# Security Please report any security issue to [https://www.sfdc.co/SubmitVuln](https://www.sfdc.co/SubmitVuln) as soon as it is discovered. This library limits its runtime dependencies in diff --git a/docs/BENCHMARK_GUIDE.md b/docs/BENCHMARK_GUIDE.md index e1d0b5b..27193f0 100644 --- a/docs/BENCHMARK_GUIDE.md +++ b/docs/BENCHMARK_GUIDE.md @@ -17,7 +17,7 @@ specific questions. All benchmarks share the same schema so numbers compare directly: -``` +```sql measurements(id INT NOT NULL, sensor_id INT, value DOUBLE, timestamp BIGINT) -- 24 bytes / row ``` @@ -145,7 +145,7 @@ block from the suite's stdout. ### Platform: macOS (Apple Silicon) -**Hardware / software** +#### Hardware / software - **OS:** Darwin 26.6.2 (aarch64) - **CPU:** Apple M3 Max (14 physical / 14 logical cores) @@ -301,7 +301,9 @@ unchanged — opt into `ArrowInserter` and `executeQueryColumnar` / ### Platform: Linux (x86_64) -**Hardware / software** *(placeholder — replace with `host` block from your suite run)* +#### Hardware / software + +*Placeholder — replace with the `host` block from your suite run.* - **OS:** (e.g. Ubuntu 24.04) - **CPU:** @@ -326,7 +328,7 @@ unchanged — opt into `ArrowInserter` and `executeQueryColumnar` / ### Platform: Windows (x86_64, native) -**Hardware / software** +#### Hardware / software - **OS:** Windows 11 (build 26100) (x86_64) - **CPU:** Intel(R) Core(TM) i9-10980XE @ 3.00 GHz (18 physical / 36 logical cores) @@ -417,7 +419,9 @@ same process. ### Platform: Windows (x86_64 / WSL2) -**Hardware / software** *(placeholder)* +#### Hardware / software + +*Placeholder — fill in from the `host` block of your suite run.* - **OS:** (e.g. Ubuntu 22.04 under WSL2) - **CPU:** diff --git a/docs/GITHUB_OPERATIONS.md b/docs/GITHUB_OPERATIONS.md index 87ad463..a79b0ad 100644 --- a/docs/GITHUB_OPERATIONS.md +++ b/docs/GITHUB_OPERATIONS.md @@ -51,7 +51,7 @@ the GitHub Release after merging the release PR) or via manual `workflow_dispatch` with an explicit tag input (for re-runs or emergency releases). Structure: -``` +```text verify ← full test suite + hyperd URL check, single-platform │ └─► publish ← crates.io publish in dependency order @@ -104,7 +104,7 @@ without needing Rust toolchains or manual hyperd setup. **Structure:** -``` +```text verify-ci ← checks that CI passed for this commit (gh api commit status) │ └─► build-npm (matrix × 4 platforms) @@ -340,7 +340,7 @@ semver: `feat!:` will bump `1.2.3` → `2.0.0` as expected. To stabilize the API and cut `1.0.0`, add a `Release-As: 1.0.0` footer to a conventional-commit on `main`: -``` +```text feat: stabilize public API Release-As: 1.0.0 @@ -351,7 +351,7 @@ Release-As: 1.0.0 For an `-rc.N` / `-alpha.N` / `-beta.N` release, add a footer to a commit on `main`: -``` +```text Release-As: 0.2.0-rc.1 ``` diff --git a/docs/ROW_MAPPING.md b/docs/ROW_MAPPING.md index 88289c5..88c0d7a 100644 --- a/docs/ROW_MAPPING.md +++ b/docs/ROW_MAPPING.md @@ -8,7 +8,7 @@ struct mapping. Start with the simplest that fits your situation. All six forms are demonstrated end-to-end in one runnable example: -``` +```bash cargo run -p hyperdb-api --example row_mapping_forms ``` diff --git a/docs/TRANSACTIONS.md b/docs/TRANSACTIONS.md index 57bf2b3..f108a12 100644 --- a/docs/TRANSACTIONS.md +++ b/docs/TRANSACTIONS.md @@ -28,7 +28,13 @@ txn.commit()?; #### 1. Exclusive Borrowing -`Connection::transaction(&mut self)` takes a mutable (exclusive) borrow of the connection, and `Transaction<'conn>` holds `&'conn mut Connection`. While the `Transaction` exists, the Rust borrow checker prevents any other code from accessing the raw connection — not even for read-only operations. This eliminates an entire class of bugs where application code accidentally issues SQL statements outside the transaction scope, causing data races or logic errors. The protection is enforced at compile time with zero runtime cost. +`Connection::transaction(&mut self)` takes a mutable (exclusive) borrow of the +connection, and `Transaction<'conn>` holds `&'conn mut Connection`. While the +`Transaction` exists, the Rust borrow checker prevents any other code from +accessing the raw connection — not even for read-only operations. This +eliminates an entire class of bugs where application code accidentally issues +SQL statements outside the transaction scope, causing data races or logic +errors. The protection is enforced at compile time with zero runtime cost. ```rust let mut conn = Connection::connect(endpoint, "db.hyper", CreateMode::DoNotCreate)?; diff --git a/docs/superpowers/.markdownlint.json b/docs/superpowers/.markdownlint.json new file mode 100644 index 0000000..a4fdbe4 --- /dev/null +++ b/docs/superpowers/.markdownlint.json @@ -0,0 +1,33 @@ +// Directory-scoped rules for docs/superpowers/ (specs and plans). +// +// These are point-in-time planning artifacts: a spec and plan are written +// before a feature, reviewed alongside the change that implements it, and then +// left alone (AGENTS.md, "Documentation Conventions"). Reflowing prose or +// re-indenting code blocks in a settled record is churn no reader benefits +// from, and it is the riskiest place to do it -- these are the longest files in +// the repo. So the two rules that would force a rewrite are off here: +// +// MD013 line-length -- plan prose runs to 1500+ characters per line +// MD046 code-block-style -- plans mix indented and fenced blocks +// +// Every other rule still applies, deliberately. New specs and plans are +// written all the time, and MD040 (untagged fences) plus the heading rules are +// the feedback worth having while authoring one. This is a relaxation, not an +// exclusion: an exclusion would leave the editor silent on a directory that is +// actively written to. +// +// "extends" is load-bearing, not decoration. A nested config REPLACES ancestor +// configuration rather than merging with it -- verified empirically: without +// this line, the root's `MD024.siblings_only` reverts to the default and fires +// on plan files, and `MD060: false` would likewise be lost, re-enabling the +// rule that was disabled for mangling links. Extending the root config keeps +// all of it and overrides only the two rules below. +// +// Governs both tools from one file: markdownlint-cli2 resolves config per +// directory, and the VS Code extension walks up from the file being edited +// (".markdownlint.{jsonc,json,...} file in the same or parent directory"). +{ + "extends": "../../.markdownlint.json", + "MD013": false, + "MD046": false +} diff --git a/docs/superpowers/plans/2026-07-09-kv-store-m2-mcp.md b/docs/superpowers/plans/2026-07-09-kv-store-m2-mcp.md index bbbd874..de5091b 100644 --- a/docs/superpowers/plans/2026-07-09-kv-store-m2-mcp.md +++ b/docs/superpowers/plans/2026-07-09-kv-store-m2-mcp.md @@ -782,7 +782,7 @@ Expected: FAIL — `README missing mention of kv_get` (README not yet updated). - [ ] **Step 3: Add the `### Key-value store` subsection** under `## Tool index` in `readme.rs`: -``` +```markdown ### Key-value store (scratchpad) - `kv_set` — save a variable/state/summary/JSON under a store + key (upsert). @@ -862,7 +862,7 @@ Expected: PASS, all cases. Real output. `hyperdb-mcp/CHANGELOG.md` under `## [Unreleased]` → `### Added`: -``` +```markdown - `kv_get`, `kv_set`, `kv_delete`, `kv_list`, `kv_list_stores`, `kv_size`, `kv_pop`, `kv_clear` tools — a key-value scratchpad backed by the `hyperdb-api` KV store, routable to any database via the standard `database`/`persist` parameters. - `hyper://schema/kv` resource describing the KV table schema, durability rule, and LEFT JOIN enrichment pattern. ``` diff --git a/docs/superpowers/plans/2026-07-11-kv-mcp-llm-ergonomics.md b/docs/superpowers/plans/2026-07-11-kv-mcp-llm-ergonomics.md index f170e97..2e81acc 100644 --- a/docs/superpowers/plans/2026-07-11-kv-mcp-llm-ergonomics.md +++ b/docs/superpowers/plans/2026-07-11-kv-mcp-llm-ergonomics.md @@ -1245,7 +1245,7 @@ Replace `kv_set` (`server.rs:3123-3139`). Resolve the value from exactly one of Update the `#[tool(description = ...)]` on `kv_set` (`server.rs:3120-3121`) to document the new behavior. Append to the existing description: -``` +```text Returns {stored, created, value_bytes}; `created:false` means an existing value was overwritten. Pass overwrite=false to avoid clobbering (skips + returns stored:false, existed:true). Pass value_path= to store a file's contents server-side instead of `value` (exactly one of value/value_path; reads any server-readable path — no sandbox). ``` @@ -1346,7 +1346,7 @@ Replace the `kv_size` handler body in `hyperdb-mcp/src/server.rs:3189-3204` (the Update the `#[tool(description = ...)]` on `kv_size` (around `server.rs:3186-3187`) to document the `bytes` field. Extend the description to say: -``` +```text Returns {store, size, bytes} where `size` is the key count and `bytes` is the total `OCTET_LENGTH` of all values (0 for empty stores). ``` @@ -1729,7 +1729,7 @@ Replace the `kv_list` handler (`server.rs:3169-3184`) to accept `KvListParams` a Locate the `#[tool(description = ...)]` attribute on `kv_list` (a few lines above `:3169`). Append to the existing description: -``` +```text Pass values=true to return full (key, value) pairs as an `entries` array instead of just keys — useful for reading a whole store without N×kv_get. ``` diff --git a/hyperdb-api-core/docs/DEVELOPMENT-client.md b/hyperdb-api-core/docs/DEVELOPMENT-client.md index 1974fef..747eb52 100644 --- a/hyperdb-api-core/docs/DEVELOPMENT-client.md +++ b/hyperdb-api-core/docs/DEVELOPMENT-client.md @@ -18,7 +18,7 @@ For user-facing documentation, see [README.md](README.md). ### Module Map -``` +```text src/ lib.rs # Crate root, re-exports config.rs # Config builder for TCP connections @@ -129,7 +129,7 @@ in-flight streaming queries. machine that handles all three transfer modes. The state transitions and per-mode paths are documented in that module's rustdoc. Key states: -``` +```text ReadInitialResults -> RequestStatus -> ReadStatus -> RequestResults -> ReadResults -> Finished ``` diff --git a/hyperdb-api-core/docs/DEVELOPMENT-protocol.md b/hyperdb-api-core/docs/DEVELOPMENT-protocol.md index 7de9a90..197722c 100644 --- a/hyperdb-api-core/docs/DEVELOPMENT-protocol.md +++ b/hyperdb-api-core/docs/DEVELOPMENT-protocol.md @@ -29,7 +29,7 @@ encoding, use LittleEndian for the value bytes. The inline doc comments in ### Module Layout -``` +```text src/ lib.rs # Crate root, re-exports message/ diff --git a/hyperdb-api-core/docs/DEVELOPMENT-types.md b/hyperdb-api-core/docs/DEVELOPMENT-types.md index fd0e921..b234abf 100644 --- a/hyperdb-api-core/docs/DEVELOPMENT-types.md +++ b/hyperdb-api-core/docs/DEVELOPMENT-types.md @@ -23,7 +23,7 @@ For user-facing documentation (type mapping, serialization traits, API surface), ### Data Flow -``` +```text User Rust value │ ▼ diff --git a/hyperdb-api-derive/README.md b/hyperdb-api-derive/README.md index b4cbd85..c623e5c 100644 --- a/hyperdb-api-derive/README.md +++ b/hyperdb-api-derive/README.md @@ -151,12 +151,12 @@ With `features = ["compile-time"]` and `HYPERD_PATH` set, `query_as!` validates Bad SQL produces a `compile_error!` pointing at the SQL string literal: -``` +```text error: column "emai1" does not exist on any table in the query; check for a typo or a renamed/dropped column ``` -``` +```text error: `User` requires column "email" but the query does not project it; add it to the SELECT list or remove the field from `User` ``` diff --git a/hyperdb-api-node/AGENTS.md b/hyperdb-api-node/AGENTS.md index 864b82e..bcd041b 100644 --- a/hyperdb-api-node/AGENTS.md +++ b/hyperdb-api-node/AGENTS.md @@ -10,7 +10,7 @@ For documentation conventions specific to JavaScript/TypeScript, see the [Docume `hyperdb-api-node` is a **napi-rs** native addon that exposes the Rust `hyperdb-api` crate to JavaScript and TypeScript. The Rust side (`src/*.rs`) compiles to a `.node` shared library; the JS side (`index.js`, `index.d.ts`, `pool.mjs`, `arrow.mjs`) adds ergonomic wrappers on top. -``` +```text ┌──────────────────────────────────────────────────┐ │ JS/TS application code │ ├──────────────────────────────────────────────────┤ diff --git a/hyperdb-api-node/DEVELOPMENT.md b/hyperdb-api-node/DEVELOPMENT.md index 183d722..1bc370e 100644 --- a/hyperdb-api-node/DEVELOPMENT.md +++ b/hyperdb-api-node/DEVELOPMENT.md @@ -9,7 +9,7 @@ see [README.md](README.md). `hyperdb-api-node` is a three-layer stack: pure-JS extensions on top of napi-rs bindings on top of the Rust `hyperdb-api` crate. -``` +```text ┌───────────────────────────────────────────────────────┐ │ JS extensions │ │ index.js — native loader + tagged templates, │ diff --git a/hyperdb-api-node/README.md b/hyperdb-api-node/README.md index 2de0bd4..a4a1075 100644 --- a/hyperdb-api-node/README.md +++ b/hyperdb-api-node/README.md @@ -19,7 +19,7 @@ This package provides a native Node.js addon that gives JavaScript and TypeScrip ## Class Overview -``` +```text HyperProcess ──creates──▶ Connection ──returns──▶ RowData ConnectionBuilder ─builds─▶ Connection ──returns──▶ QueryStream ──yields──▶ RowData ConnectionPool ──pools──▶ Connection ──returns──▶ ColumnarStream ──yields──▶ ColumnarChunk diff --git a/hyperdb-api-node/examples/hyper-explorer/README.md b/hyperdb-api-node/examples/hyper-explorer/README.md index b03c1db..bb442b7 100644 --- a/hyperdb-api-node/examples/hyper-explorer/README.md +++ b/hyperdb-api-node/examples/hyper-explorer/README.md @@ -264,7 +264,7 @@ sequenceDiagram ## Project structure -``` +```text hyper-explorer/ ├── server/ # Express backend (TypeScript, tsx) │ ├── index.ts # Express, CORS, JSON, HyperProcess, pool, PORT, shutdown diff --git a/hyperdb-api-salesforce/DEVELOPMENT.md b/hyperdb-api-salesforce/DEVELOPMENT.md index 4f33a89..dbcf723 100644 --- a/hyperdb-api-salesforce/DEVELOPMENT.md +++ b/hyperdb-api-salesforce/DEVELOPMENT.md @@ -8,7 +8,7 @@ Internal architecture and contributor notes for the `hyperdb-api-salesforce` cra Authentication with Salesforce Data Cloud uses a two-stage token exchange: -``` +```text Stage 1: App --> POST {login_url}/services/oauth2/token --> OAuth Access Token Stage 2: App --> POST {instance_url}/services/a360/token --> DC JWT ``` diff --git a/hyperdb-api/CHANGELOG.md b/hyperdb-api/CHANGELOG.md index 841e023..42b030f 100644 --- a/hyperdb-api/CHANGELOG.md +++ b/hyperdb-api/CHANGELOG.md @@ -111,7 +111,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/). unification applied it workspace-wide. See the `hyperdb-bootstrap` and `hyperdb-api-salesforce` entries for detail. -- **BREAKING:** `KvStore::set`, `KvStore::set_as`, and `KvStore::set_batch` (plus their `AsyncKvStore` twins) now return `SetOutcome` or `BatchSetOutcome` instead of `Result<()>`, reporting whether each write created a new key or overwrote an existing one. The `created` signal eliminates silent data loss when an LLM accidentally clobbers existing KV data. Callers that ignored the `Result` (statement-position `set("k","v")?;`) — including `let _ = set(...)?;` — still compile unchanged. The genuinely breaking cases are callers that named the unit return (`let x: () = set(...)?;`) or that returned `set(...)` where a `Result<()>` was expected; these now see `SetOutcome`/`BatchSetOutcome` and must adapt. +- **BREAKING:** `KvStore::set`, `KvStore::set_as`, and `KvStore::set_batch` + (plus their `AsyncKvStore` twins) now return `SetOutcome` or + `BatchSetOutcome` instead of `Result<()>`, reporting whether each write + created a new key or overwrote an existing one. The `created` signal + eliminates silent data loss when an LLM accidentally clobbers existing KV + data. Callers that ignored the `Result` (statement-position `set("k","v")?;`) + — including `let _ = set(...)?;` — still compile unchanged. The genuinely + breaking cases are callers that named the unit return (`let x: () = + set(...)?;`) or that returned `set(...)` where a `Result<()>` was expected; + these now see `SetOutcome`/`BatchSetOutcome` and must adapt. ### Added diff --git a/hyperdb-api/DEVELOPMENT.md b/hyperdb-api/DEVELOPMENT.md index 14d0c79..87bb973 100644 --- a/hyperdb-api/DEVELOPMENT.md +++ b/hyperdb-api/DEVELOPMENT.md @@ -198,7 +198,7 @@ This is a cross-crate change. Start at the bottom of the stack: ### Test Structure -``` +```text hyperdb-api/tests/ # Integration tests (require live hyperd) hyperdb-api/tests/common/ # Shared test utilities (TestConnection) hyperdb-api/src/result.rs # Unit tests (arrow_path_tests, no hyperd needed) diff --git a/hyperdb-api/tests/stress_test/README.md b/hyperdb-api/tests/stress_test/README.md index e6a6b8c..26510af 100644 --- a/hyperdb-api/tests/stress_test/README.md +++ b/hyperdb-api/tests/stress_test/README.md @@ -178,7 +178,7 @@ Results from actual runs on a Mac Studio (M2 Ultra, 192 GB RAM): ### Moderate Load (30s, 12 threads, seed=7777) -``` +```text Duration: 30.5s Operations: 25,185 (0 failures) Throughput: 827 ops/sec, 297,808 insert-rows/sec @@ -190,7 +190,7 @@ Disk used: 856 MB ### High Load (2 min, 18 threads, seed=9999) -``` +```text Config: 5 DBs, 8 inserters, 6 query, 4 mixed think_time=0–5ms, batch=1k–50k rows Duration: 121.0s @@ -204,7 +204,7 @@ Disk used: 2,012 MB ### Replay of High Load Run -``` +```text Duration: 120.8s Operations: 13,720 (0 failures) Throughput: 113.5 ops/sec, 191,867 insert-rows/sec @@ -229,7 +229,7 @@ Disk used: 1,979 MB ## File Structure -``` +```text hyperdb-api/tests/ ├── stress_test_main.rs # Test entry points (#[ignore]) └── stress_test/ diff --git a/hyperdb-mcp/CHANGELOG.md b/hyperdb-mcp/CHANGELOG.md index 1886b5a..4e122eb 100644 --- a/hyperdb-mcp/CHANGELOG.md +++ b/hyperdb-mcp/CHANGELOG.md @@ -7,37 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] -### Changed - -- The `arrow` and `parquet` dependencies moved from **58** to **59**. Not a - library-API change (this crate ships a binary), but it removes the `thrift` - dependency and the Apache Thrift excessive-size-allocation advisory with it: - `parquet` 58.x pinned `thrift ^0.17`, and `parquet` 59 dropped thrift - entirely. - -- `Engine::execute_in_transaction` now calls `hyperdb-api`'s `*_unguarded` - transaction methods instead of the deprecated `begin_transaction` / `commit` - / `rollback`, which 1.0.0 removed. No behavior change: the helper still takes - `&self`, so the RAII guard remains unavailable to it, and it still rolls back - before resuming an unwind. The `#[allow(deprecated)]` it needed is gone. - Moving to the guard still waits on - [issue #72](https://github.com/tableau/hyper-api-rust/issues/72). -- **BREAKING:** the minimum supported Rust version is now **1.88**, up from - 1.81, and the crate is compiled with **edition 2024**. 1.88 is the version - Red Hat Enterprise Linux 9.7 ships as `rust-toolset`. - -### Fixed - -- Public documentation on `PersistentAttachOutcome`, `ensure_exists_in`, - `list_in`, `upsert_stub_in`, `set_metadata_in` and `reconcile_in` no longer - links to private items, which made `cargo doc` fail under - `RUSTDOCFLAGS="-D warnings"`. The prose still names the internal helpers; it - just no longer tries to hyperlink to items a reader cannot navigate to. - ### Added -- **`kv_set_many` tool** — atomic batch write accepting an array of `{key, value}` entries. Validates all keys before opening the transaction; an invalid key aborts the whole batch without writing anything. Default behavior (`overwrite` absent or `true`) reports `{stored, created, overwritten, total_bytes}`; with `overwrite: false`, existing keys are skipped (not errors) and the response reports `{stored, created, skipped, total_bytes}` where `created` is the number of keys newly inserted. `total_bytes` sums all submitted values, so it is an upper bound on bytes actually persisted when keys are skipped or duplicated. Each oversized entry (> 1 MiB) adds a keyed `warning` to a `warnings` array. -- **`kv_set` — `value_path` parameter** — absolute path to a file whose contents become the value (read server-side). Provide exactly one of `value` or `value_path`; neither or both is `INVALID_ARGUMENT`. Reads any path the server process can read (same posture as `load_file` — no sandbox), with I/O errors preserved (`PermissionDenied` → `ErrorCode::PermissionDenied`, not collapsed to `FileNotFound`). A hard 64 MiB size cap is enforced against the file's metadata *before* reading, so a stray path to a huge file is rejected with `INVALID_ARGUMENT` instead of being slurped into memory. +- **`kv_set_many` tool** — atomic batch write accepting an array of + `{key, value}` entries. Validates all keys before opening the transaction; an + invalid key aborts the whole batch without writing anything. Default behavior + (`overwrite` absent or `true`) reports + `{stored, created, overwritten, total_bytes}`; with `overwrite: false`, + existing keys are skipped (not errors) and the response reports + `{stored, created, skipped, total_bytes}` where `created` is the number of + keys newly inserted. `total_bytes` sums all submitted values, so it is an + upper bound on bytes actually persisted when keys are skipped or duplicated. + Each oversized entry (> 1 MiB) adds a keyed `warning` to a `warnings` array. +- **`kv_set` — `value_path` parameter** — absolute path to a file whose + contents become the value (read server-side). Provide exactly one of `value` + or `value_path`; neither or both is `INVALID_ARGUMENT`. Reads any path the + server process can read (same posture as `load_file` — no sandbox), with I/O + errors preserved (`PermissionDenied` → `ErrorCode::PermissionDenied`, not + collapsed to `FileNotFound`). A hard 64 MiB size cap is enforced against the + file's metadata *before* reading, so a stray path to a huge file is rejected + with `INVALID_ARGUMENT` instead of being slurped into memory. - **`kv_set` — `overwrite` parameter** (default `true`) — when `false`, skips the write if the key already exists (calls `set_if_absent` instead of `set`), returning `{stored: false, created: false, existed: true}` with the original value unchanged. Eliminates silent data-loss from accidental overwrites. - **`kv_set` — `created` and `value_bytes` in response** — `created: true` means the key was newly inserted, `false` means an existing value was overwritten. `value_bytes` reports the UTF-8 byte length of the written value. - **`kv_set` — soft size warning** — values exceeding 1 MiB trigger a non-fatal `warning` field in the response steering the LLM toward `load_data` or a real table for large payloads. The write always succeeds. @@ -74,6 +63,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Changed +- The `arrow` and `parquet` dependencies moved from **58** to **59**. Not a + library-API change (this crate ships a binary), but it removes the `thrift` + dependency and the Apache Thrift excessive-size-allocation advisory with it: + `parquet` 58.x pinned `thrift ^0.17`, and `parquet` 59 dropped thrift + entirely. + +- `Engine::execute_in_transaction` now calls `hyperdb-api`'s `*_unguarded` + transaction methods instead of the deprecated `begin_transaction` / `commit` + / `rollback`, which 1.0.0 removed. No behavior change: the helper still takes + `&self`, so the RAII guard remains unavailable to it, and it still rolls back + before resuming an unwind. The `#[allow(deprecated)]` it needed is gone. + Moving to the guard still waits on + [issue #72](https://github.com/tableau/hyper-api-rust/issues/72). +- **BREAKING:** the minimum supported Rust version is now **1.88**, up from + 1.81, and the crate is compiled with **edition 2024**. 1.88 is the version + Red Hat Enterprise Linux 9.7 ships as `rust-toolset`. - **KV attachment/read-only clarification (supersedes the shorthand in the Added notes above).** The global `--read-only` guard leaves the four KV readers available, but every `kv_*` call targeting a user attachment still @@ -91,6 +96,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ### Fixed +- Public documentation on `PersistentAttachOutcome`, `ensure_exists_in`, + `list_in`, `upsert_stub_in`, `set_metadata_in` and `reconcile_in` no longer + links to private items, which made `cargo doc` fail under + `RUSTDOCFLAGS="-D warnings"`. The prose still names the internal helpers; it + just no longer tries to hyperlink to items a reader cannot navigate to. - **Hyper-format export side-effect correction (supersedes the older Unreleased note below).** Export does not mutate its source database, but it creates or replaces the requested destination `.hyper` file and materializes diff --git a/hyperdb-mcp/DEVELOPMENT.md b/hyperdb-mcp/DEVELOPMENT.md index 99a2c17..c4f79bb 100644 --- a/hyperdb-mcp/DEVELOPMENT.md +++ b/hyperdb-mcp/DEVELOPMENT.md @@ -101,7 +101,16 @@ Every ingest function (`ingest_json`, `ingest_csv`, `ingest_parquet_file`, `inge Three edges to this guarantee, all documented in `src/engine.rs`: 1. **DDL auto-commits.** Hyper commits `CREATE TABLE` / `DROP TABLE` immediately, regardless of the surrounding transaction. In `replace` mode the original table is already gone by the time INSERTs start, so a failed replace-mode ingest leaves an empty table rather than restoring the original. Append mode is fully atomic because it issues DDL only when the target doesn't exist and, when it does, no data is lost on failure. -2. **Panic safety.** `execute_in_transaction` wraps the closure in `catch_unwind(AssertUnwindSafe(...))`, issues a best-effort ROLLBACK on panic, and `resume_unwind`s the original payload. Without this, a panic inside the closure (unwrap on None, indexing OOB, arithmetic overflow) would leave an open transaction and every subsequent tool call would hit "transaction already in progress" — classified as `InternalError`, not `ConnectionLost`, so the reconnect path at `with_engine` would not rescue it and the engine would stay wedged until restart. Tested via `execute_in_transaction_rolls_back_on_panic` in `tests/transaction_tests.rs`. +2. **Panic safety.** `execute_in_transaction` wraps the closure in + `catch_unwind(AssertUnwindSafe(...))`, issues a best-effort ROLLBACK on + panic, and `resume_unwind`s the original payload. Without this, a panic + inside the closure (unwrap on None, indexing OOB, arithmetic overflow) would + leave an open transaction and every subsequent tool call would hit + "transaction already in progress" — classified as `InternalError`, not + `ConnectionLost`, so the reconnect path at `with_engine` would not rescue it + and the engine would stay wedged until restart. Tested via + `execute_in_transaction_rolls_back_on_panic` in + `tests/transaction_tests.rs`. 3. **Post-error wire-protocol quirk.** After a mid-transaction Hyper-level error (e.g. a NOT NULL violation on INSERT), the first SELECT after rollback may return an empty result set due to residual bytes on the connection. Retrying the query once restores normal behavior; the rollback itself is always correct. The `query_resilient` helper in `tests/transaction_tests.rs` is the robust pattern. --- @@ -217,15 +226,49 @@ Logs land next to the persistent file when one is supplied (so users find them i ## Daemon Mode Internals -`Engine::new` defaults to *daemon mode* — it tries `daemon::spawn::ensure_daemon(resolve_port_scan())` first, which discovers an existing daemon via `~/.hyperdb/daemon.json` (overridable via `HYPERDB_STATE_DIR`), else scans the port range for a running daemon, else auto-spawns one on the first free port as a detached background process. The Engine then connects via TCP (`Connection::connect(endpoint, …)`) without owning any `HyperProcess`, and records the daemon's `health_port` so the server's debounced `HEARTBEAT` targets the actual discovered port rather than re-resolving. +`Engine::new` defaults to *daemon mode* — it tries +`daemon::spawn::ensure_daemon(resolve_port_scan())` first, which discovers an +existing daemon via `~/.hyperdb/daemon.json` (overridable via +`HYPERDB_STATE_DIR`), else scans the port range for a running daemon, else +auto-spawns one on the first free port as a detached background process. The +Engine then connects via TCP (`Connection::connect(endpoint, …)`) without +owning any `HyperProcess`, and records the daemon's `health_port` so the +server's debounced `HEARTBEAT` targets the actual discovered port rather than +re-resolving. Falls back to local mode (per-session `hyperd` via `HyperProcess::new`) when the daemon can't be reached (including `AllOccupied` — the whole scan range is held by foreign processes), or always when `--no-daemon` is passed. -**Port resolution + identity.** `resolve_port_scan()` returns a `PortScan { base, span }`: when `HYPERDB_DAEMON_PORT` is set it pins that exact port (`span = 1`); otherwise it scans `span = DAEMON_PORT_SCAN_SPAN` (16) ports up from `DEFAULT_DAEMON_BASE_PORT` (7485 — deliberately *not* 7484, which is hyperd's conventional gRPC port). `probe_port` classifies each port as `OurDaemon` / `Camped` / `Refused`: liveness is no longer a bare TCP connect but an identity handshake — `health::ping_identified` sends `PING` and requires the reply's first two tokens to be exactly `PONG` and `hyperdb-mcp` (the third token is the daemon version). A foreign process that merely accepts TCP is `Camped` and skipped; only a `Refused` (connection-refused) port is treated as free to spawn on. `discover()` applies the same identity check before trusting `daemon.json`, so a stale or foreign-owned file is detected and removed. - -**Version takeover.** When discovery finds a running daemon, `maybe_take_over` compares the client's `version::MCP_VERSION` against the daemon's reported version via the pure `client_should_take_over` helper (`semver`). If the client is *strictly newer* it sends `STOP` (which drops the daemon's `HyperProcess`, stopping `hyperd`), waits for the health port to stop answering the identity ping, then respawns a fresh daemon on the same port. Equal/older/unparseable versions reuse the daemon — never a downgrade-kill. This makes upgrades take effect immediately instead of waiting for the old daemon to disappear. - -**Idle shutdown is opt-in.** `DaemonConfig.idle_timeout` is `Option`, set only when `--idle-timeout` or `HYPERDB_DAEMON_IDLE_TIMEOUT` is provided (flag wins over env). With neither set the idle-monitor branch of the `run_daemon` `tokio::select!` is replaced by `std::future::pending()` and never fires — the daemon (and `hyperd`) stay resident indefinitely so clients never pay the cold-start "restarting, please retry" round-trip. `DaemonState::last_activity` and the debounced `HEARTBEAT` plumbing still exist and only matter when the timeout is enabled. The hyperd restart-limit shutdown (below) is independent and always active. +**Port resolution + identity.** `resolve_port_scan()` returns a +`PortScan { base, span }`: when `HYPERDB_DAEMON_PORT` is set it pins that exact +port (`span = 1`); otherwise it scans `span = DAEMON_PORT_SCAN_SPAN` (16) ports +up from `DEFAULT_DAEMON_BASE_PORT` (7485 — deliberately *not* 7484, which is +hyperd's conventional gRPC port). `probe_port` classifies each port as +`OurDaemon` / `Camped` / `Refused`: liveness is no longer a bare TCP connect +but an identity handshake — `health::ping_identified` sends `PING` and requires +the reply's first two tokens to be exactly `PONG` and `hyperdb-mcp` (the third +token is the daemon version). A foreign process that merely accepts TCP is +`Camped` and skipped; only a `Refused` (connection-refused) port is treated as +free to spawn on. `discover()` applies the same identity check before trusting +`daemon.json`, so a stale or foreign-owned file is detected and removed. + +**Version takeover.** When discovery finds a running daemon, `maybe_take_over` +compares the client's `version::MCP_VERSION` against the daemon's reported +version via the pure `client_should_take_over` helper (`semver`). If the client +is *strictly newer* it sends `STOP` (which drops the daemon's `HyperProcess`, +stopping `hyperd`), waits for the health port to stop answering the identity +ping, then respawns a fresh daemon on the same port. Equal/older/unparseable +versions reuse the daemon — never a downgrade-kill. This makes upgrades take +effect immediately instead of waiting for the old daemon to disappear. + +**Idle shutdown is opt-in.** `DaemonConfig.idle_timeout` is `Option`, +set only when `--idle-timeout` or `HYPERDB_DAEMON_IDLE_TIMEOUT` is provided +(flag wins over env). With neither set the idle-monitor branch of the +`run_daemon` `tokio::select!` is replaced by `std::future::pending()` and never +fires — the daemon (and `hyperd`) stay resident indefinitely so clients never +pay the cold-start "restarting, please retry" round-trip. +`DaemonState::last_activity` and the debounced `HEARTBEAT` plumbing still exist +and only matter when the timeout is enabled. The hyperd restart-limit shutdown +(below) is independent and always active. ### hyperd liveness monitoring and restart @@ -252,7 +295,21 @@ Two new code paths fire `report_hyperd_error_to_daemon` (best-effort, 200ms time ### Known limitations -- **Hung-but-alive `hyperd`** (TCP listening, but unresponsive to queries) is NOT detected. The monitor's `try_wait()` returns `None` for a hung process; client tool calls hang on the read side without producing a `ConnectionLost` error. Operator recovery is `hyperdb-mcp daemon stop` followed by reconnect. Note the tradeoff introduced by resident-by-default: the idle timeout used to be an implicit backstop that reaped a wedged daemon after 30 min, after which the next client respawned a fresh one. With idle shutdown now opt-in (off by default), a hung-but-alive `hyperd` stays wedged until a client reports an error (fast-path `REPORT_HYPERD_ERROR`, which fires on a client-side `ConnectionLost`) or an operator runs `daemon stop`. This is an accepted tradeoff — keeping `hyperd` warm avoids the cold-start "restarting, please retry" round-trip on every active session, and genuine hyperd hangs are rare. A future enhancement could add a daemon-side liveness probe (a periodic trivial query with a timeout) to close the "all clients idle + hyperd hung" gap without reintroducing cold-start latency. +- **Hung-but-alive `hyperd`** (TCP listening, but unresponsive to queries) is + NOT detected. The monitor's `try_wait()` returns `None` for a hung process; + client tool calls hang on the read side without producing a `ConnectionLost` + error. Operator recovery is `hyperdb-mcp daemon stop` followed by reconnect. + Note the tradeoff introduced by resident-by-default: the idle timeout used to + be an implicit backstop that reaped a wedged daemon after 30 min, after which + the next client respawned a fresh one. With idle shutdown now opt-in (off by + default), a hung-but-alive `hyperd` stays wedged until a client reports an + error (fast-path `REPORT_HYPERD_ERROR`, which fires on a client-side + `ConnectionLost`) or an operator runs `daemon stop`. This is an accepted + tradeoff — keeping `hyperd` warm avoids the cold-start "restarting, please + retry" round-trip on every active session, and genuine hyperd hangs are rare. + A future enhancement could add a daemon-side liveness probe (a periodic + trivial query with a timeout) to close the "all clients idle + hyperd hung" + gap without reintroducing cold-start latency. - **Watchers** auto-recover from hyperd restarts: when an ingest fails with a connection-lost error, the watcher rebuilds its connection pool against the engine's current endpoint and retries the file once. Persistent failures (the second attempt also fails) fall through to the standard `failed/` move so a single broken file can't keep the watcher pinned in retry loops. See `src/daemon/{mod,discovery,health,run,spawn}.rs` for the full implementation. @@ -268,7 +325,14 @@ See `src/daemon/{mod,discovery,health,run,spawn}.rs` for the full implementation - **No `spawn_blocking`**: Engine calls are synchronous in an async runtime. Works today because `rmcp` handles dispatch, but explicit `spawn_blocking` would be more robust. - **Watcher thread per directory**: Each `watch_directory` call spawns a dedicated OS thread. For many concurrent watchers, a shared thread pool would be more efficient. - **Single shared `Connection`**: `Engine` owns exactly one `Connection` to `hyperd`, and every tool call serializes on an `Arc>>`. `hyperd` itself can handle many concurrent connections, so a `Connection` pool (or a dedicated watcher-thread `Connection`) would stop a long-running ingest from blocking unrelated tool calls. -- **`list_changed` is not fired on append-mode table creation**: Both `load_*` append mode and `watch_directory` auto-create the target table via `CREATE TABLE IF NOT EXISTS` when it doesn't exist, but neither path plumbs a "created vs existed" signal back from the ingest layer, so clients subscribed to `hyper://tables` miss the new entry until they rescan. The fix is to have `create_table` report whether it actually created vs. no-oped, bubble that up through the ingest return, and have both callers fire `notify_list_changed` only when a new table appears. +- **`list_changed` is not fired on append-mode table creation**: Both `load_*` + append mode and `watch_directory` auto-create the target table via + `CREATE TABLE IF NOT EXISTS` when it doesn't exist, but neither path plumbs a + "created vs existed" signal back from the ingest layer, so clients subscribed + to `hyper://tables` miss the new entry until they rescan. The fix is to have + `create_table` report whether it actually created vs. no-oped, bubble that up + through the ingest return, and have both callers fire `notify_list_changed` + only when a new table appears. --- diff --git a/hyperdb-mcp/README.md b/hyperdb-mcp/README.md index 535086c..fb394de 100644 --- a/hyperdb-mcp/README.md +++ b/hyperdb-mcp/README.md @@ -237,7 +237,14 @@ describe({ database: "persistent" }) sample({ table: "customers", database: "persistent" }) ``` -The `database` parameter is available on `query`, `execute`, `load_data`, `load_file`, `load_files`, `watch_directory`, `describe`, `sample`, `chart`, `export`, and `set_table_metadata`. The shorthand `persist: true` (sugar for `database: "persistent"`) is available on `load_data`, `load_file`, `load_files`, and `watch_directory`. Read tools generally accept a read-only user attachment; write tools require a writable one. The exception is the KV family: every `kv_*` call to a user attachment requires it to be writable because the backing table may need initialization. +The `database` parameter is available on `query`, `execute`, `load_data`, +`load_file`, `load_files`, `watch_directory`, `describe`, `sample`, `chart`, +`export`, and `set_table_metadata`. The shorthand `persist: true` (sugar for +`database: "persistent"`) is available on `load_data`, `load_file`, +`load_files`, and `watch_directory`. Read tools generally accept a read-only +user attachment; write tools require a writable one. The exception is the KV +family: every `kv_*` call to a user attachment requires it to be writable +because the backing table may need initialization. Every successful database-routed response includes the canonical `resolved_database`: `"local"`, `"persistent"`, or the lowercase attached @@ -332,7 +339,7 @@ If hyperd repeatedly fails to start (3 attempts within 60 seconds — e.g., misc Ingest inline data and run a SQL query in a single call. -``` +```text query_data(data: '[{"region":"West","revenue":1200},...]', sql: 'SELECT region, SUM(revenue) FROM data GROUP BY region') ``` @@ -348,7 +355,7 @@ query_data(data: '[{"region":"West","revenue":1200},...]', sql: 'SELECT region, Ingest a file and run a SQL query in a single call. Streams from disk — handles files of any size. -``` +```text query_file(path: '/tmp/sales.parquet', sql: 'SELECT TOP 10 * FROM sales ORDER BY amount DESC') ``` @@ -365,7 +372,7 @@ query_file(path: '/tmp/sales.parquet', sql: 'SELECT TOP 10 * FROM sales ORDER BY Load inline data into a named local, persistent, or attached-database table. -``` +```text load_data(table: 'customers', data: '[{"id":1,"name":"Alice"},...]') ``` @@ -381,7 +388,7 @@ load_data(table: 'customers', data: '[{"id":1,"name":"Alice"},...]') Load a file into a named local, persistent, or attached-database table. -``` +```text load_file(table: 'orders', path: '/tmp/orders.csv') ``` @@ -404,7 +411,7 @@ local table. Pass the absolute path to the Iceberg table root (the directory containing `metadata/` and `data/`); hyperd's native Iceberg reader derives the schema and resolves the snapshot. -``` +```text load_iceberg(table: 'sales', path: '/lake/warehouse/db/sales') ``` @@ -423,7 +430,7 @@ Iceberg table metadata. Run a **read-only** SQL query against local (default), persistent, or an attached database. Accepts `SELECT`, `WITH`, `EXPLAIN`, `SHOW`, `VALUES`. For DDL/DML use `execute`. -``` +```text query(sql: 'SELECT c.name, SUM(o.amount) FROM orders o JOIN customers c ON o.customer_id = c.id GROUP BY c.name') ``` @@ -431,7 +438,7 @@ query(sql: 'SELECT c.name, SUM(o.amount) FROM orders o JOIN customers c ON o.cus Execute one or more **mutating** SQL statements as an atomic batch: `CREATE TABLE`, `INSERT`, `UPDATE`, `DELETE`, `DROP TABLE`, `ALTER`, `COPY`, etc. `sql` is an array of statements; multi-element batches run inside a transaction (all commit or all roll back). Single-element batches auto-commit, same as a one-off statement. Returns the per-statement affected row counts plus a total. Disabled in read-only mode. -``` +```text // Single statement (auto-commit) execute(sql: ['CREATE TABLE archived_orders AS SELECT * FROM orders WHERE year < 2024']) @@ -459,7 +466,7 @@ List all tables in the selected database with their schemas, column types, and r Return the schema, total row count, and first N rows of a table in a single call. -``` +```text sample(table: 'orders', n: 10) ``` @@ -483,7 +490,7 @@ Use it **before** `load_file` whenever you are unsure about types, or **after** reported `type` + `min` / `max` directly into a partial `schema` override on the subsequent `load_file` call. -``` +```text inspect_file(path: '/tmp/owid-population.csv') ``` @@ -531,7 +538,7 @@ only for the lifetime of the server process. #### `save_query` -``` +```text save_query(name: 'top_5_customers', sql: 'SELECT customer, SUM(amount) AS total FROM orders GROUP BY customer ORDER BY total DESC LIMIT 5', description: 'Biggest spenders this year') ``` @@ -547,7 +554,7 @@ first if you intend to overwrite. Non-read-only SQL is rejected with #### `delete_query` -``` +```text delete_query(name: 'top_5_customers') ``` @@ -584,7 +591,7 @@ Nine tools cover the surface: | `kv_pop` | Destructively read-and-remove the lowest-keyed entry (atomic) | `store`, `database`, `persist` | | `kv_clear` | Delete all keys in a store (returns count removed) | `store`, `database`, `persist` | -``` +```text kv_set(store: 'session', key: 'last_report', value: '{"rows": 4210}', database: 'persistent') kv_get(store: 'session', key: 'last_report') ``` @@ -607,7 +614,7 @@ Key properties: Write query results or a table to a file. -``` +```text export(table: 'orders', path: '~/Desktop/orders.parquet', format: 'parquet') export(sql: 'SELECT ...', path: '~/Desktop/analysis.hyper', format: 'hyper') ``` @@ -630,7 +637,7 @@ destination and materializes every user table from the selected source into it. Render a bounded quick diagnostic from a SQL query. This convenience tool is for inspecting or sharing one chart, not for dashboard/layout composition. -``` +```text chart(sql: 'SELECT product, SUM(revenue) as total FROM sales GROUP BY product', chart_type: 'bar', x: 'product', y: 'total', title: 'Revenue by Product') ``` @@ -684,7 +691,7 @@ bound, never zero. Monitor a directory for data files and auto-append them to a target table. -``` +```text watch_directory(path: '/tmp/inbox', table: 'events') unwatch_directory(path: '/tmp/inbox') ``` @@ -899,7 +906,7 @@ Hyper uses the Salesforce Data Cloud SQL dialect (PostgreSQL-compatible with ext Hyper does **not** support `ON CONFLICT` or `INSERT ... ON DUPLICATE KEY`. Use the `execute` tool's atomic batch shape instead: -``` +```text execute(sql: [ "UPDATE settings SET value = 'dark' WHERE key = 'theme'", "INSERT INTO settings (key, value) SELECT 'theme', 'dark' \ @@ -927,7 +934,7 @@ Full reference: [Data Cloud SQL Reference](https://developer.salesforce.com/docs ## CLI Reference -``` +```text hyperdb-mcp [OPTIONS] [COMMAND] Commands: diff --git a/hyperdb-mcp/ROADMAP.md b/hyperdb-mcp/ROADMAP.md index 18df1b8..96227a4 100644 --- a/hyperdb-mcp/ROADMAP.md +++ b/hyperdb-mcp/ROADMAP.md @@ -38,7 +38,7 @@ survive hyperd crashes transparently. Example — JOIN across a scratch `.hyper` file and the primary workspace, then land the result: -``` +```text attach_database(alias="src", kind="local_file", path="/tmp/scratch.hyper") copy_query( sql="SELECT s.id, s.name, p.amount FROM src.public.customers s JOIN orders p ON s.id = p.customer_id", @@ -68,7 +68,7 @@ keep these workarounds in mind: 1. **`.hyper` → `.hyper` export then load.** Single-table transfer. Fastest because `.hyper` is Hyper's native format. - ``` + ```text # in HyperDB (sandbox) export(table="scratch_data", path="/tmp/scratch.hyper", format="hyper") # then in HyperDB-persistent diff --git a/hyperdb-mcp/SMOKE_TESTS.md b/hyperdb-mcp/SMOKE_TESTS.md index 5ebbd91..4c0b817 100644 --- a/hyperdb-mcp/SMOKE_TESTS.md +++ b/hyperdb-mcp/SMOKE_TESTS.md @@ -66,7 +66,7 @@ it started in. The final section is a verification checklist for that. - The `hyperdb` MCP tools connected and responding. - Confirm the server is up and note its mode before you start: -``` +```text status ``` @@ -122,7 +122,7 @@ resource. ## 2. Create / read / overwrite (upsert) -``` +```text kv_set store=smoke key=greeting value="hello world" → {"stored": true, "created": true, "value_bytes": 11, "store": "smoke", "key": "greeting", "resolved_database": "local"} kv_get store=smoke key=greeting → {"found": true, "value": "hello world", "resolved_database": "local"} kv_get store=smoke key=does_not_exist → {"found": false, "value": null, "resolved_database": "local"} @@ -132,7 +132,7 @@ A miss is **not** an error — `found: false` with a `null` value. Batch writes are atomic and validate every key before writing: -``` +```text kv_set_many store=smoke_batch entries=[{"key":"batch_a","value":"A"},{"key":"batch_b","value":"B"}] → {"stored": 2, "created": 2, "overwritten": 0, "total_bytes": 2, "resolved_database": "local"} kv_list store=smoke_batch @@ -144,7 +144,7 @@ kv_clear store=smoke_batch **Overwrite must not create a duplicate row** (the backing table is indexless; `kv_set` is an app-side upsert): -``` +```text kv_size store=smoke → {"store": "smoke", "size": 1, "bytes": 11, "resolved_database": "local"} kv_set store=smoke key=greeting value="HELLO AGAIN" → {"stored": true, "resolved_database": "local", ...} kv_size store=smoke → {"store": "smoke", "size": 1, "bytes": 11, "resolved_database": "local"} # still 1, not 2 @@ -157,7 +157,7 @@ kv_get store=smoke key=greeting → {"found": true, " Seed a few keys, then list: -``` +```text kv_set store=smoke key=alpha value=1 kv_set store=smoke key=bravo value=2 kv_set store=smoke key=charlie value=3 @@ -175,7 +175,7 @@ emptied store disappears from the list; see §5). ## 4. Value fidelity — JSON, empty, large -``` +```text kv_set store=smoke key=config value='{"retries": 3, "nested": {"flag": true}}' kv_get store=smoke key=config → {"found": true, "value": "{\"retries\": 3, \"nested\": {\"flag\": true}}", "resolved_database": "local"} # byte-for-byte @@ -195,7 +195,7 @@ must stay distinct from a miss `{"found": false, "value": null}`. **Delete is idempotent and reports whether the key existed:** -``` +```text kv_delete store=smoke key=greeting → {"deleted": true, "resolved_database": "local", ...} # existed kv_delete store=smoke key=greeting → {"deleted": false, "resolved_database": "local", ...} # already gone — no error kv_delete store=smoke key=never_existed → {"deleted": false, "resolved_database": "local", ...} @@ -204,7 +204,7 @@ kv_delete store=smoke key=never_existed → {"deleted": false, "resolved_datab **`kv_pop` destructively removes the lowest-keyed entry** (a work-queue drain in ascending key order): -``` +```text # with keys [alpha, bravo, charlie, config, empty_val, big_blob] present kv_pop store=smoke → {"found": true, "key": "alpha", "value": "1", "resolved_database": "local"} kv_pop store=smoke → {"found": true, "key": "big_blob", "value": "...", "resolved_database": "local"} # 'b' < 'c' @@ -213,7 +213,7 @@ kv_pop store=smoke → {"found": true, "key": "bravo", "value": "2", "res **`kv_clear` empties the store and returns the count removed:** -``` +```text kv_size store=smoke → {"store": "smoke", "size": N, "bytes": B, "resolved_database": "local"} kv_clear store=smoke → {"store": "smoke", "removed": N, "resolved_database": "local"} kv_size store=smoke → {"store": "smoke", "size": 0, "bytes": 0, "resolved_database": "local"} @@ -224,7 +224,7 @@ of the remaining values' UTF-8 byte lengths at that point. **Empty-store edge cases:** -``` +```text kv_pop store=smoke → {"found": false, "resolved_database": "local"} # nothing to pop kv_clear store=smoke → {"store": "smoke", "removed": 0, "resolved_database": "local"} # idempotent kv_list_stores → {"count": 0, "stores": [], "resolved_database": "local"} # emptied store drops out @@ -238,7 +238,7 @@ kv_list_stores → {"count": 0, "stores": [], "resolved_database": "loca Violations are rejected as **`INVALID_ARGUMENT`** (not `INTERNAL_ERROR`) with a message that names the offending byte or the actual length: -``` +```text kv_set store=smoke key="has a space" value=x → error INVALID_ARGUMENT: "invalid name: KV key contains an invalid byte 0x20; allowed: A-Z a-z 0-9 _ . -" @@ -260,7 +260,7 @@ Only relevant when the server runs with `--read-only` (`status` shows `"read_only": true`). Start such a server yourself for this check — do not assume the shared daemon is read-only. -``` +```text # readers work: kv_get store=smoke key=k → {"found": false, "value": null, "resolved_database": "local"} kv_list store=smoke → {"store": "smoke", "count": 0, "keys": [], "resolved_database": "local"} @@ -299,7 +299,7 @@ an explicit `database` wins over `persist: true` (for example, `database=PeRsIsTeNt` resolves to `persistent`; mixed-case attached aliases resolve to the registry's lowercase alias. -``` +```text kv_set store=smoke_routing key=where value="local" # → local (default) kv_set store=smoke_routing key=where value="persistent" database=persistent # → persistent kv_set store=smoke_routing key=where2 value="via-flag" persist=true # → persistent (same DB) @@ -329,7 +329,7 @@ The backing table `_hyperdb_kv_store(store_name, key, value)` is hidden from store: annotate analytical rows with scratchpad metadata via a plain SQL join. **Run this in the local DB** (create a `smoke_`-prefixed table): -``` +```text kv_set store=product_notes key=P1 value="flagship - review pricing Q3" kv_set store=product_notes key=P3 value="discontinue candidate" @@ -351,7 +351,7 @@ Expected: P1 and P3 carry their notes; **P2 survives with `note: null`** ## 10. Table is hidden but accessible -``` +```text describe → table list does NOT include _hyperdb_kv_store query SELECT COUNT(*) FROM _hyperdb_kv_store → succeeds (directly queryable) ``` @@ -388,7 +388,7 @@ DB constraint — that limitation is documented, not a smoke-test failure.) Purge every scratch store and table, then confirm the databases are back to their starting state: -``` +```text kv_clear store=smoke kv_clear store=smoke_routing kv_clear store=smoke_routing database=persistent