diff --git a/reference/configuration/options.md b/reference/configuration/options.md index d1212841..aa317964 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -232,7 +232,7 @@ logging: level: warn root: ~/hdb/log stdStreams: false - auditLog: false + auditLog: true rotation: interval: 1D maxSize: 100M @@ -244,7 +244,7 @@ logging: - `path` — Explicit log file path (overrides `root`) - `stdStreams` — Write to stdout/stderr; _Default_: `false` - `console` — Include `console.*` output; _Default_: `true` -- `auditLog` — Enable table transaction audit logging; _Default_: `false` +- `auditLog` — Enable table transaction audit logging; _Default_: `true` - `auditRetention` — Audit log retention duration; _Default_: `3d` - `external` — Logging for components using the logger API; sub-options: `level`, `path` - `rotation.enabled` / `rotation.compress` / `rotation.interval` / `rotation.maxSize` / `rotation.path` — Log file rotation (activates when `interval` or `maxSize` is set) @@ -280,7 +280,7 @@ replication: - `hostname` — This instance's hostname within the cluster. Subject to the same bare hostname or IP literal requirement as [`node.hostname`](#node) — a URL, `host:port`, or non-string value fails startup. When both are set, `node.hostname` wins. - `url` — WebSocket URL peers use to connect to this instance -- `databases` — Databases to replicate; _Default_: `"*"` (all). Each entry supports `name` and `sharded` +- `databases` — Databases to replicate; _Default_: `"*"` (all). Each entry supports `name`, `replicateTo`, and `sharded` - `routes` — Peer nodes; URL strings or `{hostname, port, startTime, revokedCertificates}` objects - `port` — Replication port - `securePort` — Secure replication port; _Default_: `9933` (changed from `9925` in v4.5.0) @@ -312,7 +312,7 @@ storage: ``` - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. -- `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` +- `writeAsync` — Disable disk sync on commit for the legacy LMDB storage engine (**disables local disk durability guarantees on LMDB**). Has no effect on RocksDB, which does not `fsync` commits; see [Storage Tuning — Durability vs. Throughput](../database/storage-tuning.md#durability-vs-throughput); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) diff --git a/reference/database/overview.md b/reference/database/overview.md index 406b251a..0a90c72d 100644 --- a/reference/database/overview.md +++ b/reference/database/overview.md @@ -109,7 +109,7 @@ For deeper coverage of each database feature, see the dedicated pages in this se - **[Schema](./schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax - **[API](./api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code - **[Data Loader](./data-loader.md)** — Loading seed or initial data into tables as part of component deployment -- **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance +- **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using RocksDB (or legacy LMDB) with universal indexing and ACID transactions - **[Storage Tuning](./storage-tuning.md)** — Tuning `storage.*` options for production: durability vs. throughput, compression, blob paths, reclamation - **[Jobs](./jobs.md)** — Asynchronous bulk data operations (CSV import/export, S3 import/export) - **[System Tables](./system-tables.md)** — Harper internal tables for analytics, data loader state, and other system features diff --git a/reference/database/storage-algorithm.md b/reference/database/storage-algorithm.md index de3d15c5..220cf5df 100644 --- a/reference/database/storage-algorithm.md +++ b/reference/database/storage-algorithm.md @@ -17,12 +17,12 @@ Harper's storage layer is decoupled from any specific query language. Data inser ## ACID Compliance -Harper provides full ACID compliance on each node: +Harper provides ACID transaction guarantees: - **Atomicity**: All writes in a transaction either fully commit or fully roll back - **Consistency**: Each transaction moves data from one valid state to another - **Isolation**: Reads use snapshots and do not block writes; writes do not block reads -- **Durability**: RocksDB commits are persisted via its Write-Ahead Log (WAL); LMDB uses memory-mapped file writes +- **Durability**: On RocksDB, when the [transaction log](./transaction.md) is enabled (the default), committed writes are recorded in Harper's transaction log in the OS page cache (without `fsync`; RocksDB's own WAL is disabled) and replayed on startup, surviving Harper process crashes on the node (disabling the transaction log via `logging.auditLog: false` or `@table(audit: false)` also disables process-crash recovery for unflushed memtable writes). Durability across OS, power, or host failures is provided by [replication confirmation](../replication/sharding.md) across independent nodes (see [Storage Tuning — Durability vs. Throughput](./storage-tuning.md#durability-vs-throughput)). LMDB uses memory-mapped file writes (`fsync` on commit by default unless [`storage.writeAsync`](./storage-tuning.md#storagewriteasync) is enabled) Harper uses application-level locking to serialize schema changes and table creation, ensuring write ordering without deadlocks. @@ -69,7 +69,7 @@ Harper inherits strong performance properties from its storage engines: - **LSM-tree writes**: Optimized for write-heavy workloads via log-structured merge trees - **Block cache**: Configurable in-memory block cache (defaults to 25% of available system memory) -- **WAL durability**: Write-Ahead Log provides crash recovery without sacrificing throughput +- **Transaction log recovery**: When transaction logging is enabled (the default), commits are written to Harper's transaction log in the OS page cache without blocking on `fsync` and replayed on startup for process-crash recovery - **Compression**: Native support for multiple compression algorithms per level **LMDB (legacy)**: diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 9cade5b2..b04616e3 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -10,21 +10,38 @@ For a quick reference of every option, see [Configuration Options — `storage`] ## Durability vs. Throughput +On RocksDB (the default storage engine), Harper does not call `fsync` or `fdatasync` when committing a transaction, and RocksDB's own Write-Ahead Log (WAL) is disabled for table stores. When the [transaction log](./transaction.md) is enabled (the default via [`logging.auditLog: true`](../logging/configuration.md) and [`@table(audit: true)`](./transaction.md#enabling-the-transaction-log-per-table)), each acknowledged commit is written to Harper's transaction log (`database//transaction_logs/`) using a standard write to the OS page cache. On startup, Harper replays the transaction log to recover any committed writes that had not yet been flushed from RocksDB's in-memory memtables to table files. + +What an acknowledged write survives on a single node: + +- **Harper process crash or restart**: Survived when transaction logging is enabled (the default). Committed bytes are already in the OS page cache, so the operating system preserves and flushes them to disk even if the Harper process terminates unexpectedly, and Harper replays the transaction log on restart. If transaction logging is disabled globally (`logging.auditLog: false`) or for a table (`@table(audit: false)`), no transaction-log entry is written while RocksDB's WAL remains off, so unflushed memtable writes are also lost on a Harper process crash. +- **OS crash, kernel panic, power loss, or host failure**: Recent writes that the OS has not yet flushed from the page cache to disk can be lost. On a single-node or Harper core deployment (where replication is not available), durability against OS or power failure depends on periodic OS page-cache writeback. The database remains structurally consistent — only the most recent un-flushed commits are at risk. + +For durability against machine, OS, power, or storage failure in a multi-node (Harper Pro) cluster, Harper relies on **replication confirmation** rather than local disk `fsync`. In production failures such as host failure or storage hardware loss, a local disk fails or becomes unreachable together with the rest of the machine, whereas an acknowledgement from a peer node on an independent machine means the write has been committed off-host. Because confirming RocksDB peers also commit to their own OS page cache without `fsync` and confirmation counts any `` responding peers, surviving a power outage or site-wide failure requires peers in separate power or failure domains and setting `` greater than the number of peers that share the writer's failure domain. + +To require confirmation from `` peer nodes before a write returns, use [replication confirmation](../replication/sharding.md): + +- **Operations API**: Set [`replicatedConfirmation: `](../replication/sharding.md#replication-control-via-operations-api) in the operation payload. Omit `replicateTo` to use the database's configured replication (every node by default, unless [`replication.databases[].replicateTo`](../replication/sharding.md#replication-count) is set), or set `replicateTo` only when you also want to limit residency (dynamic sharding) to a fixed set of nodes. +- **Resource API**: Set [`context.replicatedConfirmation = `](../replication/sharding.md#programmatic-replication-control) in your resource method (leaving `context.replicateTo` unset to use the database's configured replication, unless you also want to restrict residency). +- **REST**: Pass [`X-Replicate-To: *;confirm=`](../replication/sharding.md#replication-control-via-rest-header) on the request to wait for `` confirmations while keeping the database's configured replication (`*` leaves per-request residency unset; passing a number `` or node list in `X-Replicate-To: ;confirm=` pins the record's residency to `` additional nodes from those present at write time). + ### `storage.writeAsync` + + Type: `boolean` Default: `false` -Disables `fsync` on commit. Writes return as soon as data is queued to the page cache, dramatically increasing throughput on write-heavy workloads. +Applies only to the legacy LMDB storage engine. This setting has no effect on RocksDB, which never calls `fsync` on commit. -**This disables durability guarantees.** A power loss or OS crash between the application commit and the OS flushing pages to disk can lose recently committed transactions. The database itself remains structurally consistent — only the most recent writes are at risk. +On LMDB, enabling `writeAsync` disables `fsync` on commit so writes return as soon as data is queued to the OS page cache, matching RocksDB's commit behavior and increasing throughput on write-heavy workloads. A power loss or OS crash before the OS flushes pages to disk can lose recently committed transactions, while the database remains structurally consistent. -Enable only when: +Enable on LMDB databases when: +- Durability across machine failures is handled via [replication confirmation](../replication/sharding.md). - The data is reproducible from an upstream source (e.g., caches with `sourcedFrom`). -- The workload is bulk ingest where some loss on crash is acceptable and the operation can be re-run. -- Replication provides durability — peers acknowledge writes before they could be lost. +- The workload is bulk ingest where some loss on OS crash or power loss is acceptable and the operation can be re-run. ```yaml storage: @@ -367,11 +384,15 @@ Retains the pre-compaction backup files after `compactOnStart` runs. Recommended **Write-heavy ingest, durability via replication:** +On RocksDB (the default), commits already avoid local disk `fsync`; pair a higher queue-time threshold in `harper-config.yaml` with per-write [replication confirmation](../replication/sharding.md) (`replicatedConfirmation` in the Operations or Resource API, or `X-Replicate-To: *;confirm=` over REST) when writes must be confirmed by peer nodes before returning (on legacy LMDB databases, also add `writeAsync: true`): + ```yaml storage: - writeAsync: true prefetchWrites: true maxTransactionQueueTime: 60s + # Per-write replication confirmation is set on the operation/request + # (e.g. replicatedConfirmation: 1 or X-Replicate-To: *;confirm=1), + # not in harper-config.yaml. ``` **Read-heavy cache layer with large blobs:** @@ -399,6 +420,7 @@ storage: - [Configuration Options](../configuration/options.md) — full list of `storage` options - [Storage Algorithm](./storage-algorithm.md) — how Harper stores records and indexes on disk +- [Sharding](../replication/sharding.md) — configuring `replicateTo` and replication confirmation for cross-node durability - [Schema — `@table(randomAccessFields:)`](./schema.md#randomaccessfields) — pinning one table's record encoding - [Compaction](./compaction.md) — reclaiming space inside existing database files - [Resource API — `sourcedFrom`](../resources/resource-api.md#sourcedfromresource-options) — caching tables that interact with reclamation diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 02d46fb5..90f5eb73 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -30,14 +30,17 @@ There are two main approaches: ### Replication Count -The simplest way to limit replication is to configure a replication count. Set `replicateTo` in the `replication` section of `harper-config.yaml` to specify how many additional nodes data should be replicated to: +The simplest way to limit replication is to configure a per-database replication count. Set `replicateTo` on a database entry under `replication.databases` in `harper-config.yaml` to specify how many additional nodes data should be replicated to: ```yaml replication: - replicateTo: 2 + databases: + - system + - name: data + replicateTo: 2 ``` -This ensures each record is stored on three nodes total (the node that first stored it, plus two others). +This ensures each record in `data` is stored on three nodes total (the node that first stored it, plus two others). Because setting `replication.databases` to a list replaces the `'*'` default (all databases), include `system` and every other database the cluster should continue replicating (see [Data Selection](./overview.md#data-selection)). ### Replication Control via REST Header @@ -58,7 +61,12 @@ PUT /MyTable/3 X-Replicate-To: node1,node2 ``` -The `confirm` parameter can be combined with explicit node lists. +The `confirm` parameter can be combined with explicit node lists, or with `*` to require confirmation while keeping the cluster's default replication without pinning residency to a fixed node count: + +```http +PUT /MyTable/3 +X-Replicate-To: *;confirm=1 +``` ### Replication Control via Operations API