From 390c2d0ccb788f2d45cf42cb91b61d96072bc650 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 16 Oct 2026 14:54:04 -0600 Subject: [PATCH 1/5] docs: clarify RocksDB no-fsync durability and LMDB-only writeAsync Explain that RocksDB commits write to Harper's transaction log in the OS page cache without calling fsync (with RocksDB's own WAL disabled) and rely on replication confirmation for durability across OS, power, or host failures. Mark storage.writeAsync as LMDB-only and update the storage algorithm and tuning pages accordingly. Refs HarperFast/harper#3243 Dispatch-Task: docs-no-fsync-durability Co-Authored-By: Gemini --- reference/configuration/options.md | 2 +- reference/database/storage-algorithm.md | 4 ++-- reference/database/storage-tuning.md | 31 ++++++++++++++++++++----- 3 files changed, 28 insertions(+), 9 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index d12128410..b2f512687 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -312,7 +312,7 @@ storage: ``` - `engine` — The database storage engine to use for new databases. Currently supported engines are `rocksdb` and `lmdb`. The default is `rocksdb`. Existing databases will use the engine that was used when they were created. -- `writeAsync` — Disable disk sync for higher throughput (**disables durability guarantees**); _Default_: `false` +- `writeAsync` — Disable disk sync on commit for the legacy LMDB storage engine (**disables local disk durability guarantees on LMDB**). Has no effect on RocksDB, which does not `fsync` commits; see [Storage Tuning — Durability vs. Throughput](../database/storage-tuning.md#durability-vs-throughput); _Default_: `false` - `caching` — In-memory record caching; _Default_: `true` - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) diff --git a/reference/database/storage-algorithm.md b/reference/database/storage-algorithm.md index de3d15c52..01d5dbc07 100644 --- a/reference/database/storage-algorithm.md +++ b/reference/database/storage-algorithm.md @@ -22,7 +22,7 @@ Harper provides full ACID compliance on each node: - **Atomicity**: All writes in a transaction either fully commit or fully roll back - **Consistency**: Each transaction moves data from one valid state to another - **Isolation**: Reads use snapshots and do not block writes; writes do not block reads -- **Durability**: RocksDB commits are persisted via its Write-Ahead Log (WAL); LMDB uses memory-mapped file writes +- **Durability**: On RocksDB, committed writes are recorded in Harper's transaction log (without `fsync`; RocksDB's own WAL is disabled) and replayed on startup, surviving process crashes on the node; durability across OS, power, or host failures is provided by [replication confirmation](../replication/sharding.md#replication-control-via-rest-header) (see [Storage Tuning — Durability vs. Throughput](./storage-tuning.md#durability-vs-throughput)). LMDB uses memory-mapped file writes (`fsync` on commit by default unless [`storage.writeAsync`](./storage-tuning.md#storagewriteasync) is enabled) Harper uses application-level locking to serialize schema changes and table creation, ensuring write ordering without deadlocks. @@ -69,7 +69,7 @@ Harper inherits strong performance properties from its storage engines: - **LSM-tree writes**: Optimized for write-heavy workloads via log-structured merge trees - **Block cache**: Configurable in-memory block cache (defaults to 25% of available system memory) -- **WAL durability**: Write-Ahead Log provides crash recovery without sacrificing throughput +- **Transaction log recovery**: Commits are written to Harper's transaction log in the OS page cache without blocking on `fsync` and replayed on startup for process-crash recovery - **Compression**: Native support for multiple compression algorithms per level **LMDB (legacy)**: diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 9cade5b28..6bf01966f 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -10,21 +10,38 @@ For a quick reference of every option, see [Configuration Options — `storage`] ## Durability vs. Throughput +On RocksDB (the default storage engine), Harper does not call `fsync` or `fdatasync` when committing a transaction. RocksDB's own Write-Ahead Log (WAL) is disabled for table stores; instead, each acknowledged commit is written to Harper's transaction log (`database//transaction_logs/...txnlog`) using a standard write to the OS page cache. On startup, Harper replays the transaction log to recover any committed writes that had not yet been flushed to RocksDB table files. + +What an acknowledged write survives on a single node: + +- **Harper process crash or restart**: Survived. Committed bytes are already in the OS page cache, so the operating system preserves and flushes them to disk even if the Harper process terminates unexpectedly, and Harper replays the transaction log on restart. +- **OS crash, kernel panic, power loss, or host failure**: Recent writes that the OS has not yet flushed from the page cache to disk can be lost. The database remains structurally consistent — only the most recent un-flushed commits are at risk. + +For durability against machine, OS, power, or storage failure, Harper relies on **replication confirmation** rather than local disk `fsync`. An acknowledgement from another node is a much stronger durability guarantee than a local drive's `fsync`, because in the failures that matter in production — host failure, storage hardware loss, or datacenter outage — a local disk fails or becomes unreachable together with the rest of the machine. + +To require confirmation from one or more peer nodes before a write returns, use [replication confirmation](../replication/sharding.md): + +- **REST**: Pass [`X-Replicate-To: ;confirm=`](../replication/sharding.md#replication-control-via-rest-header) on the request. +- **Operations API**: Set [`replicatedConfirmation`](../replication/sharding.md#replication-control-via-operations-api) (alongside `replicateTo`) in the operation payload. +- **Resource API**: Set [`context.replicatedConfirmation`](../replication/sharding.md#programmatic-replication-control) in your resource method. + ### `storage.writeAsync` + + Type: `boolean` Default: `false` -Disables `fsync` on commit. Writes return as soon as data is queued to the page cache, dramatically increasing throughput on write-heavy workloads. +Applies only to the legacy LMDB storage engine. This setting has no effect on RocksDB, which never calls `fsync` on commit. -**This disables durability guarantees.** A power loss or OS crash between the application commit and the OS flushing pages to disk can lose recently committed transactions. The database itself remains structurally consistent — only the most recent writes are at risk. +On LMDB, enabling `writeAsync` disables `fsync` on commit so writes return as soon as data is queued to the OS page cache, matching RocksDB's commit behavior and increasing throughput on write-heavy workloads. A power loss or OS crash before the OS flushes pages to disk can lose recently committed transactions, while the database remains structurally consistent. -Enable only when: +Enable on LMDB databases when: +- Durability across machine failures is handled via [replication confirmation](../replication/sharding.md). - The data is reproducible from an upstream source (e.g., caches with `sourcedFrom`). -- The workload is bulk ingest where some loss on crash is acceptable and the operation can be re-run. -- Replication provides durability — peers acknowledge writes before they could be lost. +- The workload is bulk ingest where some loss on OS crash or power loss is acceptable and the operation can be re-run. ```yaml storage: @@ -367,9 +384,10 @@ Retains the pre-compaction backup files after `compactOnStart` runs. Recommended **Write-heavy ingest, durability via replication:** +On RocksDB (the default), commits already avoid local disk `fsync`; pair a higher queue-time threshold with [replication confirmation](../replication/sharding.md#replication-control-via-rest-header) when writes must be confirmed by peer nodes before returning (on legacy LMDB databases, also add `writeAsync: true`): + ```yaml storage: - writeAsync: true prefetchWrites: true maxTransactionQueueTime: 60s ``` @@ -399,6 +417,7 @@ storage: - [Configuration Options](../configuration/options.md) — full list of `storage` options - [Storage Algorithm](./storage-algorithm.md) — how Harper stores records and indexes on disk +- [Sharding](../replication/sharding.md) — configuring `replicateTo` and replication confirmation for cross-node durability - [Schema — `@table(randomAccessFields:)`](./schema.md#randomaccessfields) — pinning one table's record encoding - [Compaction](./compaction.md) — reclaiming space inside existing database files - [Resource API — `sourcedFrom`](../resources/resource-api.md#sourcedfromresource-options) — caching tables that interact with reclamation From 07f8b6c4f26b4b6c8d633d790f1c7ba1bd001860 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 16 Oct 2026 15:03:07 -0600 Subject: [PATCH 2/5] docs: clarify transaction-log crash recovery and replication residency Qualify that process-crash recovery on RocksDB requires the transaction log to be enabled (the default via logging.auditLog: true and @table(audit: true)), distinguish replication confirmation from replicateTo residency, note failure domain and single-node constraints, correct logging.auditLog default to true in options.md, and update the Storage Algorithm summary in database/overview.md. Refs HarperFast/harper#3243 Dispatch-Task: docs-no-fsync-durability Co-Authored-By: Gemini --- reference/configuration/options.md | 4 ++-- reference/database/overview.md | 2 +- reference/database/storage-algorithm.md | 6 +++--- reference/database/storage-tuning.md | 21 ++++++++++++--------- 4 files changed, 18 insertions(+), 15 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index b2f512687..0cb9a22d1 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -232,7 +232,7 @@ logging: level: warn root: ~/hdb/log stdStreams: false - auditLog: false + auditLog: true rotation: interval: 1D maxSize: 100M @@ -244,7 +244,7 @@ logging: - `path` — Explicit log file path (overrides `root`) - `stdStreams` — Write to stdout/stderr; _Default_: `false` - `console` — Include `console.*` output; _Default_: `true` -- `auditLog` — Enable table transaction audit logging; _Default_: `false` +- `auditLog` — Enable table transaction audit logging; _Default_: `true` - `auditRetention` — Audit log retention duration; _Default_: `3d` - `external` — Logging for components using the logger API; sub-options: `level`, `path` - `rotation.enabled` / `rotation.compress` / `rotation.interval` / `rotation.maxSize` / `rotation.path` — Log file rotation (activates when `interval` or `maxSize` is set) diff --git a/reference/database/overview.md b/reference/database/overview.md index 406b251a9..0a90c72d5 100644 --- a/reference/database/overview.md +++ b/reference/database/overview.md @@ -109,7 +109,7 @@ For deeper coverage of each database feature, see the dedicated pages in this se - **[Schema](./schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax - **[API](./api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code - **[Data Loader](./data-loader.md)** — Loading seed or initial data into tables as part of component deployment -- **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance +- **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using RocksDB (or legacy LMDB) with universal indexing and ACID transactions - **[Storage Tuning](./storage-tuning.md)** — Tuning `storage.*` options for production: durability vs. throughput, compression, blob paths, reclamation - **[Jobs](./jobs.md)** — Asynchronous bulk data operations (CSV import/export, S3 import/export) - **[System Tables](./system-tables.md)** — Harper internal tables for analytics, data loader state, and other system features diff --git a/reference/database/storage-algorithm.md b/reference/database/storage-algorithm.md index 01d5dbc07..220cf5dff 100644 --- a/reference/database/storage-algorithm.md +++ b/reference/database/storage-algorithm.md @@ -17,12 +17,12 @@ Harper's storage layer is decoupled from any specific query language. Data inser ## ACID Compliance -Harper provides full ACID compliance on each node: +Harper provides ACID transaction guarantees: - **Atomicity**: All writes in a transaction either fully commit or fully roll back - **Consistency**: Each transaction moves data from one valid state to another - **Isolation**: Reads use snapshots and do not block writes; writes do not block reads -- **Durability**: On RocksDB, committed writes are recorded in Harper's transaction log (without `fsync`; RocksDB's own WAL is disabled) and replayed on startup, surviving process crashes on the node; durability across OS, power, or host failures is provided by [replication confirmation](../replication/sharding.md#replication-control-via-rest-header) (see [Storage Tuning — Durability vs. Throughput](./storage-tuning.md#durability-vs-throughput)). LMDB uses memory-mapped file writes (`fsync` on commit by default unless [`storage.writeAsync`](./storage-tuning.md#storagewriteasync) is enabled) +- **Durability**: On RocksDB, when the [transaction log](./transaction.md) is enabled (the default), committed writes are recorded in Harper's transaction log in the OS page cache (without `fsync`; RocksDB's own WAL is disabled) and replayed on startup, surviving Harper process crashes on the node (disabling the transaction log via `logging.auditLog: false` or `@table(audit: false)` also disables process-crash recovery for unflushed memtable writes). Durability across OS, power, or host failures is provided by [replication confirmation](../replication/sharding.md) across independent nodes (see [Storage Tuning — Durability vs. Throughput](./storage-tuning.md#durability-vs-throughput)). LMDB uses memory-mapped file writes (`fsync` on commit by default unless [`storage.writeAsync`](./storage-tuning.md#storagewriteasync) is enabled) Harper uses application-level locking to serialize schema changes and table creation, ensuring write ordering without deadlocks. @@ -69,7 +69,7 @@ Harper inherits strong performance properties from its storage engines: - **LSM-tree writes**: Optimized for write-heavy workloads via log-structured merge trees - **Block cache**: Configurable in-memory block cache (defaults to 25% of available system memory) -- **Transaction log recovery**: Commits are written to Harper's transaction log in the OS page cache without blocking on `fsync` and replayed on startup for process-crash recovery +- **Transaction log recovery**: When transaction logging is enabled (the default), commits are written to Harper's transaction log in the OS page cache without blocking on `fsync` and replayed on startup for process-crash recovery - **Compression**: Native support for multiple compression algorithms per level **LMDB (legacy)**: diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 6bf01966f..2d630bb29 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -10,20 +10,20 @@ For a quick reference of every option, see [Configuration Options — `storage`] ## Durability vs. Throughput -On RocksDB (the default storage engine), Harper does not call `fsync` or `fdatasync` when committing a transaction. RocksDB's own Write-Ahead Log (WAL) is disabled for table stores; instead, each acknowledged commit is written to Harper's transaction log (`database//transaction_logs/...txnlog`) using a standard write to the OS page cache. On startup, Harper replays the transaction log to recover any committed writes that had not yet been flushed to RocksDB table files. +On RocksDB (the default storage engine), Harper does not call `fsync` or `fdatasync` when committing a transaction, and RocksDB's own Write-Ahead Log (WAL) is disabled for table stores. When the [transaction log](./transaction.md) is enabled (the default via [`logging.auditLog: true`](../logging/configuration.md) and [`@table(audit: true)`](./transaction.md#enabling-the-transaction-log-per-table)), each acknowledged commit is written to Harper's transaction log (`database//transaction_logs/`) using a standard write to the OS page cache. On startup, Harper replays the transaction log to recover any committed writes that had not yet been flushed from RocksDB's in-memory memtables to table files. What an acknowledged write survives on a single node: -- **Harper process crash or restart**: Survived. Committed bytes are already in the OS page cache, so the operating system preserves and flushes them to disk even if the Harper process terminates unexpectedly, and Harper replays the transaction log on restart. -- **OS crash, kernel panic, power loss, or host failure**: Recent writes that the OS has not yet flushed from the page cache to disk can be lost. The database remains structurally consistent — only the most recent un-flushed commits are at risk. +- **Harper process crash or restart**: Survived when transaction logging is enabled (the default). Committed bytes are already in the OS page cache, so the operating system preserves and flushes them to disk even if the Harper process terminates unexpectedly, and Harper replays the transaction log on restart. If transaction logging is disabled globally (`logging.auditLog: false`) or for a table (`@table(audit: false)`), no transaction-log entry is written while RocksDB's WAL remains off, so unflushed memtable writes are also lost on a Harper process crash. +- **OS crash, kernel panic, power loss, or host failure**: Recent writes that the OS has not yet flushed from the page cache to disk can be lost. On a single-node deployment, durability against OS or power failure depends on periodic OS page-cache writeback. The database remains structurally consistent — only the most recent un-flushed commits are at risk. -For durability against machine, OS, power, or storage failure, Harper relies on **replication confirmation** rather than local disk `fsync`. An acknowledgement from another node is a much stronger durability guarantee than a local drive's `fsync`, because in the failures that matter in production — host failure, storage hardware loss, or datacenter outage — a local disk fails or becomes unreachable together with the rest of the machine. +For durability against machine, OS, power, or storage failure in a multi-node cluster, Harper relies on **replication confirmation** rather than local disk `fsync`. In production failures such as host failure or storage hardware loss, a local disk fails or becomes unreachable together with the rest of the machine, whereas an acknowledgement from a peer node on an independent machine means the write has been committed off-host. Because confirming RocksDB peers also commit to their own OS page cache without `fsync`, surviving a power outage or site-wide failure requires confirming peers in separate power or failure domains. -To require confirmation from one or more peer nodes before a write returns, use [replication confirmation](../replication/sharding.md): +To require confirmation from `` peer nodes before a write returns, use [replication confirmation](../replication/sharding.md): -- **REST**: Pass [`X-Replicate-To: ;confirm=`](../replication/sharding.md#replication-control-via-rest-header) on the request. -- **Operations API**: Set [`replicatedConfirmation`](../replication/sharding.md#replication-control-via-operations-api) (alongside `replicateTo`) in the operation payload. -- **Resource API**: Set [`context.replicatedConfirmation`](../replication/sharding.md#programmatic-replication-control) in your resource method. +- **Operations API**: Set [`replicatedConfirmation: `](../replication/sharding.md#replication-control-via-operations-api) in the operation payload. Omit `replicateTo` to keep full-cluster replication, or set `replicateTo` only when you also want to limit residency (dynamic sharding) to a subset of nodes. +- **Resource API**: Set [`context.replicatedConfirmation = `](../replication/sharding.md#programmatic-replication-control) in your resource method (leaving `context.replicateTo` unset unless you also want to restrict residency). +- **REST**: Pass [`X-Replicate-To: ;confirm=`](../replication/sharding.md#replication-control-via-rest-header) on the request. Because `` in this header also sets how many additional nodes store the record, set `` to the full peer count (or list all target nodes) when you want full-cluster residency rather than sharding the record to `` peers. ### `storage.writeAsync` @@ -384,12 +384,15 @@ Retains the pre-compaction backup files after `compactOnStart` runs. Recommended **Write-heavy ingest, durability via replication:** -On RocksDB (the default), commits already avoid local disk `fsync`; pair a higher queue-time threshold with [replication confirmation](../replication/sharding.md#replication-control-via-rest-header) when writes must be confirmed by peer nodes before returning (on legacy LMDB databases, also add `writeAsync: true`): +On RocksDB (the default), commits already avoid local disk `fsync`; pair a higher queue-time threshold in `harper-config.yaml` with per-write [replication confirmation](../replication/sharding.md) (`replicatedConfirmation` in the Operations or Resource API, or `X-Replicate-To: ;confirm=` over REST with `` set to your peer count) when writes must be confirmed by peer nodes before returning (on legacy LMDB databases, also add `writeAsync: true`): ```yaml storage: prefetchWrites: true maxTransactionQueueTime: 60s + # Per-write replication confirmation is set on the operation/request + # (e.g. replicatedConfirmation: 1 or X-Replicate-To: ;confirm=1), + # not in harper-config.yaml. ``` **Read-heavy cache layer with large blobs:** From ed8a9560242a05418ecbb74681adedf21ca98187 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 16 Oct 2026 15:08:03 -0600 Subject: [PATCH 3/5] docs: recommend X-Replicate-To: *;confirm= for full-cluster REST confirmation Document X-Replicate-To: *;confirm= in storage-tuning.md and sharding.md so REST clients can require peer confirmation without pinning residency to a fixed node list, note the interaction with replication.replicateTo when omitting replicateTo, and explicitly note that replication requires Harper Pro. Refs HarperFast/harper#3243 Dispatch-Task: docs-no-fsync-durability Co-Authored-By: Gemini --- reference/database/storage-tuning.md | 14 +++++++------- reference/replication/sharding.md | 7 ++++++- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 2d630bb29..78684b237 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -15,15 +15,15 @@ On RocksDB (the default storage engine), Harper does not call `fsync` or `fdatas What an acknowledged write survives on a single node: - **Harper process crash or restart**: Survived when transaction logging is enabled (the default). Committed bytes are already in the OS page cache, so the operating system preserves and flushes them to disk even if the Harper process terminates unexpectedly, and Harper replays the transaction log on restart. If transaction logging is disabled globally (`logging.auditLog: false`) or for a table (`@table(audit: false)`), no transaction-log entry is written while RocksDB's WAL remains off, so unflushed memtable writes are also lost on a Harper process crash. -- **OS crash, kernel panic, power loss, or host failure**: Recent writes that the OS has not yet flushed from the page cache to disk can be lost. On a single-node deployment, durability against OS or power failure depends on periodic OS page-cache writeback. The database remains structurally consistent — only the most recent un-flushed commits are at risk. +- **OS crash, kernel panic, power loss, or host failure**: Recent writes that the OS has not yet flushed from the page cache to disk can be lost. On a single-node or Harper core deployment (where replication is not available), durability against OS or power failure depends on periodic OS page-cache writeback. The database remains structurally consistent — only the most recent un-flushed commits are at risk. -For durability against machine, OS, power, or storage failure in a multi-node cluster, Harper relies on **replication confirmation** rather than local disk `fsync`. In production failures such as host failure or storage hardware loss, a local disk fails or becomes unreachable together with the rest of the machine, whereas an acknowledgement from a peer node on an independent machine means the write has been committed off-host. Because confirming RocksDB peers also commit to their own OS page cache without `fsync`, surviving a power outage or site-wide failure requires confirming peers in separate power or failure domains. +For durability against machine, OS, power, or storage failure in a multi-node (Harper Pro) cluster, Harper relies on **replication confirmation** rather than local disk `fsync`. In production failures such as host failure or storage hardware loss, a local disk fails or becomes unreachable together with the rest of the machine, whereas an acknowledgement from a peer node on an independent machine means the write has been committed off-host. Because confirming RocksDB peers also commit to their own OS page cache without `fsync`, surviving a power outage or site-wide failure requires confirming peers in separate power or failure domains. To require confirmation from `` peer nodes before a write returns, use [replication confirmation](../replication/sharding.md): -- **Operations API**: Set [`replicatedConfirmation: `](../replication/sharding.md#replication-control-via-operations-api) in the operation payload. Omit `replicateTo` to keep full-cluster replication, or set `replicateTo` only when you also want to limit residency (dynamic sharding) to a subset of nodes. -- **Resource API**: Set [`context.replicatedConfirmation = `](../replication/sharding.md#programmatic-replication-control) in your resource method (leaving `context.replicateTo` unset unless you also want to restrict residency). -- **REST**: Pass [`X-Replicate-To: ;confirm=`](../replication/sharding.md#replication-control-via-rest-header) on the request. Because `` in this header also sets how many additional nodes store the record, set `` to the full peer count (or list all target nodes) when you want full-cluster residency rather than sharding the record to `` peers. +- **Operations API**: Set [`replicatedConfirmation: `](../replication/sharding.md#replication-control-via-operations-api) in the operation payload. Omit `replicateTo` to use the cluster's configured replication (every node by default, unless [`replication.replicateTo`](../replication/sharding.md#replication-count) is set), or set `replicateTo` only when you also want to limit residency (dynamic sharding) to a fixed set of nodes. +- **Resource API**: Set [`context.replicatedConfirmation = `](../replication/sharding.md#programmatic-replication-control) in your resource method (leaving `context.replicateTo` unset to use the cluster's configured replication, unless you also want to restrict residency). +- **REST**: Pass [`X-Replicate-To: *;confirm=`](../replication/sharding.md#replication-control-via-rest-header) on the request to wait for `` confirmations while keeping the cluster's configured replication (`*` leaves per-request residency unset; passing a number `` or node list in `X-Replicate-To: ;confirm=` pins the record's residency to `` additional nodes from those present at write time). ### `storage.writeAsync` @@ -384,14 +384,14 @@ Retains the pre-compaction backup files after `compactOnStart` runs. Recommended **Write-heavy ingest, durability via replication:** -On RocksDB (the default), commits already avoid local disk `fsync`; pair a higher queue-time threshold in `harper-config.yaml` with per-write [replication confirmation](../replication/sharding.md) (`replicatedConfirmation` in the Operations or Resource API, or `X-Replicate-To: ;confirm=` over REST with `` set to your peer count) when writes must be confirmed by peer nodes before returning (on legacy LMDB databases, also add `writeAsync: true`): +On RocksDB (the default), commits already avoid local disk `fsync`; pair a higher queue-time threshold in `harper-config.yaml` with per-write [replication confirmation](../replication/sharding.md) (`replicatedConfirmation` in the Operations or Resource API, or `X-Replicate-To: *;confirm=` over REST) when writes must be confirmed by peer nodes before returning (on legacy LMDB databases, also add `writeAsync: true`): ```yaml storage: prefetchWrites: true maxTransactionQueueTime: 60s # Per-write replication confirmation is set on the operation/request - # (e.g. replicatedConfirmation: 1 or X-Replicate-To: ;confirm=1), + # (e.g. replicatedConfirmation: 1 or X-Replicate-To: *;confirm=1), # not in harper-config.yaml. ``` diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 02d46fb56..0c3ed8eeb 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -58,7 +58,12 @@ PUT /MyTable/3 X-Replicate-To: node1,node2 ``` -The `confirm` parameter can be combined with explicit node lists. +The `confirm` parameter can be combined with explicit node lists, or with `*` to require confirmation while keeping the cluster's default replication without pinning residency to a fixed node count: + +```http +PUT /MyTable/3 +X-Replicate-To: *;confirm=1 +``` ### Replication Control via Operations API From 4db5dd8b120e9cc08152cca63d3ca76802c624cc Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 16 Oct 2026 15:12:31 -0600 Subject: [PATCH 4/5] docs: clarify any-k confirmation failure domains and replication.databases[].replicateTo Note in storage-tuning.md that replication confirmation counts any responding peers (so must exceed the number of peers in the writer's failure domain to survive a site-wide failure), and reference per-database replication.databases[].replicateTo in storage-tuning.md and sharding.md. Refs HarperFast/harper#3243 Dispatch-Task: docs-no-fsync-durability Co-Authored-By: Gemini --- reference/database/storage-tuning.md | 8 ++++---- reference/replication/sharding.md | 8 +++++--- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 78684b237..b04616e33 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -17,13 +17,13 @@ What an acknowledged write survives on a single node: - **Harper process crash or restart**: Survived when transaction logging is enabled (the default). Committed bytes are already in the OS page cache, so the operating system preserves and flushes them to disk even if the Harper process terminates unexpectedly, and Harper replays the transaction log on restart. If transaction logging is disabled globally (`logging.auditLog: false`) or for a table (`@table(audit: false)`), no transaction-log entry is written while RocksDB's WAL remains off, so unflushed memtable writes are also lost on a Harper process crash. - **OS crash, kernel panic, power loss, or host failure**: Recent writes that the OS has not yet flushed from the page cache to disk can be lost. On a single-node or Harper core deployment (where replication is not available), durability against OS or power failure depends on periodic OS page-cache writeback. The database remains structurally consistent — only the most recent un-flushed commits are at risk. -For durability against machine, OS, power, or storage failure in a multi-node (Harper Pro) cluster, Harper relies on **replication confirmation** rather than local disk `fsync`. In production failures such as host failure or storage hardware loss, a local disk fails or becomes unreachable together with the rest of the machine, whereas an acknowledgement from a peer node on an independent machine means the write has been committed off-host. Because confirming RocksDB peers also commit to their own OS page cache without `fsync`, surviving a power outage or site-wide failure requires confirming peers in separate power or failure domains. +For durability against machine, OS, power, or storage failure in a multi-node (Harper Pro) cluster, Harper relies on **replication confirmation** rather than local disk `fsync`. In production failures such as host failure or storage hardware loss, a local disk fails or becomes unreachable together with the rest of the machine, whereas an acknowledgement from a peer node on an independent machine means the write has been committed off-host. Because confirming RocksDB peers also commit to their own OS page cache without `fsync` and confirmation counts any `` responding peers, surviving a power outage or site-wide failure requires peers in separate power or failure domains and setting `` greater than the number of peers that share the writer's failure domain. To require confirmation from `` peer nodes before a write returns, use [replication confirmation](../replication/sharding.md): -- **Operations API**: Set [`replicatedConfirmation: `](../replication/sharding.md#replication-control-via-operations-api) in the operation payload. Omit `replicateTo` to use the cluster's configured replication (every node by default, unless [`replication.replicateTo`](../replication/sharding.md#replication-count) is set), or set `replicateTo` only when you also want to limit residency (dynamic sharding) to a fixed set of nodes. -- **Resource API**: Set [`context.replicatedConfirmation = `](../replication/sharding.md#programmatic-replication-control) in your resource method (leaving `context.replicateTo` unset to use the cluster's configured replication, unless you also want to restrict residency). -- **REST**: Pass [`X-Replicate-To: *;confirm=`](../replication/sharding.md#replication-control-via-rest-header) on the request to wait for `` confirmations while keeping the cluster's configured replication (`*` leaves per-request residency unset; passing a number `` or node list in `X-Replicate-To: ;confirm=` pins the record's residency to `` additional nodes from those present at write time). +- **Operations API**: Set [`replicatedConfirmation: `](../replication/sharding.md#replication-control-via-operations-api) in the operation payload. Omit `replicateTo` to use the database's configured replication (every node by default, unless [`replication.databases[].replicateTo`](../replication/sharding.md#replication-count) is set), or set `replicateTo` only when you also want to limit residency (dynamic sharding) to a fixed set of nodes. +- **Resource API**: Set [`context.replicatedConfirmation = `](../replication/sharding.md#programmatic-replication-control) in your resource method (leaving `context.replicateTo` unset to use the database's configured replication, unless you also want to restrict residency). +- **REST**: Pass [`X-Replicate-To: *;confirm=`](../replication/sharding.md#replication-control-via-rest-header) on the request to wait for `` confirmations while keeping the database's configured replication (`*` leaves per-request residency unset; passing a number `` or node list in `X-Replicate-To: ;confirm=` pins the record's residency to `` additional nodes from those present at write time). ### `storage.writeAsync` diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 0c3ed8eeb..1b3d0484c 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -30,14 +30,16 @@ There are two main approaches: ### Replication Count -The simplest way to limit replication is to configure a replication count. Set `replicateTo` in the `replication` section of `harper-config.yaml` to specify how many additional nodes data should be replicated to: +The simplest way to limit replication is to configure a per-database replication count. Set `replicateTo` on a database entry under `replication.databases` in `harper-config.yaml` to specify how many additional nodes data should be replicated to: ```yaml replication: - replicateTo: 2 + databases: + - name: data + replicateTo: 2 ``` -This ensures each record is stored on three nodes total (the node that first stored it, plus two others). +This ensures each record in that database is stored on three nodes total (the node that first stored it, plus two others). ### Replication Control via REST Header From 8996b507241a20151234a9525070dababb260084 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 16 Oct 2026 15:15:38 -0600 Subject: [PATCH 5/5] docs: include system in replication.databases example and note allowlist behavior Include system in the replication.databases example in sharding.md, note that setting replication.databases to a list replaces the '*' default, and list replicateTo in the replication.databases sub-options in options.md. Refs HarperFast/harper#3243 Dispatch-Task: docs-no-fsync-durability Co-Authored-By: Gemini --- reference/configuration/options.md | 2 +- reference/replication/sharding.md | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 0cb9a22d1..aa317964f 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -280,7 +280,7 @@ replication: - `hostname` — This instance's hostname within the cluster. Subject to the same bare hostname or IP literal requirement as [`node.hostname`](#node) — a URL, `host:port`, or non-string value fails startup. When both are set, `node.hostname` wins. - `url` — WebSocket URL peers use to connect to this instance -- `databases` — Databases to replicate; _Default_: `"*"` (all). Each entry supports `name` and `sharded` +- `databases` — Databases to replicate; _Default_: `"*"` (all). Each entry supports `name`, `replicateTo`, and `sharded` - `routes` — Peer nodes; URL strings or `{hostname, port, startTime, revokedCertificates}` objects - `port` — Replication port - `securePort` — Secure replication port; _Default_: `9933` (changed from `9925` in v4.5.0) diff --git a/reference/replication/sharding.md b/reference/replication/sharding.md index 1b3d0484c..90f5eb737 100644 --- a/reference/replication/sharding.md +++ b/reference/replication/sharding.md @@ -35,11 +35,12 @@ The simplest way to limit replication is to configure a per-database replication ```yaml replication: databases: + - system - name: data replicateTo: 2 ``` -This ensures each record in that database is stored on three nodes total (the node that first stored it, plus two others). +This ensures each record in `data` is stored on three nodes total (the node that first stored it, plus two others). Because setting `replication.databases` to a list replaces the `'*'` default (all databases), include `system` and every other database the cluster should continue replicating (see [Data Selection](./overview.md#data-selection)). ### Replication Control via REST Header