docs: clarify RocksDB no-fsync durability and LMDB-only writeAsync

[kriszyp]

⊙ Problem

Several v5 reference pages state or imply that Harper calls fsync on every commit on RocksDB (the default storage engine), that RocksDB's WAL persists committed writes, and that storage.writeAsync disables fsync on RocksDB. In reality, Harper disables RocksDB's WAL for table stores. It writes commits to its own transaction log in the OS page cache, without calling fsync. storage.writeAsync applies only to LMDB.

🤔 Should the docs state plainly that RocksDB commits are not fsynced by default and rely on replication confirmation for host-failure durability?

Yes, as specified in HarperFast/harper#3243 (durability-via-replication-position). Documenting the actual commit path stops operators from assuming single-node disk-sync durability or setting storage.writeAsync on RocksDB. It directs multi-node Harper Pro deployments to peer replication confirmation (replicatedConfirmation / X-Replicate-To: *;confirm=<k>).

💡 Solution

• Add durability intro paragraphs under ## Durability vs. Throughput in storage-tuning.md covering what an acknowledged RocksDB write survives (process crashes via transaction-log replay) and does not (host/OS/power failure before OS flush), and how peer replication confirmation provides cross-host durability.
• Remove the RocksDB WAL claim in storage-algorithm.md and mark storage.writeAsync as LMDB-only (<EngineBadge engines="LMDB" />) in storage-tuning.md and options.md.
• Align logging.auditLog default (true) and replication.databases[].replicateTo in options.md and sharding.md, and update the Storage Algorithm card summary in overview.md.

🔧 Changes

• reference/database/storage-tuning.md — ⚠️ Look hardest: adds durability intro paragraphs under ## Durability vs. Throughput covering the RocksDB transaction log (database/<db>/transaction_logs/), OS page-cache writeback, and replicatedConfirmation / X-Replicate-To: *;confirm=<k> failure-domain guidance, adds <EngineBadge engines="LMDB" /> on storage.writeAsync, and updates the write-heavy and durability-critical workload recipes.
• reference/database/storage-algorithm.md — qualifies the opening ACID statement, updates the Durability bullet, and replaces the RocksDB WAL bullet with transaction-log replay in the OS page cache when transaction logging is enabled (logging.auditLog: true and @table(audit: true)).
• reference/configuration/options.md — fixes logging.auditLog default from false to true and its reference table entry, documents replicateTo under replication.databases entries, and marks storage.writeAsync as <EngineBadge engines="LMDB" />.
• reference/replication/sharding.md — updates the replicateTo prose and replication.databases config example (with system included alongside data), and uses X-Replicate-To: *;confirm=1 in the REST confirmation example.
• reference/database/overview.md — updates the Storage Algorithm card summary from LMDB-only wording to "RocksDB (or legacy LMDB)".

🤔 Should we name the on-disk database/<db>/transaction_logs/ path and X-Replicate-To: *;confirm=<k> syntax (expose-txnlog-path, document-star-header)?

Yes. Naming database/<db>/transaction_logs/ shows the concrete directory operators see under the root path rather than an abstract reference to the transaction log. Documenting X-Replicate-To: *;confirm=<k> (server/REST.ts:278) lets REST callers request confirmation without pinning residency to a numeric replicateTo count.

🤔 Should replicateTo configuration examples use replication.databases[].replicateTo (docs-follow-code-for-replicateto)?

Yes. Harper core reads replicateTo from each replication.databases entry (resources/Table.ts:1496-1503). The example includes system because overriding the default '*' allowlist would otherwise silently turn off system database replication.

✅ Verification

• Not testable end-to-end at runtime (documentation-only changes across five Markdown reference pages).
• Executed npm run format:write and npm run format:check (clean across the repository).
• Executed npm run build (clean production Docusaurus build across all 410 docs and versioned v4 docs with zero broken links or anchors).

Refs HarperFast/harper#3243

_{🤖 Generated by Google Gemini; posted via @kriszyp.}

Related PRs: #723 overlaps (both edit reference/replication/sharding.md REST header section), 2 others independent
Complexity: medium

Origin — the dispatch brief this PR was written from

Correct Harper's v5 documentation so that it states plainly that Harper does NOT fsync commits on the RocksDB storage engine (the default), does not plan to by default, and that durability against machine/OS/power failure comes from replication confirmation, not disk sync. Today several pages say or imply that commits are fsynced by default and that storage.writeAsync: true is what turns fsync off. That is wrong for RocksDB.

Acceptance

• No v5 doc page states or implies that Harper fsyncs commits by default on RocksDB, or that writeAsync disables fsync on RocksDB.
• storage-tuning.md has a clear durability section: what an acknowledged write survives (process crash) and does not (OS/power/host failure before OS flush), and that replication confirmation (X-Replicate-To confirm / replicatedConfirmation) is the mechanism for durability across failures, linking reference/replication/sharding.md.
• writeAsync is documented as LMDB-only in options.md and storage-tuning.md.
• storage-algorithm.md no longer claims RocksDB WAL persistence.
• Do not document any opt-in fsync setting (none exists). Do not edit v4 versioned docs or release notes.
• Docs build/lint passes. PR body references harper#3243 (Refs, not Fixes).

_{Dispatch: task docs-no-fsync-durability · queued by kriszyp-session · ran by gemini/gemini-4-argon-high/low · worker kzyp-xps-1}

_{Review-Coverage: authored=gemini; ran=claude; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse,muse; rounds=5; full=1 @ 8996b50}

_{Review-Attention: skim ~2m (decisions: durability-via-replication-position, expose-txnlog-path, document-star-header, docs-follow-code-for-replicateto) @ 8996b50}

[gemini-code-assist]

Code Review

This pull request updates the documentation to reflect the transition to RocksDB as the default storage engine, detailing its durability model, transaction log recovery mechanism, and replication confirmation features. It also clarifies that the storage.writeAsync option is legacy and only applies to LMDB, and updates configuration examples for sharding and replication counts. There are no review comments, so I have no feedback to provide.

[github-actions]

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-733

This preview will update automatically when you push new commits.