Repository navigation
ADR-0021: true in-place WRITE_MAP (spike) — DRAFT, do not merge - #88
Merged
Merged
Conversation
…, no heap staging, no commit write-back
Behind WRITE_MAP only, and only when the backing brokers map slices
(Backing::dirty_in_map — the real WriteMapBacking); the default heap-staged
path (TXN-45a) is byte-identical for every other backing, including the miri
test backing and the fault-injection backend.
- zerodb-io: MmapWritable::slice_mut, the brokered dirty-page &mut (one new
unsafe block, SAFETY contract stated); WriteMapBacking implements
Backing::{dirty_in_map, map_dirty_page}.
- zerodb-core: Backing gains the two brokered methods (safe defaults, no new
unsafe in core); DirtyStore grows a Slot::{Heap,Map} frame representation —
in-map frames are realized at their final file offset at allocation time
(COW copy, new tree page, overflow run), views stay tied to &self/&mut self
(TXN-39/41); commit C2 and spills skip in-map frames; unspill re-tracks in
place; the general split copies the in-map frame out to a heap scratch
(LMDB's own WRITEMAP split copy).
- Spilling under in-place WRITE_MAP degenerates to bookkeeping (TXN-68..72
observable contract unchanged; dirty_spill writemap suite still passes
unweakened).
- SPEC 04 §6.4: TXN-45b added (in-place realization, abort = don't advance
the meta, TXN-62 applied at store time as TXN-70 already does for spills);
C2 table row and §6.3a cross-reference amended.
- New battery crates/zerodb/tests/writemap_in_place.rs: differential vs the
default mode (splits, runs, put_reserved, cursor ops, nested reads),
TXN-45b abort semantics incl. reopen + space reuse, put_reserved in-map.
- commit_census example gains a 'writemap' toggle (fair A/B per ADR-0021).
Spike A/B (macOS arm64, NO_SYNC, 20k single-put commits): WRITE_MAP commit
24.0-34.9 us -> 6.3-8.5 us, put 5.6-8.3 us -> 2.2-2.6 us; default mode
unchanged. Gate: fmt/clippy/test (623 passed)/miri (199 passed)/fuzz-quick/
crash-test-quick (214 cycles, 17 writemap sigkill) all green.
Carries the accepted ADR-0021 (true in-place WRITE_MAP) and its spike-review punch-list onto the spike branch, which was based on main before the ADR landed.
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
⛔ Files ignored due to path filters (1)
📒 Files selected for processing (31)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…in zerodb-core::dirty (ADR-0021 B1, Quentin 2026-10-02) Adds zerodb-core::dirty to the unsafe allowlist for the WRITE_MAP in-place path only: calling zerodb-io's unsafe-fn map-slice broker to realize a dirty page in the writable map at a freshly-COW'd pgno. States the required SAFETY invariants (TXN-6 single writer, TXN-62 unreferenced target, one live &mut per region tied to &mut DirtyStore, whole-map &[u8] re-derived per spill) and requires the broker be an unsafe fn (resolves B1's policy half; the code change to unsafe fn + B2/B3/ B4 remain for the production pass).
…solved Opt-in behind WRITE_MAP; the default heap-staged path is untouched. - B1 (soundness): the map-slice broker is an unsafe fn end to end (MmapWritable::slice_mut -> Backing::map_dirty_page); the one sanctioned unsafe call lives in zerodb-core::dirty::map_mut with the policy's four invariants in its SAFETY block (clippy's mut_from_ref fires on unsafe fns too, so its allow stays as a documented false-positive suppression). - B2 + a stronger M1 finding: heap-staged paths re-derive the writer's whole-map &[u8] at every spill; miri then showed that under Stacked Borrows ANY copy of a stale view - even the field retag on commit(self) - is UB at in-place-written locations, so in-map txns hold no cached whole-map reference at all and borrow the view lazily per access (RwTxn::whole_map), like readers. TXN-71 amended. - B3: the fault backend forwards the broker and journals brokered regions (deduplicated; bytes resolved at every sync seal point and at capture; MS_ASYNC seals preserve per-commit versions for the ordered sub-model); image cuts open the real writable map for the WRITE_MAP modes; an fd data write under WRITE_MAP is a violation (vacuousness tripwire); smoke test pins non-vacuity; harness summary reports journaled regions. - B4: no new loom model needed (justified in the ADR - in-place introduces no new atomic/lock-free interaction; existing reader-table/nested/stamp models cover the publish edges); 180 s stress gains the writemap-in-place churn-writer variant; M1.13 adapter borrow audit written up in the ADR plus a miri battery for the lifetime-erased write cursor over the in-place realization (heed-zerodb). - M1: zerodb_io::testmap::TestWriteMap (test-backing feature, UnsafeCell<Box<[u8]>>) runs the whole brokered discipline under miri on plain heap memory; caught the B2 insufficiency on its first run. - M2: release-mode typed TXN-62 guard in RwTxn::allocate for in-map mode (pgno > committed_last_pg or reclaimed, each page of a run); failure errors the txn instead of clobbering committed data through the map. - M3: abort-after-spill, all three nested fan-outs and the put_reserved adversarial battery parameterized over WRITE_MAP (each asserts the mode engaged); the crash battery runs in-place via B3. - m1: SPEC 04 TXN-71 / par.6.1 C5a wording / TXN-45b amended; SPEC 06 REC-20 gains the in-map journaling amendment. Gate: fmt, clippy -D warnings, test --workspace (60 bins, 417/0), miri -p zerodb-core (201/0, incl. the new in-place batteries), miri -p heed-zerodb erased_cursor (2/0), fuzz-quick (no findings), crash-test-quick (235 cycles, 0 violations, 2591 in-place regions), loom (9/0, 1136 s), stress 180 s (3/0, in-place variant 1089 spilling commits vs 2660 reader double-walks). Perf: commit_census writemap NO_SYNC 20k medians - pre-spike heap-staged 45.9 us -> hardened in-place 11.3 us (~4x, spike-level win preserved; spike itself 10.6 us, within noise of hardened).
Owner
Author
Hardening pass complete + Graviton NVMe headlineProduction hardening (commit Graviton4 m8gd NVMe, YCSB, cap 2 GB, no-sync (medians of 2)
In-place WRITE_MAP makes ZeroDB 1.58× plain LMDB and closes the gap to LMDB-writemap to 0.61–0.65× (from ~0.44× heap-staged); write p50 drops ~3× (13.6→4.4µs). The residual to LMDB-writemap is the non-copy per-commit CPU (free-list save, Two residuals for maintainer ratification (not correctness gaps)
Still a draft / do-not-merge pending maintainer review of those two + the honest residual risks in 🤖 Generated with Claude Code |
…ration in zerodb-core::env + the clippy::mut_from_ref allows (Quentin 2026-10-02) Widens the WRITE_MAP in-place sanction to cover the Backing::map_dirty_page unsafe-fn declaration at its trait-definition site (zerodb-core::env; declaration-only allow(unsafe_code), trivially-safe default body, actual map unsafe stays in zerodb-io) and accepts the clippy::mut_from_ref allows on the broker's unsafe-fn surface (clippy fires on unsafe fn too; contracts inline).
qdequele
marked this pull request as ready for review
October 5, 2026 08:54
qdequele
added a commit
that referenced
this pull request
Oct 5, 2026
testmap.rs (Linux-only, so missed by the macOS clippy run) and writemap_in_place.rs arrived with #88; the 1.98 MSRV enables manual_is_multiple_of there too.
qdequele
added a commit
that referenced
this pull request
Oct 5, 2026
* Bump MSRV to 1.98, CI actions to latest, refresh lockfiles - MSRV 1.80 -> 1.98 (workspace, heed-shim, fuzz): the toolchain Meilisearch pins in its rust-toolchain.toml. A minor-release item under ADR-0013 rule 6; recorded in CHANGELOG. - The higher MSRV unlocks clippy lints the old one suppressed (manual_is_multiple_of, unnecessary_map_or -> is_none_or, chunks_exact_to_as_chunks); fixed across the workspace. The two PIL decoders now use as_chunks::<8>() and drop their try_into().expect(). - GitHub Actions: checkout v4 -> v7.0.1, upload-artifact v4 -> v7.0.1, download-artifact v4 -> v8.0.1, dtolnay/rust-toolchain master @ 2026-10-01; rust-cache already at v2.9.2. All still SHA-pinned. - cargo update on all three lockfiles (semver-compatible only). Every direct dependency was already on its latest major; heed / lmdb-master-sys / heed-traits / heed-types stay at the oracle pins. * Fix is_multiple_of lints in code merged from main testmap.rs (Linux-only, so missed by the macOS clippy run) and writemap_in_place.rs arrived with #88; the 1.98 MSRV enables manual_is_multiple_of there too.
qdequele
added a commit
that referenced
this pull request
Oct 5, 2026
…ught up to date (#91) * docs: sweep stale content after #88/#89/#90 Docs only, no code changes. Checked against main at d155fe6. - PERF-GAP-VS-LMDB: item statuses and the roadmap now match what landed (in-place WRITE_MAP, meta free-list annex, spilling, validation cache, forced inlining) and what was parked (lazy validation, O_DSYNC meta write). Drifted line refs are replaced with function names. The 1.8x commit figure is relabelled as the macOS run; on Linux, durable commits are at parity. - BENCH-MAP: benchmark cross-references updated after the delete_range, cursor and annex changes. - PLAN: Phase 3 status table; the 2.8, 3.1, 3.4 and 3.7 notes updated. - DECISIONS and ADR headers: implementation notes only, no decision changed. 0019's implementation was parked in PR #86. 0021 and 0022 merged as #88 and #89. - DIVERGENCES: D-004 no longer claims DUPSORT was implemented. D-014 and D-022 factual notes fixed. No status or approval changed. - CHANGELOG [Unreleased]: format v2 is a breaking change (v1 files are refused at open, migrate with dump then load), plus in-place WRITE_MAP, the annex and NO_READ_AHEAD. Notes that the v0.1.0 tag was never pushed. - README, COMPATIBILITY, TOOLS, CONSUMER-GATE, CONTRIBUTING, UPSTREAM-BUGS: format v2, the new options, release state, the CI description, mimalloc, and the memory behaviour of load and check. - SPEC: non-normative cross-references only. - ci.yml: header comment only. - PROGRESS: one appended line covering #88, #89, #90 and this sweep. * spec: bring SPEC 00–06 up to date with the engine as it evolved The maintainer authorized amending normative SPEC text to describe the engine as it is now. Only deliberate changes are folded in: accepted ADRs, approved divergences, and changes merged and kept. No code/spec disagreement turned out to be a correctness bug. All 160 rule IDs (TXN/GC/REC/BT/INV) are kept, none renumbered or retired. Each file carries a "Revised 2026-10-05" note. - 02 pages: FORMAT_VERSION 2 and v1 refused at open (ADR-0022), plus the annex offset constant. non_free_pages_size uses the GC-23 definition. The checksum field is reserved and written as 0. DUPSORT layout is parked. - 01 flags: NO_READ_AHEAD landed. WRITE_MAP is now in place (ADR-0021). fdatasync after msync runs on every platform. The 2.8a pins that were pending are adopted as spec text, with DUPSORT itself parked. - 00 API: WRITE_MAP in-place note. The rest of the surface re-verified, with no change needed. - 03 btree: point get through find_exact. The integer fast path is exactly memcmp order. Where dirty bytes live (heap staging, in-place WRITE_MAP, spill/unspill). Cursor-path retention covers deletes only. put_reserved uses a single descent. Allocation order now includes the annex. Same-size overwrite is scoped to inline values: the large-value same-size case was never built, tracked in #6. DUPSORT is marked parked. - 04 txn: snapshot fields include the annex. Frame pool realization note. - 05 gc: annex section marked ratified (PR #89). Drain representation note. - 06 recovery: format v2 at REC-1, v2 CRC offsets at REC-8 and REC-22. REC-12 states the shipped whole-map msync plus fdatasync, a superset of the ranged msync, which stays planned in #45; C3 then C4 then C5 order verified. REC-7 notes the parked O_DSYNC meta write. Also in PERF-GAP: the used-portion COW copy experiment was measured flat and reverted on 2026-10-02. Its record lived only in 279ceba on an unmerged branch. * Prepare the repo for public readers - Root: CLAUDE.md becomes AGENTS.md, rewritten as a public guide for human and AI contributors: project rules, unsafe policy, checks, repo map, style. Agent model names, milestone process and personal ratification notes are gone. PLAN.md, PROGRESS.md and .claude/ (agent definitions, slash commands) are removed from the tree; git history keeps them. /CLAUDE.md and /.claude/ are gitignored so local assistant configuration stays local. - Code comments: internal shorthand replaced with plain words in about 200 files: milestone codes, perf-inventory codes, "Phase N" roadmap labels, divergence IDs, and references to the removed files. Spec rule IDs (TXN-41, GC-16, ...) and ADR numbers are kept; they point to public docs. Every changed .rs line is a comment, except three user-visible strings cleaned the same way: the zerodb-tools usage text, the MDB_NOSUBDIR error, and the compacting-copy comparator error, which also lost a run of stray spaces. - Approval records: names and chat quotes become "maintainer, <date>" in DIVERGENCES, DECISIONS, ADRs, SPEC and code. Decisions and dates kept. - Docs: CONTRIBUTING points to AGENTS.md. References to removed files are rewritten; in historical records (ADRs, bench reports) they are plain text. SECURITY: write-transaction memory is bounded by spilling now, except a single large value. Checks (Rust 1.99, macOS aarch64): cargo fmt --check clean. cargo clippy --workspace --all-targets -D warnings clean, and the same for the x86_64-unknown-linux-gnu target on the engine crates. cargo test --workspace: 653 passed, 0 failed. cargo doc: the same 37 pre-existing warnings as main, none new. * Cut history, process notes and restatements from comments and docs Comments (about 140 files; comment lines 15,047 → 14,738): removed dated history ("since 2026-07-21", "amended …", "pre-fix"), review and process artifacts (review finding IDs, coverage-pass notes, "do not weaken (AGENTS.md rule 2)" boilerplate), bug-discovery stories (each regression test keeps one line on what it guards), and long verbatim mdb.c / lmdb.h quotes (now one sentence plus the reference). Kept every SAFETY comment and atomic Ordering justification, every invariant and reason a decision was made, LMDB-parity facts, spec rule IDs, ADR numbers and complete public API docs. Comments that contradicted the code are corrected: - heed-zerodb `Database::put_reserved` # Errors: a failing closure returns Io and keeps the entry; it does not return Encoding. - heed-zerodb txn module doc and heed_suite: RwTxn is Send (the writer lock is thread-agnostic, TXN-6); they described an old !Send design. - dirty.rs SPARE_CAP: ZeroDB does spill (TXN-68). meta.rs: the CRC covers [0,172) plus the annex ids. lib.rs and page/mod.rs: page::raw is not the only unsafe module. - Three doc comments were attached to the wrong item and are moved. One dead intra-doc link is removed. The oracle gc_churn band/page size and the multi_db coverage list are fixed, along with a few other stale harness descriptions. Only comments changed in .rs files (checked mechanically; two trailing comments after an unchanged `},`), plus one fix: examples/decode_artifact.rs now splits off the engine-mode byte the way the diff_ops fuzz target does, and prints the mode. It used to decode the mode byte as an op and print a shifted sequence. Docs: - PERF-GAP-VS-LMDB.md goes from 1,160 to 331 lines. It now shows where ZeroDB stands per area, which technique closed each gap, what was tried and dropped, and what remains with issue links. Every number cites its report in benches/results. A lookup table maps the old item codes still cited by ADRs, SPEC and the perf ledger. - BENCH-MAP.md goes from 315 to 270 lines: the rung map is kept, history removed. - DECISIONS.md and DIVERGENCES.md: index rows cut to one short line each. All 22 divergences are kept, with no status or approval changed. - benches/results: nothing deleted; every report is either the latest of its kind or referenced. Checks (Rust 1.99): fmt clean. clippy -D warnings clean on macOS and on the x86_64-unknown-linux-gnu engine crates. cargo test --workspace: 653 passed, 0 failed. cargo doc: same 37 pre-existing warnings as main. No broken relative links in any Markdown file. * Address review: unreleased 0.1.0 notes, WRITE_MAP durability row - CHANGELOG [0.1.0]: say plainly that it was never released. Drop the claims that crates were published and binaries attached, drop the tag from the patch snippet, and replace the links to the missing v0.1.0 tag. The `## [0.1.0]` heading stays, since the release workflow matches it. - SPEC 01 Table 1, MDB_WRITEMAP: commit flushes the whole map with msync, and a synchronous flush also fdatasyncs on every platform (REC-12). This row now matches §S7 and the landed-flags table.
qdequele
added a commit
that referenced
this pull request
Oct 5, 2026
* docs: sweep stale content after #88/#89/#90 Docs only, no code changes. Checked against main at d155fe6. - PERF-GAP-VS-LMDB: item statuses and the roadmap now match what landed (in-place WRITE_MAP, meta free-list annex, spilling, validation cache, forced inlining) and what was parked (lazy validation, O_DSYNC meta write). Drifted line refs are replaced with function names. The 1.8x commit figure is relabelled as the macOS run; on Linux, durable commits are at parity. - BENCH-MAP: benchmark cross-references updated after the delete_range, cursor and annex changes. - PLAN: Phase 3 status table; the 2.8, 3.1, 3.4 and 3.7 notes updated. - DECISIONS and ADR headers: implementation notes only, no decision changed. 0019's implementation was parked in PR #86. 0021 and 0022 merged as #88 and #89. - DIVERGENCES: D-004 no longer claims DUPSORT was implemented. D-014 and D-022 factual notes fixed. No status or approval changed. - CHANGELOG [Unreleased]: format v2 is a breaking change (v1 files are refused at open, migrate with dump then load), plus in-place WRITE_MAP, the annex and NO_READ_AHEAD. Notes that the v0.1.0 tag was never pushed. - README, COMPATIBILITY, TOOLS, CONSUMER-GATE, CONTRIBUTING, UPSTREAM-BUGS: format v2, the new options, release state, the CI description, mimalloc, and the memory behaviour of load and check. - SPEC: non-normative cross-references only. - ci.yml: header comment only. - PROGRESS: one appended line covering #88, #89, #90 and this sweep. * spec: bring SPEC 00–06 up to date with the engine as it evolved The maintainer authorized amending normative SPEC text to describe the engine as it is now. Only deliberate changes are folded in: accepted ADRs, approved divergences, and changes merged and kept. No code/spec disagreement turned out to be a correctness bug. All 160 rule IDs (TXN/GC/REC/BT/INV) are kept, none renumbered or retired. Each file carries a "Revised 2026-10-05" note. - 02 pages: FORMAT_VERSION 2 and v1 refused at open (ADR-0022), plus the annex offset constant. non_free_pages_size uses the GC-23 definition. The checksum field is reserved and written as 0. DUPSORT layout is parked. - 01 flags: NO_READ_AHEAD landed. WRITE_MAP is now in place (ADR-0021). fdatasync after msync runs on every platform. The 2.8a pins that were pending are adopted as spec text, with DUPSORT itself parked. - 00 API: WRITE_MAP in-place note. The rest of the surface re-verified, with no change needed. - 03 btree: point get through find_exact. The integer fast path is exactly memcmp order. Where dirty bytes live (heap staging, in-place WRITE_MAP, spill/unspill). Cursor-path retention covers deletes only. put_reserved uses a single descent. Allocation order now includes the annex. Same-size overwrite is scoped to inline values: the large-value same-size case was never built, tracked in #6. DUPSORT is marked parked. - 04 txn: snapshot fields include the annex. Frame pool realization note. - 05 gc: annex section marked ratified (PR #89). Drain representation note. - 06 recovery: format v2 at REC-1, v2 CRC offsets at REC-8 and REC-22. REC-12 states the shipped whole-map msync plus fdatasync, a superset of the ranged msync, which stays planned in #45; C3 then C4 then C5 order verified. REC-7 notes the parked O_DSYNC meta write. Also in PERF-GAP: the used-portion COW copy experiment was measured flat and reverted on 2026-10-02. Its record lived only in 279ceba on an unmerged branch. * Prepare the repo for public readers - Root: CLAUDE.md becomes AGENTS.md, rewritten as a public guide for human and AI contributors: project rules, unsafe policy, checks, repo map, style. Agent model names, milestone process and personal ratification notes are gone. PLAN.md, PROGRESS.md and .claude/ (agent definitions, slash commands) are removed from the tree; git history keeps them. /CLAUDE.md and /.claude/ are gitignored so local assistant configuration stays local. - Code comments: internal shorthand replaced with plain words in about 200 files: milestone codes, perf-inventory codes, "Phase N" roadmap labels, divergence IDs, and references to the removed files. Spec rule IDs (TXN-41, GC-16, ...) and ADR numbers are kept; they point to public docs. Every changed .rs line is a comment, except three user-visible strings cleaned the same way: the zerodb-tools usage text, the MDB_NOSUBDIR error, and the compacting-copy comparator error, which also lost a run of stray spaces. - Approval records: names and chat quotes become "maintainer, <date>" in DIVERGENCES, DECISIONS, ADRs, SPEC and code. Decisions and dates kept. - Docs: CONTRIBUTING points to AGENTS.md. References to removed files are rewritten; in historical records (ADRs, bench reports) they are plain text. SECURITY: write-transaction memory is bounded by spilling now, except a single large value. Checks (Rust 1.99, macOS aarch64): cargo fmt --check clean. cargo clippy --workspace --all-targets -D warnings clean, and the same for the x86_64-unknown-linux-gnu target on the engine crates. cargo test --workspace: 653 passed, 0 failed. cargo doc: the same 37 pre-existing warnings as main, none new. * Cut history, process notes and restatements from comments and docs Comments (about 140 files; comment lines 15,047 → 14,738): removed dated history ("since 2026-07-21", "amended …", "pre-fix"), review and process artifacts (review finding IDs, coverage-pass notes, "do not weaken (AGENTS.md rule 2)" boilerplate), bug-discovery stories (each regression test keeps one line on what it guards), and long verbatim mdb.c / lmdb.h quotes (now one sentence plus the reference). Kept every SAFETY comment and atomic Ordering justification, every invariant and reason a decision was made, LMDB-parity facts, spec rule IDs, ADR numbers and complete public API docs. Comments that contradicted the code are corrected: - heed-zerodb `Database::put_reserved` # Errors: a failing closure returns Io and keeps the entry; it does not return Encoding. - heed-zerodb txn module doc and heed_suite: RwTxn is Send (the writer lock is thread-agnostic, TXN-6); they described an old !Send design. - dirty.rs SPARE_CAP: ZeroDB does spill (TXN-68). meta.rs: the CRC covers [0,172) plus the annex ids. lib.rs and page/mod.rs: page::raw is not the only unsafe module. - Three doc comments were attached to the wrong item and are moved. One dead intra-doc link is removed. The oracle gc_churn band/page size and the multi_db coverage list are fixed, along with a few other stale harness descriptions. Only comments changed in .rs files (checked mechanically; two trailing comments after an unchanged `},`), plus one fix: examples/decode_artifact.rs now splits off the engine-mode byte the way the diff_ops fuzz target does, and prints the mode. It used to decode the mode byte as an op and print a shifted sequence. Docs: - PERF-GAP-VS-LMDB.md goes from 1,160 to 331 lines. It now shows where ZeroDB stands per area, which technique closed each gap, what was tried and dropped, and what remains with issue links. Every number cites its report in benches/results. A lookup table maps the old item codes still cited by ADRs, SPEC and the perf ledger. - BENCH-MAP.md goes from 315 to 270 lines: the rung map is kept, history removed. - DECISIONS.md and DIVERGENCES.md: index rows cut to one short line each. All 22 divergences are kept, with no status or approval changed. - benches/results: nothing deleted; every report is either the latest of its kind or referenced. Checks (Rust 1.99): fmt clean. clippy -D warnings clean on macOS and on the x86_64-unknown-linux-gnu engine crates. cargo test --workspace: 653 passed, 0 failed. cargo doc: same 37 pre-existing warnings as main. No broken relative links in any Markdown file. * Address review: unreleased 0.1.0 notes, WRITE_MAP durability row - CHANGELOG [0.1.0]: say plainly that it was never released. Drop the claims that crates were published and binaries attached, drop the tag from the patch snippet, and replace the links to the missing v0.1.0 tag. The `## [0.1.0]` heading stays, since the release workflow matches it. - SPEC 01 Table 1, MDB_WRITEMAP: commit flushes the whole map with msync, and a synchronous flush also fdatasyncs on every platform (REC-12). This row now matches §S7 and the landed-flags table. * Release 0.2.0 - Version 0.2.0 for zerodb-core, zerodb-io, zerodb, zerodb-tools and zerodb-oracle, with the exact internal pins updated. Lockfiles refreshed. heed-zerodb and heed-shim stay at 0.22.1, the heed line they mirror. - CHANGELOG [0.2.0] is self-contained, since 0.1.0 was prepared but never released: what ZeroDB is, how to install it, features, performance with sources, verification (including that the consumer test suites last ran 2026-09-09), requirements, and known gaps. The never-released [0.1.0] section is folded in; [Unreleased] is reopened. - README install snippet (crates.io and the v0.2.0 tag) and status, TOOLS install instructions, SECURITY's unsafe inventory (adds the in-place WRITE_MAP path), and a stale "since 0.1.0" in COMPATIBILITY.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADR-0021: true in-place WRITE_MAP — DRAFT, do not merge
Opt-in behind the existing
WRITE_MAPflag; the default (heap-staged) path isuntouched (both the flag and the brokering backing are required to engage
it). Under
WRITE_MAP, dirty pages are written straight into the mapped file attheir freshly-COW'd page numbers — no heap dirty frame, no commit write-back —
as LMDB's
MDB_WRITEMAPdoes.Why
On the current tree, ADR-0017 spilling already closed the memory/page-cache gap
(ZeroDB's anon memory ≈ LMDB, fewer disk reads). The residual no-sync write gap
is per-operation commit CPU, and the biggest structural piece of it is that
ZeroDB's
WRITE_MAPstill heap-staged then memcpy'd into the map at commit. Afair comparison (both engines
WRITE_MAPon) put ZeroDB-writemap at only0.42–0.53× of LMDB-writemap. This PR removes the heap stage + the commit
write-back.
Graviton4 m8gd NVMe — YCSB, cap 2 GB, no-sync (medians of 2)
In-place WRITE_MAP makes ZeroDB 1.58× plain LMDB and closes the gap to
LMDB-writemap to 0.61–0.65× (from ~0.44× heap-staged); write p50 drops ~3×
(13.6 → 4.4 µs). The remaining 0.61–0.65× is LMDB-writemap's 1.2 µs commit vs
ZeroDB's 4.4 µs — the other per-commit CPU (free-list save,
allocate, metaencode), a separate effort that also helps the default path. (x86 bench-server
reproduces the shape: 1.19× plain LMDB, 0.65× LMDB-writemap.
commit_census:~4.1× per commit vs pre-spike heap-staged.)
Gate (final tree, commit
0972be9)clippy
-D warningsclean ·cargo test --workspace641/0 ·miri -p zerodb-core201/0 (+ adapter cursor miri 2/0) · fuzz-quick clean ·crash-test-quick now exercises in-place (2,591 journaled map regions, with a
vacuousness tripwire so a WRITE_MAP crash cycle cannot silently run the heap
path) · loom 9/0 · stress 180s 3/0 incl. a WRITE_MAP in-place writer
(1,089 spilled commits).
What the production pass resolved (spec-review punch-list)
unsafe fnend-to-end — no safe code mints&mut [u8]from&self; sole sanctioned call indirty::map_mut(CLAUDE.md unsafe policyamended to allow it).
UnsafeCellmiri test backing (M1) caught that per-spillre-derivation was insufficient under Stacked Borrows → in-place txns now hold
no cached whole-map reference and borrow the view lazily, like
RoTxn.twinned under WRITE_MAP (no test weakened); m1 SPEC TXN-71/§6.1/C5a/TXN-45b
Maintainer calls — resolved
#[allow(clippy::mut_from_ref)]on the broker's
unsafe fnsurface (clippy 1.97 fires it onunsafe fntoo;each carries the exclusivity contract inline).
Backing::map_dirty_page's declaration-only#[allow(unsafe_code)]at its trait-definition site inzerodb-core::env(actual map
unsafestays inzerodb-io).Documented residual risk (inherent, not a blocker)
Real-mmap cross-thread aliasing rests on the documented SAFETY argument +
loom/stress/crash evidence, not a machine check — as for every mmap engine. miri
covers the single-threaded brokered discipline on heap memory. See
docs/adr/0021-writemap-in-place.md. Ready for maintainer review / merge.(The separately-approved B12 "used-portion COW copy" lever was A/B-flat and is
not in this PR — recorded as a negative result in PERF-GAP B12.)
🤖 Generated with Claude Code