Repository navigation
feat(core): stage a captured inventory one page at a time (#445) - #1010
Conversation
The inventory store and capture_inventory hold every sealed page in memory, and the format allows a million entries. A page cannot be written to its final place as it is sealed: that place is keyed by the page-set digest, which needs every page's envelope digest, and sealing uses a fresh nonce, so a second pass would bind different bytes. StreamedCapture takes the entries page by page, assigns the page index and generation, checks that they are valid and strictly ascending across pages, seals each page once and stages its envelope in a private inventory/pending-* directory, keeping one page and one reference per page in memory. The total is announced up front because the inventory digest commits to it first. finish builds the successor with the same function capture_inventory ends in, and load_staged_page confirms a staged page against its reference before it is opened. Staged files are never authority. Promotion under the root lock is the next stage.
🤖 CodeAnt AI — Review Status
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
Reviewer's GuideImplements stage one of streaming inventory capture: callers announce the total, push validated pages that are sealed once and durably stored in randomized inert pending directories, then finish to produce the same validated successor manifest as the existing in-memory capture; staged pages can be authenticated on readback, while promotion and commit remain deferred to stage two. Sequence diagram for staged streaming inventory capturesequenceDiagram
participant Caller
participant StreamedCapture
participant InventoryVerifier
participant PendingStore
participant SuccessorBuilder
Caller->>StreamedCapture: begin(ctx, committed, fence, live, entry_count)
StreamedCapture->>StreamedCapture: assert_page_promote_authority()
StreamedCapture->>StreamedCapture: assert_root_named_manifest()
StreamedCapture->>StreamedCapture: assert_inventory_open()
StreamedCapture-->>Caller: StreamedCapture
loop Each page
Caller->>StreamedCapture: push_page(ctx, entries)
StreamedCapture->>InventoryVerifier: absorb_page(page)
StreamedCapture->>PendingStore: seal_inventory_pages()
StreamedCapture->>PendingStore: store_page_at()
PendingStore-->>StreamedCapture: synced envelope
StreamedCapture-->>Caller: StreamedCapture
end
Caller->>StreamedCapture: finish()
StreamedCapture->>InventoryVerifier: finish_digest()
StreamedCapture->>SuccessorBuilder: capture_successor()
SuccessorBuilder-->>Caller: StagedCapture
Sequence diagram for authenticated staged page loadingsequenceDiagram
participant Caller
participant PendingStore
participant PageReader
Caller->>PendingStore: load_staged_page(ctx, staged, page_index)
PendingStore->>PendingStore: read_envelope()
PendingStore->>PendingStore: content_digest(envelope)
alt reference digest matches
PendingStore->>PageReader: open_stored_page()
PageReader-->>Caller: StagedPage
else digest mismatch or invalid reference
PendingStore-->>Caller: JournalError
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
🏁 CodeAnt Quality Gate ResultsCommit: ✅ Overall Status: PASSEDQuality Gate Details
|
codeant-ai
Bot
added
the
size:XL
![codescene-access[bot]](https://avatars.githubusercontent.com/in/3755875?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
[check-pr-size] PR size is over the hard tier (normal profile): 16 files, 1718 meaningful lines, 7 commits — limit ≤20 files / ≤1200 lines / ≤10 commits. Consider splitting into smaller, independently reviewable PRs. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (5)
Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour. 📝 WalkthroughWalkthroughThe journal adds streaming inventory capture. It stages sealed pages in a private pending directory, validates the announced entry count and page ordering, builds a successor manifest, and verifies staged pages against their references. Promotion and composed commit remain outside this change. ChangesStreaming inventory capture
Priority: ➖ Normal Merge Risk: ⚪ Minimal · up to No actionable merge-blocking issue is established; the PR is mergeable after normal checks.
Comment |
![codeant-ai[bot]](https://avatars.githubusercontent.com/in/646884?s=60&v=4){kind=small}
CodeAnt Nitpicks1 code suggestion1. A same-path, same-key context can use a different
|
CodeAnt PR Risk: Medium Risk
Assessed commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f610b6475d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…quest struct (#445) CodeScene flagged two things on the first head. begin took five arguments; it now takes the context and a CaptureStart request (the manifest the root binding names, the fence, the binding and the announced total). The streaming tests made the inventory store test lose its cohesion; they are a file of their own, over the generic journal fixtures (TempDir, ObservedFs, the committed journal) that moved out of the inventory fixtures into support/journal_fixture.rs, so a test that needs only a committed journal does not carry the page fixtures and no lint suppression is needed. The two existing tests that used the inventory fixtures include both modules and are otherwise unchanged.
|
@codex review |
|
@CodeAnt-AI review |
codeant-ai
Bot
added
size:XXL
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…up after a failure (#445) Review of the first heads found four real gaps in the staging stage. A push took whichever context it was given, so a context with another key staged pages the routed journal key cannot open, and another journal directory moved the directory-sync boundary. The capture now remembers its journal directory and the binding it started from, and every push first requires the same directory and a key that opens the root-named manifest. A push or a finish that failed after earlier pages were written left them behind. A failure the caller can handle now removes the staged page files on the way out, best effort and including the page in flight, finish takes the context for that, and StagedCapture::discard does the same for a finished capture that will not be promoted. Empty directories and the files of a killed attempt stay, because the file system abstraction cannot remove a directory; reclaiming them is a recorded follow-up. A missing staged page was reported as RecoveryRequired by the reader shared with authoritative pages. A staged page is a candidate, so a missing one is now Corrupt, never a recovery state of the journal.
|
@codex review |
|
@CodeAnt-AI review |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
@codex review |
|
@CodeAnt-AI review |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 07bdc86b0b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
#445) The cleanup of a failed, short or discarded capture removed every completed page unconditionally, so a page that had been replaced after it was staged would have been deleted with the rest. Every removal now checks the file against the digest the capture recorded for it, the same rule the page in flight already followed, so only bytes this capture wrote are ever deleted.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
@codex review |
|
@CodeAnt-AI review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 27f0528e19
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…tures uniquely owned (#445) A push that failed because its page slot was already taken left the staging link that the failed promotion reports; the cleanup now removes it under the same digest rule as the staged pages. StagedCapture is a handle to files on disk, so it is no longer Clone: discarding one copy would have invalidated the other.
|
@codex review |
|
@CodeAnt-AI review |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2dac290c61
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…create test directories exclusively (#445) A promotion that succeeds but cannot unlink its staging link reports success with the link still present, and the cleanup only knew the names of the pages. The capture now stages under its own random identity instead of the caller's operation id, so the name of every staging link follows from the capture itself and the cleanup removes it, under the same digest rule as the pages. The test directory fixture moved out of the inventory fixtures with a predictable name that create_dir_all accepted when it already existed. It now creates the directory exclusively and skips a name that is taken, so Drop only removes what it created.
There was a problem hiding this comment.
Gates Passed
See analysis details in CodeScene
Quality Gate Profile: The Bare Minimum
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.
|
@codex review |
|
@CodeAnt-AI review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 63e8236c75
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Final disposition census — head
|
| Wave | Finding | Disposition |
|---|---|---|
| 1 CodeScene | begin had five arguments |
VALID_AND_FIXED — a CaptureStart request |
| 1 CodeScene | the streaming tests made gate4d_inventory_store_test lose cohesion (10.00 → 8.82) |
VALID_AND_FIXED — a test file of their own over support/journal_fixture.rs, split out of the inventory fixtures; no suppression |
| 2 CodeAnt + Codex | a push seals with whatever context it is given | VALID_AND_FIXED — the capture is bound to its journal directory and to the key that opens the root-named manifest |
| 2 CodeAnt | abandoned captures leak disk | VALID_AND_FIXED for every failure the caller can handle (failed push, failed or short finish, discard); VALID_AND_DEFERRED_WITH_ACCEPTANCE_CRITERION for a killed attempt, a dropped finished capture and the empty page directories (no directory removal in DurableFs) — reclaiming is recorded on #359 |
| 2 Codex | a missing staged page was RecoveryRequired |
VALID_AND_FIXED — Corrupt, never a recovery state of the journal |
| 3 CodeAnt | the manifest was re-read and re-opened on every push | VALID_AND_FIXED — begin keeps the authenticated bytes, a push reads nothing (a test counts the reads) |
| 3 Codex | cleanup removed the slot in flight unconditionally | VALID_AND_FIXED — removed only if it holds exactly the staged bytes |
| 4 Codex | completed pages removed unconditionally | VALID_AND_FIXED — one rule for every removal: the file must still hash to the digest the capture recorded |
| 4 Codex | the staging link of a failed promotion is left behind | VALID_AND_FIXED |
| 4 Codex | verify and delete are not atomic | INVALID_WITH_EVIDENCE — DurableFs has no unlink by handle; the directory is private, randomly named and never authority, so a racing writer there already has write access to the authoritative journal; stated as a limit in the evidence |
| 4 Codex | StagedCapture is Clone |
VALID_AND_FIXED — not Clone |
| 5 CodeAnt | TempDir accepted a predictable existing path |
VALID_AND_FIXED — exclusive creation; the code predates the PR (moved) |
| 5 Codex | the staging link of a successful promotion is lost to the cleanup | VALID_AND_FIXED — the capture stages under its own identity so every link name is known |
| 6 Codex (P3) | the evidence says eighteen tests, the file has nineteen | VALID_AND_DEFERRED_WITH_ACCEPTANCE_CRITERION — stage two rewrites that paragraph and stops quoting a count |
| CodeAnt nitpick | two different DurableFs stores behind one path and key |
INVALID_WITH_EVIDENCE — a store is identified by its path; two backing stores for one journal directory are outside the contract of every journal operation, each of which takes its file system per call |
| CodeAnt PR Risk | abandoned captures leave pending files | VALID_AND_DEFERRED_WITH_ACCEPTANCE_CRITERION — the reclamation criterion above |
Differences from the admission, disclosed in the PR and the evidence: the total is announced up front (the inventory digest commits to it first); begin takes a journal context carrying the key and the key-routing wrapper arrives with stage two; peak memory is bounded by construction and not measured by a counting file system; the tests are a file of their own.
Proof: 19 tests in gate4d_stream_capture_test, mutation-checked for every behaviour above (digest absorption, the root-named, open-inventory and context checks, the pending- name, the caller's operation id, the page directory, the digest check on read, each cleanup site, the slot in flight, the foreign file, the replaced page, both staging links, the manifest read per push, and the missing-page class). Whole crate suite (50 passing results), clippy -D warnings, fmt --check, docs:check.
Scope: staging only. No promotion, no composed commit, no C2, no graphify. PRODUCTION_AUTHORITY_SWITCH_ALLOWED = NO.
1 participant
User description
What
Stage one of the streaming capture: an inventory built and staged one page at a time, so a very large inventory no longer has to be resident as entries plus an equally large sealed copy (the format allows a million entries of up to a kilobyte).
A page cannot be written to its final place as it is sealed: that place is keyed by the page-set digest, which needs every page's envelope digest, and sealing uses a fresh nonce, so a second pass would bind different bytes. So each envelope is kept on disk, in a private pending directory with the layout of a digest directory (
inventory/pending-<revision>-<random>/page-<i>/generation-<g>.wsr1), until the digest is known.StreamedCapture::beginDISCOVER/ADMIT, not final, nothing converted); the announced total is within the bound. Nothing is createdpush_pagefinishcapture_inventoryends in; a failure removes the staged pagesStagedCapture::discardload_staged_pageCorrupt, neverRecoveryRequired(a staged page is a candidate, not authority); the envelope hashes to the reference before it is opened; opens only as this operation's page, epoch compared before the key is usedStaged files are never authority: the manifest does not name them, no reader looks for them. A failure the caller can handle removes the pages staged so far (best effort, including the page in flight), and
discarddoes the same for a finished capture; the files of a killed attempt and the empty page directories stay as inert residue, because the file system abstraction has no directory removal, and reclaiming them is a recorded follow-up.capture_inventoryis split into the page-dependent part and the successor build (capture_successor) with no behaviour change;store_pageis parameterised by directory.Differences from the admission (disclosed)
begintakes the total up front and the caller counts before it streams. The admission did not say so.begintakes a journal context that carries the key and aCaptureStartrequest (the manifest the root binding names, the fence, the binding and the announced total); the authority-level wrapper that routes the key from the root arrives with stage two, because the routed key has to be resolved again at promotion anyway.StreamedCapturehas no field that can hold a page or an envelope and a push drops both before it returns.gate4d_stream_capture_test, over the generic journal fixtures (TempDir,ObservedFs, the committed journal) that moved out ofsupport/inventory.rsintosupport/journal_fixture.rs. My first head put them into the store test, which made that file lose its cohesion (CodeScene); a separate file over the whole inventory fixtures would have needed a dead-code suppression, so the fixtures were split instead. The two existing tests that use the inventory fixtures include both modules and are otherwise unchanged.Decisions flagged on the admission: pending directory plus re-promote instead of a locator or nonce change; reclaiming abandoned pending directories is a follow-up criterion; the builder assigns indexes and generations; the in-memory API stays.
Boundary matrix
inventory/pending-*directory; additive API, no caller yetbegin) or before the next page is stagedNot in this PR
Stage two (promotion of the staged pages under the root lock and the journal mutex, re-reading and re-verifying each page, the key-routing wrapper and the composed commit), reclaiming abandoned pending directories, the C2 driver, inheriting unchanged pages, the write barrier. No production authority switch.
Verification
Local: fifteen tests in
gate4d_stream_capture_test— the streamed successor equals whatcapture_inventorybuilds from the staged pages (one entry, uneven and even splits, one page, the final capture inADMIT, a full page plus one with 4097 entries); an empty inventory stages nothing; the staged files are exactly the pending layout under apending-name; the staged pages are accepted by the existing store; every way a capture cannot start is refused with nothing created; out-of-order pages, a total exceeded or not reached and an empty page; a failure partway removes the pages it staged, also when the failing page was already promoted, a capture that ends short and a discarded capture leave no page file, and a failed removal is not an error; a push under another key or into a journal directory holding an identical copy of the root-named generation is refused before anything is staged; a missing staged page isCorrupt; two attempts under one operation id never share a directory; a changed, swapped, out-of-range or wrongly keyed staged page is refused. Mutation-checked: no digest absorption, no root-named check, no open-inventory check, nopending-prefix, a pending directory named by the caller's operation id, a shifted page directory and no digest check on read each fail the test that owns them. The whole crate suite (49 passing results),cargo clippy --all-targets -D warnings,cargo fmt --check,pnpm run docs:check.Part of #359 and #445 (Linear QNB-11). Admission: #359 issuecomment-6062112221.
PRODUCTION_AUTHORITY_SWITCH_ALLOWED = NO.Summary by Sourcery
Stage captured inventories incrementally through private pending pages while preserving existing manifest construction and authority boundaries.
New Features:
Bug Fixes:
Enhancements:
Documentation:
Tests:
Chores:
Summary by CodeRabbit
CodeAnt-AI Description
Stage large inventory captures one page at a time
What Changed
Impact
✅ Lower memory for large inventory captures✅ Fewer abandoned page files after handled failures✅ Changed or missing staged pages rejected before use💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.