feat: slim output and bump geth - #8
Open
gacevicljubisa wants to merge 8 commits into
Open
Conversation
The merge in 7f43ae5 reverted the --slim default from true (as shipped in 7290a12) back to false. Left as-is, this branch would ship slim mode off and the next batch-archive refresh would produce a full-shape snapshot regardless. The README flag dump was regenerated from the reverted code, so it also contradicted its own feature bullet. Reword the flag help now that the flag defaults on: pflag binds booleans as --slim=false, since a bare --slim sets it true, so state that form explicitly rather than describing an implicit fallback. Hoist the slim branch out of the encode loop so the two record shapes read side by side instead of being re-tested per log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017C79HXPerdogVPAPZAC3GE
Bee's snapshot consumer does not read logIndex, but the --resume cursor (PR #11) requires it in the final line to locate the last exported log within its block. Costs ~14 bytes per line before compression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q7Linc1WFpVVE4mEj6xmjG
Resolves the SaveLogsAsync overlap: the slim encode selection moved into the saveLogs helper introduced on main, so slim output and close-error propagation compose; filestore tests from both branches are combined. Also aligns README with go.mod (Go 1.25) and the current flag help. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q7Linc1WFpVVE4mEj6xmjG
martinconic
approved these changes
Aug 28, 2026
Contributor
|
LGTM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--slimflag (defaulttrue) emitting only thetypes.Logfields Bee's snapshot decoder consumes —address,topics,data,blockNumber,transactionHash— using the same hex JSON shape geth emits. Pass--slim=falseto keep the full geth shape.github.com/ethereum/go-ethereumv1.15.11 → v1.17.3 to align with Bee mainline (bee#5467).Why
The exported NDJSON is committed to batch-archive and
go:embeded into every Bee binary (pkg/postage/snapshot/archive/archive.go), so the size of this file is Bee's binary size.Bee reads the snapshot into
types.Logviajson.Decoder.Decodeinpkg/postage/snapshot/snapshot.go(parseLogs). Tracingpkg/postage/listener/listener.goandpkg/transaction/event.goshows it only consumes the five fields above; the rest —blockHash,transactionIndex,logIndex,blockTimestamp,removed— are dropped on read.blockHashis the expensive one: 32 high-entropy bytes per record that gzip cannot dedupe.Measured over the 157,920 records currently in batch-archive, both shapes compressed with the same
compress/gzipdefault levelpkg/gzipstoreuses:That is ~6.4 MB off every Bee binary once batch-archive is regenerated in slim shape.
Bee-side compatibility
No Bee change is required, and no
SlimLogtype belongs in Bee. geth's generatedLog.UnmarshalJSONrequires onlyaddress,topics,dataandtransactionHash;blockNumberand every dropped field are optional pointers. Slim emits those four plusblockNumber, so slim and full snapshots are decoder-interchangeable andlistener.gokeeps operating ontypes.Log.SlimLogstays a write-side serialization detail of this repo.Rollout is staged by Bee's
go.modpin on batch-archive: existing releases keep their full-shape blob, and only a deliberate bump picks up a slim one. Bee'sTestSnapshotLogFilterer_RealSnapshotparses the real embedded blob in CI, so a bad regeneration fails there rather than at runtime as a stalled postage sync.Note: silent-fail risk if Bee ever consumes a dropped field
Worth recording, because the failure mode is quiet. geth's generated
UnmarshalJSONdeclares every optional key as a pointer, so a missing key leaves the field at its zero value with no error:If Bee ever starts reading one of the dropped fields it would silently observe
0/common.Hash{}rather than failing loudly. Since this feeds the batch store, that is quiet data corruption rather than a crash.Current exposure is nil. Grepping all of
bee/pkg/for.BlockHash,.TxIndex,.BlockTimestampand.Removedoutside tests returns zero hits.ParseEventreads onlyTopicsandData;processEventreadsTopics[0]andTxHash;ListenreadsBlockNumber. Exactly the five fields slim emits.Per dropped field:
removed— the zero valuefalseis semantically correct. A snapshot only holds canonical, finalized logs; a reorged-out log should never be in it. Safe by construction.blockHash,blockTimestamp— would be silently wrong, but both serve reorg detection and time math, neither of which applies to a replay of finalized history. Batch expiry derives from block numbers and chainstate, not timestamps.logIndex/transactionIndex— the only plausible vector: ordering of logs within a single block. Ordering is correctness-relevant (aBatchCreated/BatchTopUppair in one block must not invert, or the topup lands on a batch that does not exist yet). But Bee does not derive it fromlogIndex:parseLogsvalidates only thatblockNumberis non-decreasing and otherwise relies on file order, which slim preserves byte-for-byte —fetchLogsemits sequentially from a single goroutine over ascending chunks, andSaveLogsAsyncwrites in channel order. The guarantee comes from the file, not from the field we dropped.Cost of re-adding both indexes as belt-and-braces, measured over the same 157,920 records:
transactionIndex+logIndexThe indexes are low-entropy and gzip nearly eliminates them;
blockHashis the ~5.8 MB item and the sole reason slim pays off. Not adding them for now, on the ordering argument above — recorded here so the tradeoff is a decision rather than an oversight.Follow-up (not in this PR)
--slimand bump Bee to it in a separate PR.parseLogsinpkg/postage/snapshot/snapshot.gostating that all othertypes.Logfields are zero and must not be relied upon, plus a test feeding a slim record throughprocessEventfor each event type.