Skip to content

feat: slim output and bump geth - #8

Open
gacevicljubisa wants to merge 8 commits into
mainfrom
feat/slim-output-and-bump-geth
Open

feat: slim output and bump geth#8
gacevicljubisa wants to merge 8 commits into
mainfrom
feat/slim-output-and-bump-geth

Conversation

@gacevicljubisa

@gacevicljubisa gacevicljubisa commented May 28, 2026

Copy link
Copy Markdown
Member

Summary

  • Add --slim flag (default true) emitting only the types.Log fields Bee's snapshot decoder consumes — address, topics, data, blockNumber, transactionHash — using the same hex JSON shape geth emits. Pass --slim=false to keep the full geth shape.
  • Bump github.com/ethereum/go-ethereum v1.15.11 → v1.17.3 to align with Bee mainline (bee#5467).
  • Add guard tests so a future geth bump that changes a kept field's encoding or adds a required field fails CI.

Why

The exported NDJSON is committed to batch-archive and go:embeded into every Bee binary (pkg/postage/snapshot/archive/archive.go), so the size of this file is Bee's binary size.

Bee reads the snapshot into types.Log via json.Decoder.Decode in pkg/postage/snapshot/snapshot.go (parseLogs). Tracing pkg/postage/listener/listener.go and pkg/transaction/event.go shows it only consumes the five fields above; the rest — blockHash, transactionIndex, logIndex, blockTimestamp, removed — are dropped on read. blockHash is the expensive one: 32 high-entropy bytes per record that gzip cannot dedupe.

Measured over the 157,920 records currently in batch-archive, both shapes compressed with the same compress/gzip default level pkg/gzipstore uses:

uncompressed gzipped
full 89,649,991 15,445,422
slim 67,593,394 8,990,863
saving 24.6% 41.8% (−6.45 MB)

That is ~6.4 MB off every Bee binary once batch-archive is regenerated in slim shape.

Bee-side compatibility

No Bee change is required, and no SlimLog type belongs in Bee. geth's generated Log.UnmarshalJSON requires only address, topics, data and transactionHash; blockNumber and every dropped field are optional pointers. Slim emits those four plus blockNumber, so slim and full snapshots are decoder-interchangeable and listener.go keeps operating on types.Log. SlimLog stays a write-side serialization detail of this repo.

Rollout is staged by Bee's go.mod pin on batch-archive: existing releases keep their full-shape blob, and only a deliberate bump picks up a slim one. Bee's TestSnapshotLogFilterer_RealSnapshot parses the real embedded blob in CI, so a bad regeneration fails there rather than at runtime as a stalled postage sync.

Note: silent-fail risk if Bee ever consumes a dropped field

Worth recording, because the failure mode is quiet. geth's generated UnmarshalJSON declares every optional key as a pointer, so a missing key leaves the field at its zero value with no error:

BlockHash *common.Hash `json:"blockHash" rlp:"-"`
...
if dec.BlockHash != nil { l.BlockHash = *dec.BlockHash }

If Bee ever starts reading one of the dropped fields it would silently observe 0 / common.Hash{} rather than failing loudly. Since this feeds the batch store, that is quiet data corruption rather than a crash.

Current exposure is nil. Grepping all of bee/pkg/ for .BlockHash, .TxIndex, .BlockTimestamp and .Removed outside tests returns zero hits. ParseEvent reads only Topics and Data; processEvent reads Topics[0] and TxHash; Listen reads BlockNumber. Exactly the five fields slim emits.

Per dropped field:

  • removed — the zero value false is semantically correct. A snapshot only holds canonical, finalized logs; a reorged-out log should never be in it. Safe by construction.
  • blockHash, blockTimestamp — would be silently wrong, but both serve reorg detection and time math, neither of which applies to a replay of finalized history. Batch expiry derives from block numbers and chainstate, not timestamps.
  • logIndex / transactionIndex — the only plausible vector: ordering of logs within a single block. Ordering is correctness-relevant (a BatchCreated / BatchTopUp pair in one block must not invert, or the topup lands on a batch that does not exist yet). But Bee does not derive it from logIndex: parseLogs validates only that blockNumber is non-decreasing and otherwise relies on file order, which slim preserves byte-for-byte — fetchLogs emits sequentially from a single goroutine over ascending chunks, and SaveLogsAsync writes in channel order. The guarantee comes from the file, not from the field we dropped.

Cost of re-adding both indexes as belt-and-braces, measured over the same 157,920 records:

gzipped
slim 8,990,863
slim + transactionIndex + logIndex 9,667,188 (+676 KB)
full 15,445,422

The indexes are low-entropy and gzip nearly eliminates them; blockHash is the ~5.8 MB item and the sole reason slim pays off. Not adding them for now, on the ordering argument above — recorded here so the tradeoff is a decision rather than an oversight.

Follow-up (not in this PR)

  1. Regenerate the batch-archive snapshot with --slim and bump Bee to it in a separate PR.
  2. Close the read-side gap. The guard tests added here protect the write side (a geth bump changing a kept field's shape). Nothing protects the read side — Bee starting to consume a sixth field. Cheapest fix, in Bee where a future reader would actually look: a contract comment above parseLogs in pkg/postage/snapshot/snapshot.go stating that all other types.Log fields are zero and must not be relied upon, plus a test feeding a slim record through processEvent for each event type.

gacevicljubisa and others added 6 commits May 28, 2026 14:29
The merge in 7f43ae5 reverted the --slim default from true (as shipped in
7290a12) back to false. Left as-is, this branch would ship slim mode off
and the next batch-archive refresh would produce a full-shape snapshot
regardless. The README flag dump was regenerated from the reverted code,
so it also contradicted its own feature bullet.

Reword the flag help now that the flag defaults on: pflag binds booleans
as --slim=false, since a bare --slim sets it true, so state that form
explicitly rather than describing an implicit fallback.

Hoist the slim branch out of the encode loop so the two record shapes
read side by side instead of being re-tested per log.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017C79HXPerdogVPAPZAC3GE
Bee's snapshot consumer does not read logIndex, but the --resume cursor
(PR #11) requires it in the final line to locate the last exported log
within its block. Costs ~14 bytes per line before compression.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7Linc1WFpVVE4mEj6xmjG
Resolves the SaveLogsAsync overlap: the slim encode selection moved into
the saveLogs helper introduced on main, so slim output and close-error
propagation compose; filestore tests from both branches are combined.
Also aligns README with go.mod (Go 1.25) and the current flag help.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7Linc1WFpVVE4mEj6xmjG
@martinconic

Copy link
Copy Markdown
Contributor

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants