Skip to content

fix(wire): BLOB columns no longer corrupt to {} — normalize ArrayBuffer to Uint8Array in both codecs#31

Merged
grrowl merged 2 commits into
mainfrom
fix/blob-wire-encoding
Jul 22, 2026
Merged

fix(wire): BLOB columns no longer corrupt to {} — normalize ArrayBuffer to Uint8Array in both codecs#31
grrowl merged 2 commits into
mainfrom
fix/blob-wire-encoding

Conversation

@grrowl

@grrowl grrowl commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Fixes #27

Problem

workerd's SqlStorage returns BLOB column values as bare ArrayBuffer. @msgpack/msgpack only special-cases ArrayBuffer.isView, so a bare ArrayBuffer fell through to encodeMap and the client silently received {} — for cold snapshots and live deltas alike. The JSON debug codec had the same gap: classify() tagged Uint8Array (u8) but not ArrayBuffer, and JSON.stringify(ArrayBuffer) is {}.

Design decision (ADR-0017)

Normalize at emission rather than rejecting BLOB-affinity columns in assertSyncCompatible. Rejection is the tool for shapes that cannot be made safe; a BLOB can be, losslessly — it is bytes, Uint8Array is the codecs' existing byte type, and a client writing the Uint8Array back stores the same bytes. The wire now has exactly one byte type: clients always receive Uint8Array, never a bare ArrayBuffer. No warning added to assertSyncCompatible — after normalization there is nothing for an author to act on.

  • Binary codec (wire/frame-codec.ts): msgpack ExtensionCodec entry (custom type 0), consulted before the encodeMap fallthrough; decodes to a copied Uint8Array (returning the subarray view would alias the wire buffer — caught in adversarial review).
  • JSON debug codec (wire/codec.ts): classify() tags ArrayBuffer as u8; decode was already symmetric.

Out of scope: >2^53 INTEGER precision loss (#10, wontfix) — happens before the wire; cross-referenced in the ADR and pinned in the test.

Tests

  • tests/row-shape.test.ts (new, end-to-end in workerd): BLOB written as new Uint8Array([1,2,3]).buffer arrives on the subscribed client as a Uint8Array with those bytes — snapshot and live delta — alongside the other row-shape guarantees (verbatim snake_case column names, JSON TEXT stays a string, INTEGER/REAL/NULL fidelity).
  • tests/codec.test.ts / tests/frame-codec.test.ts: codec-level ArrayBuffer round-trip for both codecs, plus a no-aliasing assertion on the binary path.

Full suite: 44 files / 198 tests green; tsc --noEmit clean.

Adversarial review

codex exec (gpt-5.5) reviewed the diff: found one real bug — the ext decoder returned msgpack's subarray view into the decode buffer (mutable through the transport buffer; pins the whole frame allocation behind a small BLOB). Fixed with bytes.slice() and pinned by test. It verified no ext-type collision (timestamp is built-in -1) and that all emission paths route through the updated codecs.

🤖 Generated with Claude Code

grrowl and others added 2 commits July 22, 2026 13:00
… both codecs

workerd's SqlStorage returns BLOB columns as bare ArrayBuffer.
@msgpack/msgpack only special-cases ArrayBuffer.isView, so a bare
ArrayBuffer fell through to encodeMap and arrived on the client as {} —
silently, for snapshots and deltas alike. The JSON debug codec had the
same gap: classify() tagged Uint8Array ('u8') but not ArrayBuffer.

Normalize at emission (ADR-0017) rather than rejecting BLOB columns in
assertSyncCompatible: a BLOB is bytes, Uint8Array is the codecs' existing
byte type, and the round-trip is lossless — so make it work instead of
failing loud. Binary codec registers a msgpack ExtensionCodec entry
(custom type 0) that intercepts bare ArrayBuffer before the encodeMap
fallthrough and decodes to a copied Uint8Array (a subarray view would
alias the wire buffer — caught by codex adversarial review). JSON codec
tags ArrayBuffer as 'u8'; its decode side was already symmetric.

tests/row-shape.test.ts pins the end-to-end guarantee: BLOB arrives as
Uint8Array with exact bytes for cold snapshots AND live deltas, alongside
the existing row-shape guarantees (verbatim snake_case column names, JSON
TEXT stays string, INTEGER/REAL/NULL fidelity, >2^53 rounding per #10).

Fixes #27

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@grrowl
grrowl merged commit 144c6ae into main Jul 22, 2026
1 check passed
@grrowl
grrowl deleted the fix/blob-wire-encoding branch July 22, 2026 04:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BLOB columns silently corrupt to {} over the wire (ArrayBuffer not handled by msgpack encoder)

1 participant