fix(wire): BLOB columns no longer corrupt to {} — normalize ArrayBuffer to Uint8Array in both codecs#31
Merged
Merged
Conversation
… both codecs
workerd's SqlStorage returns BLOB columns as bare ArrayBuffer.
@msgpack/msgpack only special-cases ArrayBuffer.isView, so a bare
ArrayBuffer fell through to encodeMap and arrived on the client as {} —
silently, for snapshots and deltas alike. The JSON debug codec had the
same gap: classify() tagged Uint8Array ('u8') but not ArrayBuffer.
Normalize at emission (ADR-0017) rather than rejecting BLOB columns in
assertSyncCompatible: a BLOB is bytes, Uint8Array is the codecs' existing
byte type, and the round-trip is lossless — so make it work instead of
failing loud. Binary codec registers a msgpack ExtensionCodec entry
(custom type 0) that intercepts bare ArrayBuffer before the encodeMap
fallthrough and decodes to a copied Uint8Array (a subarray view would
alias the wire buffer — caught by codex adversarial review). JSON codec
tags ArrayBuffer as 'u8'; its decode side was already symmetric.
tests/row-shape.test.ts pins the end-to-end guarantee: BLOB arrives as
Uint8Array with exact bytes for cold snapshots AND live deltas, alongside
the existing row-shape guarantees (verbatim snake_case column names, JSON
TEXT stays string, INTEGER/REAL/NULL fidelity, >2^53 rounding per #10).
Fixes #27
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #27
Problem
workerd's
SqlStoragereturns BLOB column values as bareArrayBuffer.@msgpack/msgpackonly special-casesArrayBuffer.isView, so a bareArrayBufferfell through toencodeMapand the client silently received{}— for cold snapshots and live deltas alike. The JSON debug codec had the same gap:classify()taggedUint8Array(u8) but notArrayBuffer, andJSON.stringify(ArrayBuffer)is{}.Design decision (ADR-0017)
Normalize at emission rather than rejecting BLOB-affinity columns in
assertSyncCompatible. Rejection is the tool for shapes that cannot be made safe; a BLOB can be, losslessly — it is bytes,Uint8Arrayis the codecs' existing byte type, and a client writing theUint8Arrayback stores the same bytes. The wire now has exactly one byte type: clients always receiveUint8Array, never a bareArrayBuffer. No warning added toassertSyncCompatible— after normalization there is nothing for an author to act on.wire/frame-codec.ts): msgpackExtensionCodecentry (custom type0), consulted before theencodeMapfallthrough; decodes to a copiedUint8Array(returning the subarray view would alias the wire buffer — caught in adversarial review).wire/codec.ts):classify()tagsArrayBufferasu8; decode was already symmetric.Out of scope: >2^53 INTEGER precision loss (#10, wontfix) — happens before the wire; cross-referenced in the ADR and pinned in the test.
Tests
tests/row-shape.test.ts(new, end-to-end in workerd): BLOB written asnew Uint8Array([1,2,3]).bufferarrives on the subscribed client as aUint8Arraywith those bytes — snapshot and live delta — alongside the other row-shape guarantees (verbatim snake_case column names, JSON TEXT stays a string, INTEGER/REAL/NULL fidelity).tests/codec.test.ts/tests/frame-codec.test.ts: codec-level ArrayBuffer round-trip for both codecs, plus a no-aliasing assertion on the binary path.Full suite: 44 files / 198 tests green;
tsc --noEmitclean.Adversarial review
codex exec(gpt-5.5) reviewed the diff: found one real bug — the ext decoder returned msgpack's subarray view into the decode buffer (mutable through the transport buffer; pins the whole frame allocation behind a small BLOB). Fixed withbytes.slice()and pinned by test. It verified no ext-type collision (timestamp is built-in-1) and that all emission paths route through the updated codecs.🤖 Generated with Claude Code