Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions docs/specs/editions.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,73 @@ edition; each encoding's registry entry records the edition it joined in. In the
encoding may be *deprecated*, meaning writers stop emitting it — but readers keep decoding it
indefinitely, so deprecation never invalidates existing files.

Freezing an edition also freezes the *meaning* of each member: what a reader may assume about
bytes written under that ID never changes afterwards. An incompatible extension of a member is
a new serialized format with a new ID, not a mutation of the frozen one — see
[Extending a frozen format](#extending-a-frozen-format).

## Editions name serialized formats

An edition member is a **serialized format**: the encoding ID written into the file, together
with its metadata schema and the meaning of its buffers and children. The read-forever
guarantee attaches to those bytes. The in-memory encoding that produced them is an
implementation detail — it may evolve, gain capabilities, or be replaced entirely without
touching any edition.

The serialization plugin registry is the layer that maps between the two, in both directions:

- **On read**, a serialized format may be deserialized into whichever in-memory encoding the
reader prefers. For example, a serialized `vortex.alp` array with interior patches is
deserialized as a `Patched` array wrapping a patch-free ALP array: the file bytes are frozen,
but the in-memory representation improved underneath them.
- **On write**, the plugin chooses which serialized format to emit for a given array. Most
encodings have exactly one, so the serialized ID and the in-memory ID coincide — but nothing
requires this, and one in-memory encoding may own several serialized formats.

Because the write-time edition check applies to the ID that is written into the file, an
edition constrains exactly what it guarantees: the bytes a reader will meet.

### Extending a frozen format

A frozen format never changes meaning. If an in-memory encoding learns to represent something
its serialized format cannot carry — even through a metadata field that was always present but
constrained to one value — writing it under the old ID would break every reader the edition
promised could read it. Such an extension is a **new serialized format with a new ID**, staged
in a draft edition like any other new member. The in-memory encoding does not fork; it gains a
second serialized format:

- Arrays the old format can represent keep serializing under the old ID, byte-identical,
readable by every reader since the old edition froze.
- Arrays only the new format can represent serialize under the new ID, writable only once an
edition containing it is enabled.
- Readers register one plugin for both IDs; both deserialize into the same in-memory encoding.

Name revisions as a version chain on the same base name — `vortex.foo`, `vortex.foo_v2`,
`vortex.foo_v3` — never as descriptively named variants. Each format then has at most one
successor, so an encoding's serialized history reads as a list, not a tree: to know what a
reader must support, you walk one chain, and there is never a fork where two live formats
both claim to succeed the same frozen one.

### Example: multi-part decimals

`vortex.decimal_byte_parts` froze into `core2025.05.0` representing each decimal value as a
single signed integer child. Its metadata reserves a `lower_part_count` field, but every
reader of the frozen format requires it to be zero — so that field cannot be used to extend
the format after the fact. Suppose the in-memory encoding grows support for wide decimals as
a signed most-significant part plus up to three unsigned 64-bit lower parts. Then:

- A single-part array still serializes as `vortex.decimal_byte_parts` with
`lower_part_count = 0`, indistinguishable from files written before the change.
- An array carrying lower parts serializes as `vortex.decimal_byte_parts_v2` — the next link
in the format's version chain — staged in a draft edition. A writer pinned to an edition
without it cannot emit it; the compressor consults the enabled editions and only produces
multi-part arrays when the new format is allowed, so the write-time check never fires as a
surprise.
- A reader that supports the new format deserializes both IDs into the same in-memory
encoding. A reader that predates it fails on `vortex.decimal_byte_parts_v2` with an
unknown-encoding error pointing at the registry — the failure mode editions promise —
rather than crashing inside a decoder that was never taught about lower parts.

## Edition registry

Coming soon..
Loading