feat: adopt the Vortex editions model by dfa1 · Pull Request #320 · dfa1/vortex-java

dfa1 · 2026-07-27T05:23:30Z

Summary

Adopts the Vortex editions model: a frozen, additive catalog of encoding-id sets, a writer-side guard against emitting an encoding outside the declared compatibility guarantee, and reader-side unknown-encoding UX naming which edition an id belongs to.

Ground-truth data was fetched directly from the real Rust source in vortex-data/vortex#8871, since the published spec's own "Edition registry" section says "Coming soon". vortex-java already implements every core-family encoding through core2026.07.0; of the unstable family, only fastlanes.delta/vortex.patched are implemented.

`core.model`

EditionId, Edition, EditionFamily — EditionFamily is a closed CORE/UNSTABLE enum, deliberately not mirroring EncodingId/LayoutId's sealed-interface-plus-Custom shape: a private edition family carries no real cross-implementation guarantee, so there's no legitimate custom-family use case (see ADR 0023's Alternatives section).
Editions catalog: the 8 frozen/draft editions, cumulativeMembers(), owningEdition().

`writer`

WriteOptions.editions defaults to the latest frozen core edition (core2026.07.0) — verified safe: the default cascade candidate list never includes the two unstable-family encoders vortex-java implements.
withEdition(Edition) / withoutEditionGuard() (the escape hatch).
VortexWriter's guard is two-layered — a mid-implementation course correction documented in ADR 0023: it seeds out-of-edition encoding ids into EncodeContext's existing exclusion set so CascadingCompressor gracefully steers away from them during cost-based selection (this matters because fastlanes.delta is reachable through an ordinary cascading(depth) write via MaskedEncodingEncoder's nested validity-mask competition, not just an explicit custom encoder list), plus an after-the-fact backstop check for any selection path that doesn't consult the exclusion set at all.

`reader`

ReadRegistry's unknown-encoding error now names the edition an id belongs to and its minimum Vortex version (or that it's an unstable draft with no guarantee), or says it's unknown to every edition and points at allowUnknown().

No wire-format changes

Editions are a write-time/read-time policy — nothing is persisted into the file. Matches upstream's own model.

Mechanical fallout

WriteOptions gaining an 8th record component required:

29 direct-constructor call sites across 12 files updated with an explicit Map.of() (preserves today's no-guard behavior, zero behavior change).
3 existing test files that deliberately write unstable-family encoders through the explicit encoder-list overload, updated with .withoutEditionGuard().

Test plan

./mvnw verify -DskipTests (full reactor compile)
./mvnw test (full unit suite, all modules)
./mvnw verify -pl integration -am (full integration suite, including the real-world file-size comparison tests that exercise cascading writes extensively)
./mvnw javadoc:javadoc -pl core (zero errors on new types; pre-existing unrelated warnings only)
New tests: EditionIdTest, EditionsTest (golden cumulative-member sets pinning the ground-truth catalog), WriterEditionGuardTest (backstop throw + escape hatch + explicit unstable opt-in), MaskedValidityCascadeEditionExclusionTest (the exact filtering mechanism, proven against a real cost-based win/exclude pair)

Closes #301.

🤖 Generated with Claude Code

Adds a frozen, additive editions catalog mirroring the ground-truth data from vortex-data/vortex#8871 (the published spec's own registry section says "Coming soon", so the catalog was built directly from the upstream Rust source), a writer-side edition guard, and reader unknown-encoding UX naming which edition an id belongs to. core.model: - EditionId, Edition, EditionFamily (a closed CORE/UNSTABLE enum, not a sealed-interface-plus-Custom like EncodingId/LayoutId: a private edition family carries no real cross-implementation guarantee, so there's no legitimate custom-family use case) - Editions catalog: the 8 frozen/draft editions, cumulativeMembers(), owningEdition() writer: - WriteOptions gains an editions field, defaulting to the latest frozen core edition (core2026.07.0) - verified safe: the default cascade candidate list never includes the two unstable-family encoders vortex-java implements (Delta, Patched) - withEdition(Edition) / withoutEditionGuard() (the escape hatch) - VortexWriter's guard has two layers: it seeds the out-of-edition encoding ids into EncodeContext's existing exclusion set so CascadingCompressor gracefully steers away from them during cost-based selection (including nested competitions like a masked column's validity-bitmap cascade - reachable via ordinary cascading(depth) writes, not just an explicit custom encoder list), plus an after-the-fact backstop check for paths that don't consult the exclusion set at all reader: - ReadRegistry's unknown-encoding VortexException now names the edition an id belongs to and its minimum Vortex version (or that it's an unstable draft with no guarantee), or says it's unknown to every edition and points at allowUnknown() No wire-format changes - editions are a write-time/read-time policy, never persisted into the file (ADR 0023, which also records the mid-implementation course correction from a simple after-the-fact check to the two-layer filter+backstop design, and why EditionFamily ended up closed rather than mirroring EncodingId's Custom pattern). Mechanical fallout: WriteOptions' 8th record component required updating 29 direct-constructor call sites (Map.of(), preserving today's no-guard behavior) plus 3 existing test files that deliberately write unstable-family encoders through the explicit encoder-list overload (added .withoutEditionGuard()). Verified: full reactor build, full unit suite, full integration suite (including the real-world file-size comparison tests that exercise cascading writes extensively), core module javadoc build. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

dfa1 · 2026-07-27T05:29:05Z

+
+| Edition | Family | Min Vortex | Adds |
+|---|---|---|---|
+| `core2025.05.0` | `core` | 0.36.0 | 23 baseline: `fastlanes.bitpacked`/`for`, `vortex.alp`/`alprd`/`bool`/`bytebool`/`chunked`/`constant`/`datetimeparts`/`decimal`/`decimal_byte_parts`/`dict`/`ext`/`fsst`/`list`/`null`/`primitive`/`runend`/`sparse`/`struct`/`varbin`/`varbinview`/`zigzag` |


this "Min Vortex" refers to upstream: let's drop it as it is meangiless for us

dfa1 · 2026-07-27T05:30:00Z

+/// @param year    the four-digit year the edition was cut
+/// @param month   the month the edition was cut (1-12)
+/// @param version distinguishes editions cut in the same month; normally `0`
+public record EditionId(EditionFamily family, int year, int month, int version) {


let's use YearMonth class from JDK

dfa1 · 2026-07-27T05:30:34Z

+/// @param minVortexVersion the minimum Vortex release whose reader supports every encoding in
+///                         this edition, or empty if this edition is a draft (see [#isDraft()])
+/// @param added            the encodings that join the family at this edition
+public record Edition(EditionId id, Optional<String> minVortexVersion, Set<EncodingId> added) {


let's drop minVortexVersion => this refers to the JNI but it is not useful for us

dfa1 · 2026-07-27T05:31:04Z

+    public static final Edition CORE_2025_05_0 = new Edition(
+            new EditionId(EditionFamily.CORE, 2025, 5, 0),
+            Optional.of("0.36.0"),
+            idSet(


let's use EncodingId constants + EnumSet

dfa1 · 2026-07-27T05:31:47Z

        DType.Struct schema = new DType.Struct(
                List.of(ColumnName.of("v")), List.of(new DType.Primitive(PType.I64, true)), false);
-        WriteOptions opts = new WriteOptions(2, true, 0.90, 0, true, false, 256L * 1024 * 1024);
+        WriteOptions opts = new WriteOptions(2, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of());


let's import java.util.Map

Two review comments from the PR: - EditionId(EditionFamily, int year, int month, int version) -> EditionId(EditionFamily, YearMonth cutMonth, int version). java.time.YearMonth already validates the month range (throwing DateTimeException, arguably more idiomatic than a hand-rolled IllegalArgumentException for this exact domain) and is Comparable, so isAtOrBefore simplifies to a single compareTo plus the version tie-break. Drops the separate 4-digit-year guard - belt-and-braces validation on a value only ever constructed from 8 hardcoded catalog literals, not from untrusted input. - Editions' catalog now references the real EncodingId.WellKnown constants (EncodingId.VORTEX_ALP, ...) directly instead of raw wire strings parsed at runtime via EncodingId.parse(...). Matches this project's own "strings at the boundary, types inside" principle - these ids come from a hardcoded catalog, not wire input, so there was no boundary to parse a string at; using the constants also catches a typo'd id at compile time instead of it silently becoming an unintended EncodingId.Custom. Only the handful of unstable ids with no WellKnown constant yet still use EncodingId.Custom. Applied the same fix to EditionsTest's golden-set expected values. Also reverted EditionFamily#toString() (a mid-session addition) back to the enum default: overriding toString() to return something other than the declared constant name is surprising and breaks valueOf(family.toString()). EditionId#toString() now lowercases the family name explicitly at the one call site that needs the wire form. ADR 0023 and docs/reference.md updated to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

dfa1 · 2026-07-27T05:33:38Z

+    /// third-party or experimental ones, at the cost of the edition read-compatibility guarantee.
+    ///
+    /// @return a new `WriteOptions` with no edition guard at all
+    public WriteOptions withoutEditionGuard() {


let's drop this escape hatch => this defeats the purpose

dfa1

review

- Drop Edition.minVortexVersion: it names the Rust reference's own release train, not vortex-java's, so it was never meaningful here. - Drop WriteOptions#withoutEditionGuard(): the only escape hatch was guard-wide and untracked. Reaching an unstable-family encoding now requires enabling that family's edition explicitly via withEdition(Editions.UNSTABLE_...), which is both narrower and self-documenting. Updates the 9 test call sites that used the old escape hatch to reach fastlanes.delta/vortex.patched. - Implement ReadRegistry's edition-aware "no decoder registered" message, which was documented in the ADR/CHANGELOG/docs but never actually wired into decode()/decodeAsSegment() in the original PR. Add coverage for the core-edition, unstable-edition, and unknown-to-all-editions message branches, plus decodeAsSegment's error path (previously untested). - java.util.Map.of() -> Map.of() (redundant qualification) across 11 files. - Update ADR 0023 and CHANGELOG.md to match the final design; sweep docs/reference.md and docs/compatibility.md for stale references. Addresses the 6 review comments left on #320 (issue #301). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

dfa1 commented Jul 27, 2026

View reviewed changes

dfa1 merged commit 8c71934 into main Jul 27, 2026
6 checks passed

dfa1 deleted the feat/301-vortex-editions branch July 27, 2026 05:35

dfa1 commented Jul 27, 2026

View reviewed changes

dfa1 mentioned this pull request Jul 27, 2026

fix(editions): address PR #320 review findings #322

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: adopt the Vortex editions model - #320

feat: adopt the Vortex editions model#320
dfa1 merged 2 commits into
mainfrom
feat/301-vortex-editions

dfa1 commented Jul 27, 2026

Uh oh!

dfa1 Jul 27, 2026

Uh oh!

dfa1 Jul 27, 2026

Uh oh!

dfa1 Jul 27, 2026

Uh oh!

dfa1 Jul 27, 2026

Uh oh!

dfa1 Jul 27, 2026

Uh oh!

dfa1 Jul 27, 2026

Uh oh!

Uh oh!

dfa1 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

dfa1 commented Jul 27, 2026

Summary

core.model

writer

reader

No wire-format changes

Mechanical fallout

Test plan

Uh oh!

dfa1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

dfa1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

dfa1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

dfa1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

dfa1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

dfa1 Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dfa1 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

`core.model`

`writer`

`reader`