From 11632bc042eb07d753aaba80c5bb5e5c8e8b7020 Mon Sep 17 00:00:00 2001 From: Davide Angelocola Date: Mon, 27 Jul 2026 07:23:07 +0200 Subject: [PATCH 1/2] feat(writer,reader,core): adopt the Vortex editions model (#301) Adds a frozen, additive editions catalog mirroring the ground-truth data from vortex-data/vortex#8871 (the published spec's own registry section says "Coming soon", so the catalog was built directly from the upstream Rust source), a writer-side edition guard, and reader unknown-encoding UX naming which edition an id belongs to. core.model: - EditionId, Edition, EditionFamily (a closed CORE/UNSTABLE enum, not a sealed-interface-plus-Custom like EncodingId/LayoutId: a private edition family carries no real cross-implementation guarantee, so there's no legitimate custom-family use case) - Editions catalog: the 8 frozen/draft editions, cumulativeMembers(), owningEdition() writer: - WriteOptions gains an editions field, defaulting to the latest frozen core edition (core2026.07.0) - verified safe: the default cascade candidate list never includes the two unstable-family encoders vortex-java implements (Delta, Patched) - withEdition(Edition) / withoutEditionGuard() (the escape hatch) - VortexWriter's guard has two layers: it seeds the out-of-edition encoding ids into EncodeContext's existing exclusion set so CascadingCompressor gracefully steers away from them during cost-based selection (including nested competitions like a masked column's validity-bitmap cascade - reachable via ordinary cascading(depth) writes, not just an explicit custom encoder list), plus an after-the-fact backstop check for paths that don't consult the exclusion set at all reader: - ReadRegistry's unknown-encoding VortexException now names the edition an id belongs to and its minimum Vortex version (or that it's an unstable draft with no guarantee), or says it's unknown to every edition and points at allowUnknown() No wire-format changes - editions are a write-time/read-time policy, never persisted into the file (ADR 0023, which also records the mid-implementation course correction from a simple after-the-fact check to the two-layer filter+backstop design, and why EditionFamily ended up closed rather than mirroring EncodingId's Custom pattern). Mechanical fallout: WriteOptions' 8th record component required updating 29 direct-constructor call sites (Map.of(), preserving today's no-guard behavior) plus 3 existing test files that deliberately write unstable-family encoders through the explicit encoder-list overload (added .withoutEditionGuard()). Verified: full reactor build, full unit suite, full integration suite (including the real-world file-size comparison tests that exercise cascading writes extensively), core module javadoc build. Co-Authored-By: Claude Sonnet 5 --- CHANGELOG.md | 4 + CLAUDE.md | 10 +- adr/0023-vortex-editions-adoption.md | 207 ++++++++++++++++++ adr/ADR.md | 1 + .../vortex/calcite/AggregateSumNullTest.java | 2 +- .../calcite/AggregateWhereBoundaryTest.java | 4 +- .../AggregateWhereCleanPartitionTest.java | 4 +- .../dfa1/vortex/calcite/OhlcGenerator.java | 2 +- .../vortex/calcite/UnsignedColumnTest.java | 2 +- .../calcite/VortexAdapterCoverageTest.java | 2 +- .../dfa1/vortex/core/model/Edition.java | 40 ++++ .../dfa1/vortex/core/model/EditionFamily.java | 24 ++ .../dfa1/vortex/core/model/EditionId.java | 57 +++++ .../dfa1/vortex/core/model/Editions.java | 139 ++++++++++++ .../dfa1/vortex/core/model/EditionIdTest.java | 143 ++++++++++++ .../dfa1/vortex/core/model/EditionsTest.java | 206 +++++++++++++++++ docs/compatibility.md | 24 +- docs/reference.md | 67 +++++- .../JavaRoundTripIntegrationTest.java | 2 +- .../JavaWritesRustReadsIntegrationTest.java | 4 +- .../CalciteBoundaryAggregateBenchmark.java | 2 +- .../dfa1/vortex/writer/VortexWriter.java | 95 +++++++- .../dfa1/vortex/writer/WriteOptions.java | 72 +++++- .../vortex/writer/encode/EncodeContext.java | 32 ++- .../dfa1/vortex/writer/DeltaEncodingTest.java | 14 +- .../vortex/writer/NullCountPruningTest.java | 2 +- .../dfa1/vortex/writer/VortexWriterTest.java | 2 +- .../vortex/writer/WriterEditionGuardTest.java | 102 +++++++++ .../dfa1/vortex/writer/WriterZoneMapTest.java | 22 +- ...edValidityCascadeEditionExclusionTest.java | 86 ++++++++ 30 files changed, 1321 insertions(+), 52 deletions(-) create mode 100644 adr/0023-vortex-editions-adoption.md create mode 100644 core/src/main/java/io/github/dfa1/vortex/core/model/Edition.java create mode 100644 core/src/main/java/io/github/dfa1/vortex/core/model/EditionFamily.java create mode 100644 core/src/main/java/io/github/dfa1/vortex/core/model/EditionId.java create mode 100644 core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java create mode 100644 core/src/test/java/io/github/dfa1/vortex/core/model/EditionIdTest.java create mode 100644 core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java create mode 100644 writer/src/test/java/io/github/dfa1/vortex/writer/WriterEditionGuardTest.java create mode 100644 writer/src/test/java/io/github/dfa1/vortex/writer/encode/MaskedValidityCascadeEditionExclusionTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index c45d4b01..1b057c51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Adopted the Vortex [editions](https://github.com/vortex-data/vortex/blob/develop/docs/specs/editions.md) model: `WriteOptions` gates which encodings a write may emit (defaulting to the latest frozen `core` edition) and fails immediately if it would emit one outside it; `ReadRegistry`'s unknown-encoding error now names the edition an id belongs to and the minimum Vortex version needed. ([#301](https://github.com/dfa1/vortex-java/issues/301)) + ## [0.12.5] — 2026-07-26 Real-world file-size work surfaced by the [Raincloud conformance corpus](https://github.com/dfa1/vortex-java/issues/205): importing the `nyc-311` slug (18.5M rows × 42 columns) and diffing its layout tree against the `vortex-jni` reference exposed a chain of writer gaps. Closing them took `vortex-java` from 2854 MB (1.62× the size of `vortex-jni`) to 1644 MB — 0.93×, smaller than the Rust reference. diff --git a/CLAUDE.md b/CLAUDE.md index 164d3826..46f3b121 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,8 @@ fsst — io.github.dfa1.vortex.fsst: standalone FSST (Fast Static Symbol Tabl Zero dependency on core/reader/writer/FFM-wire-concerns beyond the JDK's own java.lang.foreign; writer/reader depend on it, not the other way around. core — everything lives under `io.github.dfa1.vortex.core.*`: - core.model DType, PType, TimeUnit, EncodingId, LayoutId, ColumnName, ExtensionId, TimeDtype, TimestampDtype + core.model DType, PType, TimeUnit, EncodingId, LayoutId, ColumnName, ExtensionId, TimeDtype, TimestampDtype, + EditionId, Edition, EditionFamily, Editions core.io IoBounds, PTypeIO, VortexFormat core.error VortexException core.compute FastLanes, PrimitiveArrays @@ -244,6 +245,13 @@ When stuck on encode/decode behavior, consult **in this order**: - **Small public APIs.** Don't expose internals — when in doubt, leave it out or make it private. - **POM deps** grouped with comments: `` then ``, each with project-internal (`io.github.dfa1.vortex:*`) deps first, then external. Omit empty sections. +- **Editions are a client-side write/read policy, not part of the wire format** (ADR 0023). + `WriteOptions#editions()` gates which encodings a write may emit and is checked/enforced entirely + at write time; nothing about a targeted edition is ever persisted into a `.vortex` file — the + compatibility guarantee is always re-derivable from the encoding ids already in the footer plus + the shared `Editions` catalog. `EditionFamily` is a closed enum (`CORE`/`UNSTABLE`), unlike + `EncodingId`/`LayoutId`'s sealed-interface-plus-`Custom` shape: a private edition family would + carry no real cross-implementation guarantee, so there is no legitimate use case for one. ## Documentation is part of every change diff --git a/adr/0023-vortex-editions-adoption.md b/adr/0023-vortex-editions-adoption.md new file mode 100644 index 00000000..ee9176b8 --- /dev/null +++ b/adr/0023-vortex-editions-adoption.md @@ -0,0 +1,207 @@ +# ADR 0023: Adopt the Vortex editions model as a client-side write/read policy + +- **Status:** Accepted +- **Date:** 2026-07-26 +- **Deciders:** project maintainer +- **Supersedes:** — +- **Superseded by:** — + +## Context + +Upstream Vortex (`vortex-data/vortex` [PR #8871](https://github.com/vortex-data/vortex/pull/8871)) +formalized **editions**: frozen, named, additive sets of encoding IDs, each carrying a forever +read-compatibility guarantee. Editions come in independently-versioned **families** (`core`, +`unstable` today); a writer targets at most one edition per family and may emit any encoding in +the union of its enabled editions' cumulative members. An edition with no recorded +`min_vortex_version` is a **draft** — every `unstable` edition is a draft. + +vortex-java already implements every encoding in every frozen `core` edition through +`core2026.07.0` (issue #301). This ADR is not about chasing new encodings — it is about adopting +the *editions model itself*: a frozen registry, a writer fail-fast guard ("never write a file a +conforming reader can't read"), and reader error UX naming which edition an unknown encoding +belongs to. vortex-java had no equivalent of any of this before. + +**Ground-truth data.** The published spec +([`docs/specs/editions.md`](https://github.com/vortex-data/vortex/blob/develop/docs/specs/editions.md)) +documents the concept but its own "Edition registry" section says "Coming soon" — the concrete +data is not public yet. The catalog in this codebase (`core.model.Editions`) was built directly +from the real Rust source at PR #8871 (`vortex-edition/src/lib.rs`, +`vortex/src/editions/{core,unstable}/*.rs`), not guessed at. Cross-checked against vortex-java's +own `EncodingId.WellKnown`: all 31 `core`-family encodings across the 4 frozen editions already +have a `WellKnown` constant; of the 4 draft `unstable` editions (9 total ids), only +`fastlanes.delta` and `vortex.patched` do. `EncodingId.parse(rawId)` already falls back to +`EncodingId.Custom` for the rest, so the catalog stores every id uniformly and mirrors upstream +faithfully rather than being truncated to what is implemented today. + +## Decision + +1. **Three types, not Rust's four-plus-a-session.** Rust's model (`EditionId`, `Edition`, + `EditionInclusion`, `EditionDeclaration`, plus a mutable cross-crate `EditionSession`) exists + because any Rust crate can register a declaration into the session. vortex-java's registries + are explicit and closed (no `ServiceLoader` — the same design decision already made for + `ReadRegistry`/`WriteRegistry`), so this collapses to three `core.model` types: + - `EditionId(EditionFamily family, int year, int month, int version)` — `toString()` matches + Rust's `Display` exactly (`"core2025.05.0"`, month zero-padded, version not); + `isAtOrBefore(EditionId)` orders editions within a family (cross-family is always `false`). + - `Edition(EditionId id, Optional minVortexVersion, Set added)` — folds + Rust's separate `Edition`/`EditionDeclaration` into one type; `added` is only what joins *at + this exact edition*, matching Rust's field exactly (not cumulative). + - `Editions` — a final utility class holding the 8 catalog constants, `ALL` (declaration + order), `cumulativeMembers(Edition)` (union of `added()` for every same-family edition at or + before the given one, seeded with the argument's own `added()` first), and + `owningEdition(EncodingId)` (first declaration in `ALL` whose `added()` contains the id — + correct because membership is additive and each encoding belongs to exactly one family). + +2. **`EditionFamily` is a closed enum (`CORE`, `UNSTABLE`), not a sealed + interface with a `Custom` fallback** — unlike `EncodingId`/`LayoutId`, which do allow a + `Custom` variant. A first pass modeled `EditionFamily` on that exact sealed-interface pattern + (arguing families are "open-ended" the same way encoding ids are), but the two are not + analogous: an encoding id's `Custom` variant has a real use case (a private company's own + proprietary encoding, decoded by *their own* reader — no cross-implementation agreement + needed). An edition **family** is inherently a cross-implementation compatibility promise + ("any reader supporting `core2026.07.0` can read this file"); a private, single-writer family + would carry none of that benefit, since no other reader in the ecosystem would recognize it. + There is no legitimate use case for a custom family, so the type is closed. A caller who wants + to write encodings outside any known family uses `WriteOptions#withoutEditionGuard()`, not a + fabricated family. + - `Edition` itself still cannot be made construction-restricted the same way: Java forbids a + record's canonical constructor from being *more* restrictive than the record type's own + visibility, and `Edition` must stay `public` for `WriteOptions`'s public API + (`withEdition(Edition)`). So "only `Editions`'s 8 catalog constants should exist" is + documented API contract, not something the compiler enforces — a real Java-language + limitation, not a design choice. + +3. **`WriteOptions.editions` defaults to `{CORE: Editions.CORE_2026_07_0}`, not off.** Verified + safe before defaulting it on: `VortexWriter.buildCascadeCodecs` (the candidate list every + default write uses) does not include `DeltaEncodingEncoder`/`PatchedEncodingEncoder` — the + only two `unstable`-family encoders vortex-java implements — so no default-config write can + emit an `unstable` encoding through the ordinary cascade path. vortex-java also already + implements 100% of `core`, unlike upstream Rust (which lags "by a few versions" to hedge + against readers not having caught up to encodings *it* just shipped) — so "latest frozen + `core`" carries no analogous risk here and avoids an arbitrary "how many versions to lag" + choice. `withEdition(Edition)` replaces any edition already enabled for that edition's family + (a writer may target at most one edition per family, but multiple families at once); + `withoutEditionGuard()` clears every enabled edition — the explicit escape hatch. + +4. **The guard filters candidates during selection, not just checks after encoding + completes — a mid-implementation course correction.** The original design only checked the + final `EncodeNode` tree in `VortexWriter.registerEncodingIds` and threw on a violation. Before + shipping that, tracing actual reachability surfaced a real gap: `fastlanes.delta` is a + hardcoded candidate inside `SparseEncodingEncoder.INDEX_CASCADE_CANDIDATES`, used by + `MaskedEncodingEncoder.encodeValidity` — the code path that compresses **the validity mask of + any nullable column** — whenever `WriteOptions#allowedCascading() > 0` (i.e. + `WriteOptions.cascading(depth)`, an ordinary, encouraged configuration, not just the explicit + custom-encoder-list overload the earlier "safe by default" check had audited). A pure + after-the-fact check would turn a case that works fine today (silently using + `fastlanes.delta`) into a hard write failure the moment the default guard shipped — not a + graceful degradation. + + The fix reuses a mechanism `CascadingCompressor` already had for an unrelated purpose: + `EncodeContext.excluded()` is consulted at every one of its candidate-selection sites + (`competeAndEncode`, `measureBestChild`, `findPrimitiveEncoding`), including inside nested + competitions like the one above, since `ctx` threads all the way down. `VortexWriter`'s + constructor now precomputes `editionExcluded` (the concrete encoder ids this writer holds that + fall outside the configured edition(s)' cumulative union) once, and seeds it into every + `EncodeContext` built in `writeSegment` via two new `EncodeContext.of(...)`/`ofDepth(...)` + overloads taking an initial exclusion set. `CascadingCompressor` then simply never considers + an out-of-edition candidate, falling back to the best remaining one — the guard becomes + invisible in the common case instead of a crash. + + The original after-the-fact check in `registerEncodingIds` is kept as a **backstop**, not + removed: it still fires for any selection path that does not consult + `EncodeContext#excluded()` at all (e.g. a forced `encodingOverride` via the explicit + custom-encoder-list overload with a single out-of-edition candidate, or + `MaskedEncodingEncoder#encodeValidity`'s own top-level Sparse/RunEnd/Rle/Bool choice — though + those three are all `core`-family, so this never fires there in practice). "Never silently + write a file outside the declared edition" needs a guarantee that holds regardless of which + selection path was taken, not just the ones that happen to filter gracefully. + + **A caveat surfaced while building the test for this**: empirically, + `DeltaEncodingEncoder`'s current implementation stores its transposed bases and deltas at full + `ptype` width with no further cascading into bit-packing, so it never actually wins a real + size competition against raw `vortex.primitive` storage — the concrete regression this course + correction defends against does not appear to be reachable with today's encoder + implementations. The filtering design is still the right one regardless: it is not contingent + on any particular encoder's current cost profile, and a future `unstable` encoder (or a future + improvement to `DeltaEncodingEncoder` itself) could make this exact path live at any time + without anyone revisiting the guard. + +5. **Reader UX**: `ReadRegistry`'s two "no decoder registered" throw sites (`decode`, + `decodeAsSegment`) now consult `Editions.owningEdition(id)` before throwing — naming the + edition and its `minVortexVersion` (or "draft, no compatibility guarantee" if none) when the + id is known to some edition, or saying it is unknown to all editions and pointing at + `allowUnknown()` otherwise. Both sites also switch to `VortexException`'s attributed + `(EncodingId, String)` constructor (which prefixes the id), dropping the now-redundant id from + the message body — a small, unrelated bug the same lines already had. + +6. **No wire-format changes.** The targeted edition(s) are a write-time-only policy check — + matches upstream's own model exactly ("stored on the writer's session," not a file field). + Resolves the issue's own open question ("persist edition into file metadata, or write-time + only?"): don't. No `.fbs`/`.proto` schema touched anywhere in this feature. + +## Consequences + +### Positive + +- A write can no longer silently emit an encoding outside its declared compatibility guarantee — + the failure (or graceful fallback) happens at write time, in the writer's own process, instead + of surfacing as a confusing read error in someone else's environment later. +- The default (`core2026.07.0`) is a zero-cost guardrail today: it changes no default write's + output, verified by full reactor build + unit + integration test runs before and after. +- The reader's unknown-encoding error becomes actionable ("this needs vortex >= 0.70.0") instead + of a bare, unhelpful id string. +- `EncodeContext.excluded()`'s reuse for edition filtering required zero changes to + `CascadingCompressor`, `SparseEncodingEncoder`, or `MaskedEncodingEncoder` — the mechanism + already generalized correctly the moment it was seeded from a new source. + +### Negative + +- `WriteOptions` gaining an 8th record component required mechanical (behavior-preserving) + updates at 29 direct-constructor call sites across 12 files, plus 9 existing test call sites + that deliberately exercise `unstable`-family encoders through the explicit-encoder-list + overload and needed `.withoutEditionGuard()` added. +- Two guard mechanisms (graceful filtering + after-the-fact backstop) is more moving parts than a + single check, and the split is only obvious from having traced the actual reachability gap — + a future contributor extending the guard needs to understand both halves. + +### Risks to manage + +- The filtering half depends on every future nested candidate-selection path continuing to + thread `EncodeContext` through and consult `excluded()`, the way `CascadingCompressor` already + does everywhere. A future encoder that runs its own ad hoc cost competition without accepting + or forwarding `ctx` would silently bypass the graceful-fallback half (though the backstop check + in `registerEncodingIds` still catches the final result either way). +- `Edition`'s "only construct via `Editions`'s catalog constants" rule is documentation, not a + compiler guarantee (see Decision #2) — a future reviewer must catch a hand-rolled `Edition` + in review rather than relying on the type system. +- The catalog is a point-in-time mirror of upstream's frozen/draft editions; if upstream freezes + a new `core` edition or adds `unstable` cohorts, this file needs a manual re-sync — there is no + automated check that it stays current (deliberately: this repo does not own the upstream + registry). + +## Alternatives considered + +- **After-the-fact check only, no filtering.** The original design. Rejected after tracing + `MaskedEncodingEncoder`'s validity-mask cascade: it would turn an ordinary `cascading(depth)` + write that happens to prefer an out-of-edition encoding for a validity mask into a hard + failure, not a graceful degradation to the next-best in-edition candidate. +- **`EditionFamily` as a sealed interface with a `Custom` fallback**, mirroring + `EncodingId`/`LayoutId`. Rejected: unlike a custom encoding id, a custom edition *family* + carries no real cross-implementation compatibility guarantee — no other reader in the + ecosystem would recognize it — so there is no legitimate use case for one, and the closed + `core`/`unstable` enum is simpler and matches what is actually true. +- **Lag the default edition by a few versions**, mirroring upstream's own default. Rejected: + upstream lags to hedge against its own reader ecosystem not having caught up to encodings it + *just* shipped; vortex-java implements 100% of `core` already, so there is no analogous risk to + hedge against, and "lag by how many versions" has no principled answer here. +- **Persist the targeted edition into the file footer.** Rejected: the actual compatibility + guarantee is always re-derivable after the fact from the encoding ids already present in the + footer plus the (client-side, shared) edition catalog — nothing new needs to be known that + isn't already computable from existing file content, so there is nothing to gain from a new + wire field, only a schema change to maintain. + +## References + +- Upstream PR: [vortex-data/vortex#8871](https://github.com/vortex-data/vortex/pull/8871) +- Editions spec: +- Issue: [#301](https://github.com/dfa1/vortex-java/issues/301) diff --git a/adr/ADR.md b/adr/ADR.md index dbc99f7c..28f99067 100644 --- a/adr/ADR.md +++ b/adr/ADR.md @@ -37,3 +37,4 @@ the decision shipped in (blank = not yet shipped). | 0020 | Jazzer fuzz testing infrastructure | Proposed | | | 0021 | Cardinality-bounded global dict buffering | Accepted | | | 0022 | Extract FSST into a standalone module, ported faithfully from the paper | Accepted | | +| 0023 | Adopt the Vortex editions model as a client-side write/read policy | Accepted | | diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java index 00981088..7abd3dc3 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java @@ -49,7 +49,7 @@ private SchemaPlus tableOf(long[] values, boolean[] valid) throws IOException { List.of(ColumnName.of("v")), List.of(new DType.Primitive(PType.I64, true)), false); Path file = tmp.resolve("sum-nulls.vortex"); // Large chunk so the whole column is one chunk; zone maps on so the SUM stat is emitted. - WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var writer = VortexWriter.create(ch, schema, opts)) { writer.writeChunk(Map.of(ColumnName.of("v"), new NullableData(values, valid))); diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereBoundaryTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereBoundaryTest.java index 2c2de0b3..44c19298 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereBoundaryTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereBoundaryTest.java @@ -57,7 +57,7 @@ static void write() throws Exception { .build(); // enableZoneMaps=true emits the per-chunk min/max/sum/null-count the tier-1 fold reads and the // classify() step uses to find the boundary zones. - WriteOptions opts = new WriteOptions(CHUNK, true, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(CHUNK, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); VortexWriter writer = VortexWriter.create(ch, schema, opts)) { for (int c = 0; c < CHUNKS; c++) { @@ -456,7 +456,7 @@ private static Ground reduce(java.util.function.LongPredicate predicate) { private static void writeChunks(Path file, DType.Struct schema, Map chunk0, Map chunk1) throws Exception { // chunkSize large so each writeChunk is exactly one chunk (one zone). - WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); VortexWriter writer = VortexWriter.create(ch, schema, opts)) { writer.writeChunk(chunk0); diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereCleanPartitionTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereCleanPartitionTest.java index 494ab5cc..9fb21012 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereCleanPartitionTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereCleanPartitionTest.java @@ -56,7 +56,7 @@ static void write() throws Exception { .field("val", DType.I64) .build(); // enableZoneMaps=true emits the per-chunk min/max/sum/null-count the fold reads. - WriteOptions opts = new WriteOptions(CHUNK, true, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(CHUNK, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); VortexWriter writer = VortexWriter.create(ch, schema, opts)) { for (int c = 0; c < CHUNKS; c++) { @@ -429,7 +429,7 @@ private static Path nullPartitionedFile(String name) throws Exception { private static void writeChunks(Path file, DType.Struct schema, Map chunk0, Map chunk1) throws Exception { // chunkSize large so each writeChunk is exactly one chunk (one zone). - WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); VortexWriter writer = VortexWriter.create(ch, schema, opts)) { writer.writeChunk(chunk0); diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcGenerator.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcGenerator.java index 80bd3ea8..34f0a979 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcGenerator.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcGenerator.java @@ -20,7 +20,7 @@ private OhlcGenerator() { } static void write(Path file, int totalRows, int chunkSize) throws IOException { - WriteOptions opts = new WriteOptions(chunkSize, true, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(chunkSize, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var writer = VortexWriter.create(ch, OhlcData.SCHEMA, opts)) { for (OhlcData.Batch batch : OhlcData.generate(totalRows, chunkSize)) { diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/UnsignedColumnTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/UnsignedColumnTest.java index 6619e428..d40dfacd 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/UnsignedColumnTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/UnsignedColumnTest.java @@ -110,7 +110,7 @@ private Path write(String name, WriteOptions opts, int u32, long u64) throws Exc private static WriteOptions noZoneMaps() { // Same shape as the adapter coverage test's zone-maps-off options: the second flag disables // zone maps so no per-zone SUM exists and VortexAggregates falls back to scanSum. - return new WriteOptions(65_536, false, 0.90, 0, true, false, 256L * 1024 * 1024); + return new WriteOptions(65_536, false, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); } private static ReadRegistry registry() { diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/VortexAdapterCoverageTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/VortexAdapterCoverageTest.java index f173921b..c9bd95e4 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/VortexAdapterCoverageTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/VortexAdapterCoverageTest.java @@ -278,7 +278,7 @@ void nonNumericColumn_throws() throws Exception { void noZoneMap_sumFallsBackToFullScan(@TempDir Path noStats) throws Exception { // Given — a file written with zone maps off, so no per-zone SUM exists to fold Path bare = noStats.resolve("nostats.vortex"); - WriteOptions noZoneMaps = new WriteOptions(65_536, false, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions noZoneMaps = new WriteOptions(65_536, false, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); try (var ch = FileChannel.open(bare, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var w = VortexWriter.create(ch, SCHEMA, noZoneMaps)) { w.writeChunk(Map.ofEntries( diff --git a/core/src/main/java/io/github/dfa1/vortex/core/model/Edition.java b/core/src/main/java/io/github/dfa1/vortex/core/model/Edition.java new file mode 100644 index 00000000..1b6c2cb8 --- /dev/null +++ b/core/src/main/java/io/github/dfa1/vortex/core/model/Edition.java @@ -0,0 +1,40 @@ +package io.github.dfa1.vortex.core.model; + +import java.util.Optional; +import java.util.Set; + +/// A Vortex edition: a named, frozen (or draft) set of encodings that join its family at this +/// point, carrying a forever read-compatibility guarantee once frozen. +/// +/// `added` is only the encodings that join *at this exact edition*, not the family's cumulative +/// set — [Editions#cumulativeMembers(Edition)] computes that. +/// +/// The only `Edition` instances a caller should ever construct or pass to the writer module's +/// `WriteOptions#withEdition(Edition)` are [Editions]'s 8 catalog constants, mirroring the real, +/// frozen Vortex spec — an `Edition` fabricated with an invented member set would carry no actual +/// compatibility guarantee (nothing else in the ecosystem would recognize it). The canonical +/// constructor can't be restricted below `public` (a record's constructor can't be more +/// restrictive than the record type itself, and `Edition` must stay `public` since the writer +/// module — which depends on `core` but not vice versa — references it in its own public API), so +/// this is an API contract enforced by convention and documentation, not the compiler. +/// +/// @param id the edition identifier +/// @param minVortexVersion the minimum Vortex release whose reader supports every encoding in +/// this edition, or empty if this edition is a draft (see [#isDraft()]) +/// @param added the encodings that join the family at this edition +public record Edition(EditionId id, Optional minVortexVersion, Set added) { + + /// Defensively copies `added` into an immutable set. + public Edition { + added = Set.copyOf(added); + } + + /// An edition is a **draft** until its [#minVortexVersion] has been recorded — recording it is + /// the act of freezing. A draft carries no read-compatibility guarantee and is never a default + /// write target. + /// + /// @return `true` if this edition has no recorded `minVortexVersion` + public boolean isDraft() { + return minVortexVersion.isEmpty(); + } +} diff --git a/core/src/main/java/io/github/dfa1/vortex/core/model/EditionFamily.java b/core/src/main/java/io/github/dfa1/vortex/core/model/EditionFamily.java new file mode 100644 index 00000000..49675570 --- /dev/null +++ b/core/src/main/java/io/github/dfa1/vortex/core/model/EditionFamily.java @@ -0,0 +1,24 @@ +package io.github.dfa1.vortex.core.model; + +/// The Vortex edition families. +/// +/// Unlike [EncodingId]/[LayoutId], this is a closed enum, not a sealed +/// interface with a `Custom` fallback: an edition family is a cross-implementation +/// compatibility promise ("any reader supporting `core2026.07.0` can read this file"), so a +/// private, single-writer "family" would carry no real guarantee — nothing else in the Vortex +/// ecosystem would recognize it. A caller who wants to write encodings outside any known family +/// uses the writer module's `WriteOptions#withoutEditionGuard()`, not a fabricated family. +/// +/// Deliberately does not override `toString()`: an enum's `toString()` returning anything other +/// than its declared constant name is surprising (it breaks the usual assumption that +/// `enum.toString().equals(enum.name())`, and would make `valueOf(family.toString())` fail). +/// [EditionId#toString()] is the one place that needs the lowercase wire form and computes it +/// explicitly. +public enum EditionFamily { + /// The `core` family: encodings the default writer emits, each edition frozen with a forever + /// read-compatibility guarantee. + CORE, + /// The `unstable` family: opt-in encodings with no compatibility guarantee — every `unstable` + /// edition is a draft. + UNSTABLE +} diff --git a/core/src/main/java/io/github/dfa1/vortex/core/model/EditionId.java b/core/src/main/java/io/github/dfa1/vortex/core/model/EditionId.java new file mode 100644 index 00000000..a263c630 --- /dev/null +++ b/core/src/main/java/io/github/dfa1/vortex/core/model/EditionId.java @@ -0,0 +1,57 @@ +package io.github.dfa1.vortex.core.model; + +import java.util.Locale; + +/// Identifier of a Vortex [edition](https://github.com/vortex-data/vortex/blob/develop/docs/specs/editions.md), +/// e.g. `core2026.07.0`. +/// +/// `family` names an independently versioned, additive group of encodings (`core` is the set the +/// default writer emits); the date components record when the edition was frozen and order +/// editions chronologically *within* a family — there is no ordering across families. +/// +/// @param family the edition family +/// @param year the four-digit year the edition was cut +/// @param month the month the edition was cut (1-12) +/// @param version distinguishes editions cut in the same month; normally `0` +public record EditionId(EditionFamily family, int year, int month, int version) { + + /// Validates the identifier's form: a four-digit year and a month in 1-12 (`family` needs no + /// validation of its own — [EditionFamily] is a closed enum). Editions are a client-side + /// write/read policy, never derived from untrusted file bytes, so a malformed value here is a + /// programmer error — hence [IllegalArgumentException], not + /// [io.github.dfa1.vortex.core.error.VortexException] (which is reserved for untrusted-input + /// parsing failures). + /// + /// @throws IllegalArgumentException if `year` or `month` is malformed + public EditionId { + if (year < 1000 || year > 9999) { + throw new IllegalArgumentException("edition year must be four digits, got: " + year); + } + if (month < 1 || month > 12) { + throw new IllegalArgumentException("edition month must be in 1-12, got: " + month); + } + } + + /// Returns `true` if `this` is the same edition as `other` or an earlier edition of the same + /// family. Editions of different families are never ordered. + /// + /// @param other the edition id to compare against + /// @return `true` if `this` is at or before `other` within the same family + public boolean isAtOrBefore(EditionId other) { + if (!family.equals(other.family)) { + return false; + } + if (year != other.year) { + return year < other.year; + } + if (month != other.month) { + return month < other.month; + } + return version <= other.version; + } + + @Override + public String toString() { + return "%s%d.%02d.%d".formatted(family.name().toLowerCase(Locale.ROOT), year, month, version); + } +} diff --git a/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java b/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java new file mode 100644 index 00000000..7ad6d8e5 --- /dev/null +++ b/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java @@ -0,0 +1,139 @@ +package io.github.dfa1.vortex.core.model; + +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Optional; +import java.util.Set; + +/// Catalog of Vortex [Edition]s, seeded from the ground-truth declarations in +/// [vortex-data/vortex#8871](https://github.com/vortex-data/vortex/pull/8871) (the published +/// [editions spec](https://github.com/vortex-data/vortex/blob/develop/docs/specs/editions.md)'s +/// own "Edition registry" section is not yet populated upstream, so this catalog was built +/// directly from the Rust source: `vortex-edition/src/lib.rs` and +/// `vortex/src/editions/{core,unstable}/*.rs`). +/// +/// Two families exist today: `core` (frozen, the default writer target) and `unstable` (draft, +/// opt-in, no compatibility guarantee). Membership is additive — an edition's full member set is +/// the union of everything it and every earlier edition of the same family added; see +/// [#cumulativeMembers(Edition)]. +/// +/// vortex-java implements every `core`-family encoding through `core2026.07.0`. Of `unstable`, it +/// implements only `fastlanes.delta` and `vortex.patched` — the remaining ids +/// (`vortex.zstd_buffers`, `vortex.parquet.variant`, the `vortex.tensor.*` family, +/// `vortex.onpair`) have no [EncodingId.WellKnown] constant yet and resolve to +/// [EncodingId.Custom] via [EncodingId#parse(String)], which the catalog stores uniformly with +/// the implemented ids — this catalog mirrors upstream faithfully rather than being truncated to +/// what is implemented today. +public final class Editions { + + /// The baseline `core` edition: stable encodings writable by Vortex 0.36.0. + public static final Edition CORE_2025_05_0 = new Edition( + new EditionId(EditionFamily.CORE, 2025, 5, 0), + Optional.of("0.36.0"), + idSet( + "fastlanes.bitpacked", "fastlanes.for", + "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", + "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", + "vortex.dict", "vortex.ext", "vortex.fsst", "vortex.list", "vortex.null", + "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", + "vortex.varbinview", "vortex.zigzag")); + + /// The `core` edition adding stable encodings released through June 2025. + public static final Edition CORE_2025_06_0 = new Edition( + new EditionId(EditionFamily.CORE, 2025, 6, 0), + Optional.of("0.40.0"), + idSet("vortex.pco", "vortex.sequence", "vortex.zstd")); + + /// The `core` edition adding stable encodings released through October 2025. + public static final Edition CORE_2025_10_0 = new Edition( + new EditionId(EditionFamily.CORE, 2025, 10, 0), + Optional.of("0.54.0"), + idSet("fastlanes.rle", "vortex.fixed_size_list", "vortex.listview", "vortex.masked")); + + /// The `core` edition adding stable encodings released through July 2026. + public static final Edition CORE_2026_07_0 = new Edition( + new EditionId(EditionFamily.CORE, 2026, 7, 0), + Optional.of("0.65.0"), + idSet("vortex.variant")); + + /// The May 2025 draft edition of the `unstable` family. + public static final Edition UNSTABLE_2025_05_0 = new Edition( + new EditionId(EditionFamily.UNSTABLE, 2025, 5, 0), + Optional.empty(), + idSet("fastlanes.delta")); + + /// The February 2026 draft edition of the `unstable` family. + public static final Edition UNSTABLE_2026_02_0 = new Edition( + new EditionId(EditionFamily.UNSTABLE, 2026, 2, 0), + Optional.empty(), + idSet("vortex.zstd_buffers")); + + /// The April 2026 draft edition of the `unstable` family. + public static final Edition UNSTABLE_2026_04_0 = new Edition( + new EditionId(EditionFamily.UNSTABLE, 2026, 4, 0), + Optional.empty(), + idSet( + "vortex.parquet.variant", "vortex.patched", + "vortex.tensor.cosine_similarity", "vortex.tensor.inner_product", + "vortex.tensor.l2_denorm", "vortex.tensor.l2_norm")); + + /// The June 2026 draft edition of the `unstable` family. + public static final Edition UNSTABLE_2026_06_0 = new Edition( + new EditionId(EditionFamily.UNSTABLE, 2026, 6, 0), + Optional.empty(), + idSet("vortex.onpair")); + + /// Every declared edition, in the order above. Order matters: [#owningEdition(EncodingId)] + /// returns the first entry whose `added` set contains the queried id. + public static final List ALL = List.of( + CORE_2025_05_0, CORE_2025_06_0, CORE_2025_10_0, CORE_2026_07_0, + UNSTABLE_2025_05_0, UNSTABLE_2026_02_0, UNSTABLE_2026_04_0, UNSTABLE_2026_06_0); + + private Editions() { + } + + /// Computes `edition`'s full, cumulative member set: its own `added` encodings plus every + /// encoding added by an earlier edition of the same family in [#ALL]. Seeding the result with + /// `edition.added()` itself (rather than relying solely on a scan of [#ALL]) is what makes this + /// correct for a caller-supplied `Edition` outside the catalog too, not just the 8 built-ins — + /// such a custom edition has no known earlier edition to accumulate from, so its cumulative set + /// is exactly its own. + /// + /// @param edition the edition whose cumulative membership to compute + /// @return the union of `edition`'s own additions and every earlier same-family edition's + public static Set cumulativeMembers(Edition edition) { + Set result = new LinkedHashSet<>(edition.added()); + for (Edition candidate : ALL) { + if (candidate.id().family().equals(edition.id().family()) + && candidate.id().isAtOrBefore(edition.id())) { + result.addAll(candidate.added()); + } + } + return Set.copyOf(result); + } + + /// Returns the edition `id` first joined, if it is part of any declared edition. + /// + /// Correct because membership is additive and each encoding belongs to exactly one family: + /// the first entry in [#ALL] whose `added` set contains `id` is unambiguously the edition it + /// joined at. + /// + /// @param id the encoding id to look up + /// @return the edition `id` first joined, or empty if `id` is not part of any declared edition + public static Optional owningEdition(EncodingId id) { + for (Edition candidate : ALL) { + if (candidate.added().contains(id)) { + return Optional.of(candidate); + } + } + return Optional.empty(); + } + + private static Set idSet(String... rawIds) { + Set ids = new LinkedHashSet<>(rawIds.length); + for (String rawId : rawIds) { + ids.add(EncodingId.parse(rawId)); + } + return Set.copyOf(ids); + } +} diff --git a/core/src/test/java/io/github/dfa1/vortex/core/model/EditionIdTest.java b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionIdTest.java new file mode 100644 index 00000000..51c2f4cb --- /dev/null +++ b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionIdTest.java @@ -0,0 +1,143 @@ +package io.github.dfa1.vortex.core.model; + +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +class EditionIdTest { + + @Nested + class ToString { + + @Test + void toString_zeroPadsMonthButNotVersion() { + // Given — month 5 must render as "05", version 0 must render as "0" (not "00") + EditionId sut = new EditionId(EditionFamily.CORE, 2025, 5, 0); + + // When + String result = sut.toString(); + + // Then + assertThat(result).isEqualTo("core2025.05.0"); + } + + @Test + void toString_doubleDigitMonthAndNonZeroVersion() { + // Given + EditionId sut = new EditionId(EditionFamily.UNSTABLE, 2026, 11, 2); + + // When + String result = sut.toString(); + + // Then + assertThat(result).isEqualTo("unstable2026.11.2"); + } + } + + @Nested + class IsAtOrBefore { + + @Test + void isAtOrBefore_earlierYear_isTrue() { + // Given + EditionId earlier = new EditionId(EditionFamily.CORE, 2025, 5, 0); + EditionId later = new EditionId(EditionFamily.CORE, 2026, 7, 0); + + // When + boolean result = earlier.isAtOrBefore(later); + + // Then + assertThat(result).isTrue(); + } + + @Test + void isAtOrBefore_sameEdition_isTrue() { + // Given + EditionId sut = new EditionId(EditionFamily.CORE, 2025, 5, 0); + + // When + boolean result = sut.isAtOrBefore(sut); + + // Then + assertThat(result).isTrue(); + } + + @Test + void isAtOrBefore_laterEdition_isFalse() { + // Given + EditionId later = new EditionId(EditionFamily.CORE, 2026, 7, 0); + EditionId earlier = new EditionId(EditionFamily.CORE, 2025, 5, 0); + + // When + boolean result = later.isAtOrBefore(earlier); + + // Then + assertThat(result).isFalse(); + } + + @Test + void isAtOrBefore_sameYearEarlierMonth_isTrue() { + // Given + EditionId sut = new EditionId(EditionFamily.CORE, 2025, 5, 0); + EditionId other = new EditionId(EditionFamily.CORE, 2025, 6, 0); + + // When / Then + assertThat(sut.isAtOrBefore(other)).isTrue(); + } + + @Test + void isAtOrBefore_sameYearMonthEarlierVersion_isTrue() { + // Given + EditionId sut = new EditionId(EditionFamily.CORE, 2025, 5, 0); + EditionId other = new EditionId(EditionFamily.CORE, 2025, 5, 1); + + // When / Then + assertThat(sut.isAtOrBefore(other)).isTrue(); + } + + @Test + void isAtOrBefore_differentFamily_isAlwaysFalse() { + // Given — unstable2025.05.0 is chronologically "earlier" than core2026.07.0, but + // families are never ordered against each other + EditionId unstable = new EditionId(EditionFamily.UNSTABLE, 2025, 5, 0); + EditionId core = new EditionId(EditionFamily.CORE, 2026, 7, 0); + + // When / Then + assertThat(unstable.isAtOrBefore(core)).isFalse(); + assertThat(core.isAtOrBefore(unstable)).isFalse(); + } + } + + @Nested + class Validation { + + @ParameterizedTest + @ValueSource(ints = {0, 99, 999, 10000}) + void constructor_nonFourDigitYear_throwsIllegalArgumentException(int year) { + // Given / When / Then + assertThatThrownBy(() -> new EditionId(EditionFamily.CORE, year, 5, 0)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("year"); + } + + @ParameterizedTest + @ValueSource(ints = {0, 13, -1}) + void constructor_monthOutOfRange_throwsIllegalArgumentException(int month) { + // Given / When / Then + assertThatThrownBy(() -> new EditionId(EditionFamily.CORE, 2025, month, 0)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("month"); + } + + @Test + void constructor_validValues_doesNotThrow() { + // Given / When / Then + assertThat(new EditionId(EditionFamily.CORE, 2025, 1, 0)).isNotNull(); + assertThat(new EditionId(EditionFamily.UNSTABLE, 9999, 12, 99)).isNotNull(); + } + } +} diff --git a/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java new file mode 100644 index 00000000..f72b9221 --- /dev/null +++ b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java @@ -0,0 +1,206 @@ +package io.github.dfa1.vortex.core.model; + +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +import java.util.Optional; +import java.util.Set; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static org.assertj.core.api.Assertions.assertThat; + +/// Golden tests pinning the ground-truth edition data from +/// [vortex-data/vortex#8871](https://github.com/vortex-data/vortex/pull/8871): a stray edit to +/// [Editions]'s declarations that changes a cumulative member set fails here, mirroring upstream's +/// own `validate_edition` test harness. +class EditionsTest { + + @Nested + class CumulativeMembers { + + @Test + void core2025_05_0_isExactlyItsOwnBaseline() { + // Given the first core edition — no earlier core edition to accumulate from + // When + Set result = Editions.cumulativeMembers(Editions.CORE_2025_05_0); + + // Then + assertThat(result).isEqualTo(ids( + "fastlanes.bitpacked", "fastlanes.for", + "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", + "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", + "vortex.dict", "vortex.ext", "vortex.fsst", "vortex.list", "vortex.null", + "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", + "vortex.varbinview", "vortex.zigzag")); + } + + @Test + void core2025_06_0_addsToTheBaseline() { + // Given + // When + Set result = Editions.cumulativeMembers(Editions.CORE_2025_06_0); + + // Then — baseline 23 plus pco/sequence/zstd + assertThat(result).isEqualTo(ids( + "fastlanes.bitpacked", "fastlanes.for", + "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", + "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", + "vortex.dict", "vortex.ext", "vortex.fsst", "vortex.list", "vortex.null", + "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", + "vortex.varbinview", "vortex.zigzag", + "vortex.pco", "vortex.sequence", "vortex.zstd")); + } + + @Test + void core2025_10_0_addsToPreviousCore() { + // Given + // When + Set result = Editions.cumulativeMembers(Editions.CORE_2025_10_0); + + // Then — the 26 above plus rle/fixed_size_list/listview/masked + assertThat(result).isEqualTo(ids( + "fastlanes.bitpacked", "fastlanes.for", "fastlanes.rle", + "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", + "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", + "vortex.dict", "vortex.ext", "vortex.fixed_size_list", "vortex.fsst", "vortex.list", + "vortex.listview", "vortex.masked", "vortex.null", + "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", + "vortex.varbinview", "vortex.zigzag", + "vortex.pco", "vortex.sequence", "vortex.zstd")); + } + + @Test + void core2026_07_0_isTheFullCoreSet() { + // Given — the latest frozen core edition + // When + Set result = Editions.cumulativeMembers(Editions.CORE_2026_07_0); + + // Then — every core encoding through vortex.variant; 31 total + assertThat(result).hasSize(31) + .isEqualTo(ids( + "fastlanes.bitpacked", "fastlanes.for", "fastlanes.rle", + "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", + "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", + "vortex.dict", "vortex.ext", "vortex.fixed_size_list", "vortex.fsst", "vortex.list", + "vortex.listview", "vortex.masked", "vortex.null", + "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", + "vortex.varbinview", "vortex.zigzag", + "vortex.pco", "vortex.sequence", "vortex.zstd", + "vortex.variant")); + } + + @Test + void unstable2025_05_0_isExactlyItsOwnAddition() { + // Given the first unstable edition — no earlier unstable edition to accumulate from + // When + Set result = Editions.cumulativeMembers(Editions.UNSTABLE_2025_05_0); + + // Then + assertThat(result).isEqualTo(ids("fastlanes.delta")); + } + + @Test + void unstable2026_02_0_addsToThePreviousUnstable() { + // Given + // When + Set result = Editions.cumulativeMembers(Editions.UNSTABLE_2026_02_0); + + // Then + assertThat(result).isEqualTo(ids("fastlanes.delta", "vortex.zstd_buffers")); + } + + @Test + void unstable2026_04_0_addsToPreviousUnstable() { + // Given + // When + Set result = Editions.cumulativeMembers(Editions.UNSTABLE_2026_04_0); + + // Then + assertThat(result).isEqualTo(ids( + "fastlanes.delta", "vortex.zstd_buffers", + "vortex.parquet.variant", "vortex.patched", + "vortex.tensor.cosine_similarity", "vortex.tensor.inner_product", + "vortex.tensor.l2_denorm", "vortex.tensor.l2_norm")); + } + + @Test + void unstable2026_06_0_isTheFullUnstableSet() { + // Given — the latest draft unstable edition + // When + Set result = Editions.cumulativeMembers(Editions.UNSTABLE_2026_06_0); + + // Then + assertThat(result).hasSize(9) + .isEqualTo(ids( + "fastlanes.delta", "vortex.zstd_buffers", + "vortex.parquet.variant", "vortex.patched", + "vortex.tensor.cosine_similarity", "vortex.tensor.inner_product", + "vortex.tensor.l2_denorm", "vortex.tensor.l2_norm", + "vortex.onpair")); + } + + @Test + void editionNotInAll_cumulativeSeedsFromItsOwnAddedRatherThanRequiringIdentityInAll() { + // Given — a hypothetical future core edition not (yet) declared in Editions.ALL. + // Edition's constructor is package-private (only Editions' 8 catalog constants exist + // in production), so this exercises cumulativeMembers' seeding logic directly rather + // than a reachable real-world scenario. + Edition hypothetical = new Edition( + new EditionId(EditionFamily.CORE, 2099, 1, 0), Optional.of("9.0.0"), + Set.of(EncodingId.parse("vortex.future"))); + + // When + Set result = Editions.cumulativeMembers(hypothetical); + + // Then — its own addition, plus one member from every earlier core edition + assertThat(result).contains( + EncodingId.parse("vortex.future"), // its own addition + EncodingId.VORTEX_PRIMITIVE, // core2025.05.0 + EncodingId.VORTEX_ZSTD, // core2025.06.0 + EncodingId.VORTEX_MASKED, // core2025.10.0 + EncodingId.VORTEX_VARIANT); // core2026.07.0 + } + + private Set ids(String... rawIds) { + return Stream.of(rawIds).map(EncodingId::parse).collect(Collectors.toUnmodifiableSet()); + } + } + + @Nested + class OwningEdition { + + @Test + void owningEdition_coreFamilyId_returnsTheEditionItFirstJoined() { + // Given — vortex.zstd first joins at core2025.06.0, not the baseline + // When + Optional result = Editions.owningEdition(EncodingId.VORTEX_ZSTD); + + // Then + assertThat(result).contains(Editions.CORE_2025_06_0); + } + + @Test + void owningEdition_unstableFamilyId_returnsTheDraftEdition() { + // Given + // When + Optional result = Editions.owningEdition(EncodingId.VORTEX_PATCHED); + + // Then + assertThat(result).contains(Editions.UNSTABLE_2026_04_0); + assertThat(result.get().isDraft()).isTrue(); + } + + @Test + void owningEdition_idNotInAnyEdition_returnsEmpty() { + // Given — a genuinely unknown, custom encoding no edition declares + EncodingId unknown = new EncodingId.Custom("acme.widget"); + + // When + Optional result = Editions.owningEdition(unknown); + + // Then + assertThat(result).isEmpty(); + } + } +} diff --git a/docs/compatibility.md b/docs/compatibility.md index 3858ec06..cf89709e 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -173,8 +173,10 @@ Bitpacked produces `LazyAlp(MaterializedXxx)`). ### Unknown encodings -Files containing unrecognized encoding IDs throw `VortexException` by default. Opt in to -passthrough mode to read such files without failing: +Files containing unrecognized encoding IDs throw `VortexException` by default. The message names +which [edition](#editions) the id belongs to and the minimum Vortex version needed (or that it's +an `unstable` draft with no compatibility guarantee), or says the id is unknown to every edition +if it belongs to none. Opt in to passthrough mode to read such files without failing: ```java ReadRegistry registry = ReadRegistry.builder() @@ -186,6 +188,24 @@ try (VortexReader vf = VortexReader.open(path, registry)) { } ``` +## Editions + +Frozen, named, additive sets of encoding IDs, each carrying a forever read-compatibility +guarantee once frozen (ADR 0023) — a write-time/read-time policy, not part of the wire format. +`WriteOptions.defaults()` targets the latest frozen `core` edition; see +[the reference doc](reference.md#editions) for the writer/reader integration. + +| Edition | Family | Min Vortex | Adds | +|---|---|---|---| +| `core2025.05.0` | `core` | 0.36.0 | 23 baseline: `fastlanes.bitpacked`/`for`, `vortex.alp`/`alprd`/`bool`/`bytebool`/`chunked`/`constant`/`datetimeparts`/`decimal`/`decimal_byte_parts`/`dict`/`ext`/`fsst`/`list`/`null`/`primitive`/`runend`/`sparse`/`struct`/`varbin`/`varbinview`/`zigzag` | +| `core2025.06.0` | `core` | 0.40.0 | `vortex.pco`, `vortex.sequence`, `vortex.zstd` | +| `core2025.10.0` | `core` | 0.54.0 | `fastlanes.rle`, `vortex.fixed_size_list`, `vortex.listview`, `vortex.masked` | +| `core2026.07.0` | `core` | 0.65.0 | `vortex.variant` | +| `unstable2025.05.0` | `unstable` | draft | `fastlanes.delta` ✅ implemented | +| `unstable2026.02.0` | `unstable` | draft | `vortex.zstd_buffers` ❌ not implemented | +| `unstable2026.04.0` | `unstable` | draft | `vortex.parquet.variant` ❌, `vortex.patched` ✅, `vortex.tensor.*` (4 ids) ❌ | +| `unstable2026.06.0` | `unstable` | draft | `vortex.onpair` ❌ not implemented | + ## Extension types Extension dtypes wrap a primitive storage array with a logical-id tag plus optional diff --git a/docs/reference.md b/docs/reference.md index 317d65b3..114f6fd2 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -63,6 +63,9 @@ shortcut returning a nullable copy), `withNullable(boolean)`, `DType.Struct.fiel | `EncodingId` | `sealed interface` — `WellKnown` enum + `Custom` record | Array-encoding identity; total `parse(String)` over non-blank ids; constants re-exported (`EncodingId.VORTEX_PRIMITIVE`, …) | | `LayoutId` | `sealed interface` — `WellKnown` enum + `Custom` record | Layout identity (separate namespace from encodings; `vortex.flat` is layout-only); both zoned aliases `vortex.zoned`/`vortex.stats` | | `ColumnName` | `record ColumnName(String value)` | Validated column name: non-blank, no control characters. `ColumnName.violation(String)` is the policy chokepoint shared by builder, writer, and file parser | +| `EditionFamily` | `enum` — `CORE`, `UNSTABLE` | Closed (unlike `EncodingId`/`LayoutId`): an edition family is a cross-implementation compatibility promise, so a custom family carries no real guarantee — see [Editions](#editions) | +| `EditionId` | `record EditionId(EditionFamily family, int year, int month, int version)` | e.g. `core2025.05.0`; `isAtOrBefore` orders editions within a family only | +| `Edition` | `record Edition(EditionId id, Optional minVortexVersion, Set added)` | Only `Editions`'s 8 catalog constants should be constructed (API contract, not compiler-enforced — see ADR 0023) | ## Reader API @@ -126,13 +129,22 @@ Accepted array types per column `DType`: ### `WriteOptions` (`io.github.dfa1.vortex.writer.WriteOptions`) -Record: `(int chunkSize, boolean enableZoneMaps, double compressionRatioThreshold, int allowedCascading)`. +Record: `(int chunkSize, boolean enableZoneMaps, double compressionRatioThreshold, int allowedCascading, boolean globalDict, boolean enableZstd, long globalDictMaxRetainedBytes, Map editions)`. | Factory | Defaults | |---------------------------------|---------------------------------------------------------------------------------------------------| -| `WriteOptions.defaults()` | `chunkSize=65_536`, `enableZoneMaps=true`, `compressionRatioThreshold=0.90`, `allowedCascading=0` | +| `WriteOptions.defaults()` | `chunkSize=65_536`, `enableZoneMaps=true`, `compressionRatioThreshold=0.90`, `allowedCascading=0`, `globalDict=true`, `enableZstd=false`, `globalDictMaxRetainedBytes=2 GB`, `editions={CORE: Editions.CORE_2026_07_0}` | | `WriteOptions.cascading(depth)` | Same defaults, `allowedCascading=depth` | +| Method | Notes | +|--------|-------| +| `withZoneMaps(boolean)` | Toggle per-chunk min/max/sum statistics | +| `withGlobalDict(boolean)` | Toggle the shared cross-chunk dictionary | +| `withZstd(boolean)` | Add Zstandard to the cascade codec competition | +| `withGlobalDictMaxRetainedBytes(long)` | Aggregate heap budget for buffered global-dict candidate columns | +| `withEdition(Edition)` | Enable an [edition](#editions) for its family, replacing any edition already enabled for that family | +| `withoutEditionGuard()` | Clear every enabled edition — the writer may then emit any registered encoding, at the cost of the edition compatibility guarantee | + --- ## Scan API @@ -231,6 +243,57 @@ implementations are singletons invoked directly by their `ExtensionId`. --- +## Editions + +Frozen, named, additive sets of encoding IDs, each carrying a forever read-compatibility +guarantee once frozen (see ADR 0023). vortex-java mirrors the ground-truth catalog from +[vortex-data/vortex#8871](https://github.com/vortex-data/vortex/pull/8871) — the published +[editions spec](https://github.com/vortex-data/vortex/blob/develop/docs/specs/editions.md)'s own +registry section is not populated upstream yet. Editions are a **write/read policy, not part of +the wire format** — nothing about a targeted edition is ever persisted into a `.vortex` file. + +### `Editions` (`io.github.dfa1.vortex.core.model.Editions`) + +| Member | Notes | +|--------------------------------------|------------------------------------------------------------------------------| +| `CORE_2025_05_0` … `CORE_2026_07_0` | The 4 frozen `core` editions (min Vortex 0.36.0 → 0.65.0) | +| `UNSTABLE_2025_05_0` … `UNSTABLE_2026_06_0` | The 4 draft `unstable` editions (no compatibility guarantee) | +| `ALL` | Every declared edition, in declaration order | +| `cumulativeMembers(Edition)` | The edition's own additions plus every earlier same-family edition's | +| `owningEdition(EncodingId)` | The edition an id first joined, or empty if it belongs to none | + +vortex-java implements every `core`-family encoding through `core2026.07.0`. Of `unstable`, only +`fastlanes.delta` and `vortex.patched` have an `EncodingId.WellKnown` constant; the rest +(`vortex.zstd_buffers`, `vortex.parquet.variant`, the `vortex.tensor.*` family, `vortex.onpair`) +resolve to `EncodingId.Custom` and are stored in the catalog anyway, mirroring upstream faithfully. + +### Writer integration (`WriteOptions#editions()`) + +`WriteOptions.defaults()`/`cascading(depth)` enable the latest frozen `core` edition +(`Editions.CORE_2026_07_0`) by default — verified safe: the default cascade candidate list never +includes `DeltaEncodingEncoder`/`PatchedEncodingEncoder` (the only two `unstable`-family encoders +implemented), so no default write can emit an `unstable` encoding. If a write would emit an +encoding outside the union of every enabled edition's cumulative members, `VortexWriter` fails +the write immediately with a `VortexException` naming the encoding and the configured edition(s). +Where possible (any selection routed through `CascadingCompressor`, including nested competitions +like a masked column's validity-bitmap cascade) the guard instead steers selection away from the +ineligible candidate ahead of time, falling back to the best remaining one — see ADR 0023. + +- `WriteOptions.withEdition(Edition)` enables an edition, replacing any edition already enabled + for that edition's family. Multiple families can be enabled at once (e.g. `core` and `unstable` + simultaneously); at most one edition per family. +- `WriteOptions.withoutEditionGuard()` clears every enabled edition — the escape hatch for custom + or experimental encodings, at the cost of the compatibility guarantee. + +### Reader integration + +When `ReadRegistry` hits an unregistered encoding id (and `allowUnknown()` is off), the thrown +`VortexException` names the edition the id belongs to and its minimum required Vortex version +(or that it is a draft with no guarantee), or says the id is unknown to every edition and points +at `allowUnknown()`. + +--- + ## FSST (`io.github.dfa1.vortex.fsst`) The `vortex-fsst` module is the standalone FSST (Fast Static Symbol Table) string-compression diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaRoundTripIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaRoundTripIntegrationTest.java index 85c4934b..55d317c8 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaRoundTripIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaRoundTripIntegrationTest.java @@ -50,7 +50,7 @@ void patched_i32_javaWriteJavaRead(@TempDir Path tmp) throws IOException { data[61] = 7_000_000; data[88] = 8_000_000; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I32_SCHEMA, WriteOptions.defaults(), + var sut = VortexWriter.create(ch, I32_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), List.of(new PatchedEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("v"), data)); diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java index 7b62a23d..c7998ff1 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java @@ -509,7 +509,7 @@ void javaWriter_jniReader_zoneMapped_multipleZones(@TempDir Path tmp) throws IOE // zone-map with one zone per chunk. The Rust reader must parse that layout and still // return every value (zones are a transparent pruning aux). Path file = tmp.resolve("java_zoned.vtx"); - WriteOptions zoneMapped = new WriteOptions(4, true, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions zoneMapped = new WriteOptions(4, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); long[] ids = new long[20]; double[] vals = new double[20]; for (int i = 0; i < 20; i++) { @@ -1046,7 +1046,7 @@ void javaWriter_rustReader_delta_i64(@TempDir Path tmp) throws IOException { data[i] = acc; } try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, TS_SCHEMA, WriteOptions.defaults(), + var sut = VortexWriter.create(ch, TS_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); diff --git a/performance/src/main/java/io/github/dfa1/vortex/performance/CalciteBoundaryAggregateBenchmark.java b/performance/src/main/java/io/github/dfa1/vortex/performance/CalciteBoundaryAggregateBenchmark.java index a78daad5..5896d06d 100644 --- a/performance/src/main/java/io/github/dfa1/vortex/performance/CalciteBoundaryAggregateBenchmark.java +++ b/performance/src/main/java/io/github/dfa1/vortex/performance/CalciteBoundaryAggregateBenchmark.java @@ -194,7 +194,7 @@ private static void writeFixture(Path file) throws IOException { .field("val", DType.I64) .build(); // enableZoneMaps=true emits the per-chunk min/max/sum/null-count the interior-zone fold reads. - WriteOptions opts = new WriteOptions(CHUNK_SIZE, true, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(CHUNK_SIZE, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); java.util.Random rng = new java.util.Random(SEED); try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING); diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java b/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java index c6d5038c..b0a7f3d9 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java @@ -9,6 +9,10 @@ import io.github.dfa1.vortex.core.model.DType; import io.github.dfa1.vortex.core.model.PType; import io.github.dfa1.vortex.core.io.VortexFormat; +import io.github.dfa1.vortex.core.error.VortexException; +import io.github.dfa1.vortex.core.model.Edition; +import io.github.dfa1.vortex.core.model.EditionFamily; +import io.github.dfa1.vortex.core.model.Editions; import io.github.dfa1.vortex.core.model.EncodingId; import io.github.dfa1.vortex.core.model.LayoutId; import io.github.dfa1.vortex.writer.encode.EncodeContext; @@ -66,7 +70,9 @@ import java.util.LinkedHashSet; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.Set; +import java.util.stream.Collectors; /// Writes a Vortex file. /// @@ -136,6 +142,20 @@ private static int dictMaxCardinality(boolean utf8) { private final List segs = new ArrayList<>(); private final Map> colChunks = new LinkedHashMap<>(); private final Map encodingIdx = new LinkedHashMap<>(); + // Edition guard (issue #301): the cumulative member set of every WriteOptions#editions() + // family enabled for this writer, or empty when no edition is configured (guard off). + // editionExcluded is its complement over the concrete encoders this writer actually holds + // (encodings + cascadeCodecs) - seeded into every EncodeContext's initial `excluded` set so + // CascadingCompressor's existing per-candidate exclusion check (already consulted at every + // selection site, including nested competitions like a masked column's validity-bitmap + // cascade) skips ineligible candidates and gracefully falls back, rather than the guard only + // discovering a violation after the fact. registerEncodingIds still checks editionAllowed + // directly as a backstop for paths that select an encoding without consulting excluded at all + // (e.g. a forced encodingOverride, or MaskedEncodingEncoder#encodeValidity's own top-level + // Sparse/RunEnd/Rle/Bool choice, which are all core-family and so never actually trip this in + // practice - see adr/0023-vortex-editions-adoption.md). + private final Set editionAllowed; + private final Set editionExcluded; private long bytesWritten = 0; // Global dict state: columns detected as low-cardinality on their first chunk are buffered here @@ -189,11 +209,49 @@ private VortexWriter( this.defaultRegistry = buildRegistry(encodings); this.cascadeCodecs = buildCascadeCodecs(options); this.cascadeRegistry = buildRegistry(this.cascadeCodecs); + this.editionAllowed = editionAllowed(options.editions()); + this.editionExcluded = editionExcluded(this.editionAllowed, encodings, this.cascadeCodecs); for (ColumnName name : schema.fieldNames()) { colChunks.put(name, new ArrayList<>()); } } + /// The union of every enabled edition's cumulative member set, or empty if no edition is + /// configured — an empty result means the guard is off entirely. + private static Set editionAllowed(Map editions) { + if (editions.isEmpty()) { + return Set.of(); + } + Set allowed = new LinkedHashSet<>(); + for (Edition edition : editions.values()) { + allowed.addAll(Editions.cumulativeMembers(edition)); + } + return Set.copyOf(allowed); + } + + /// The concrete encoder ids this writer actually holds that fall outside `allowed`. Seeding + /// this into every [EncodeContext]'s initial exclusion set lets [CascadingCompressor]'s + /// existing per-candidate check skip them and fall back to the best remaining candidate, + /// instead of the edition guard only surfacing as a hard failure after encoding completes. + private static Set editionExcluded( + Set allowed, List encodings, List cascadeCodecs) { + if (allowed.isEmpty()) { + return Set.of(); + } + Set excluded = new LinkedHashSet<>(); + for (EncodingEncoder enc : encodings) { + if (!allowed.contains(enc.encodingId())) { + excluded.add(enc.encodingId()); + } + } + for (EncodingEncoder enc : cascadeCodecs) { + if (!allowed.contains(enc.encodingId())) { + excluded.add(enc.encodingId()); + } + } + return Set.copyOf(excluded); + } + /// Builds a [WriteRegistry] from the given encoder list plus all built-in extension encoders. private static WriteRegistry buildRegistry(List encoders) { WriteRegistry.Builder b = WriteRegistry.builder(); @@ -650,11 +708,11 @@ private int writeSegment(DType dtype, Object data, EncodingEncoder encodingOverr // fixed first-match encoder. Without cascading, a depth-0 context is passed and the // override behaves as before. EncodeContext encodeCtx = options.allowedCascading() > 0 - ? EncodeContext.ofDepth(options.allowedCascading(), arena, cascadeRegistry) - : EncodeContext.of(arena, defaultRegistry); + ? EncodeContext.ofDepth(options.allowedCascading(), arena, cascadeRegistry, editionExcluded) + : EncodeContext.of(arena, defaultRegistry, editionExcluded); result = encodingOverride.encode(dtype, data, encodeCtx); } else if (options.allowedCascading() > 0) { - EncodeContext encodeCtx = EncodeContext.ofDepth(options.allowedCascading(), arena, cascadeRegistry); + EncodeContext encodeCtx = EncodeContext.ofDepth(options.allowedCascading(), arena, cascadeRegistry, editionExcluded); for (EncodingId excluded : excludedFromCascade) { encodeCtx = encodeCtx.withExcluded(excluded); } @@ -662,7 +720,7 @@ private int writeSegment(DType dtype, Object data, EncodingEncoder encodingOverr result = compressor.encode(dtype, data, encodeCtx); } else { EncodingEncoder encoder = findEncoder(dtype); - EncodeContext encodeCtx = EncodeContext.of(arena, defaultRegistry); + EncodeContext encodeCtx = EncodeContext.of(arena, defaultRegistry, editionExcluded); result = encoder.encode(dtype, data, encodeCtx); } // Register all encoding IDs found in the node tree @@ -701,12 +759,39 @@ private int writeSegment(DType dtype, Object data, EncodingEncoder encodingOverr } private void registerEncodingIds(EncodeNode node) { - encodingIdx.computeIfAbsent(node.encodingId(), _ -> encodingIdx.size()); + EncodingId id = node.encodingId(); + // Edition guard backstop (issue #301): editionExcluded, seeded into every EncodeContext, + // already steers CascadingCompressor away from out-of-edition candidates during selection + // (see the editionExcluded/editionAllowed fields' javadoc) - this direct check catches + // whatever still reaches here anyway: a forced encodingOverride, or a selection path that + // does not consult EncodeContext#excluded() at all. Never silently write a file outside + // the declared edition. + if (!editionAllowed.isEmpty() && !editionAllowed.contains(id)) { + throw editionViolation(id); + } + encodingIdx.computeIfAbsent(id, _ -> encodingIdx.size()); for (EncodeNode child : node.children()) { registerEncodingIds(child); } } + private VortexException editionViolation(EncodingId id) { + String configured = options.editions().values().stream() + .map(e -> e.id().toString()) + .sorted() + .collect(Collectors.joining(", ")); + Optional owning = Editions.owningEdition(id); + String hint = owning.isPresent() + ? "; it joins " + owning.get().id() + + (owning.get().isDraft() + ? " (draft, no compatibility guarantee)" + : " (requires reader >= " + owning.get().minVortexVersion().get() + ")") + + " — enable it via WriteOptions.withEdition(...)" + : "; it is not part of any known edition"; + return new VortexException(id, "outside the configured edition(s) [" + configured + "]" + hint + + ", or call WriteOptions.withoutEditionGuard() to opt out"); + } + private EncodingEncoder findEncoder(DType dtype) { for (EncodingEncoder c : encodings) { if (c.accepts(dtype)) { diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/WriteOptions.java b/writer/src/main/java/io/github/dfa1/vortex/writer/WriteOptions.java index b99740df..bb346f02 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/WriteOptions.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/WriteOptions.java @@ -1,5 +1,12 @@ package io.github.dfa1.vortex.writer; +import io.github.dfa1.vortex.core.model.Edition; +import io.github.dfa1.vortex.core.model.EditionFamily; +import io.github.dfa1.vortex.core.model.Editions; + +import java.util.HashMap; +import java.util.Map; + /// Tuning knobs for the Vortex writer. /// /// @param chunkSize target row count per chunk (default 65 536) @@ -28,6 +35,13 @@ /// cap. Trade-off: demoted columns lose the shared-dictionary size /// benefit; raise the budget on memory-rich hosts to keep more wide, /// low-cardinality columns dictionary-encoded. +/// @param editions the [Edition] enabled per family, gating which +/// encodings this writer may emit — see [#withEdition(Edition)]. Defaults to +/// the latest frozen `core` edition ([Editions#CORE_2026_07_0]): vortex-java +/// implements every `core`-family encoding, and the default cascade never +/// selects an `unstable`-family one, so this is a zero-cost guardrail against +/// ever silently widening a file's minimum required reader. Cleared entirely by +/// [#withoutEditionGuard()]. public record WriteOptions( int chunkSize, boolean enableZoneMaps, @@ -35,8 +49,14 @@ public record WriteOptions( int allowedCascading, boolean globalDict, boolean enableZstd, - long globalDictMaxRetainedBytes + long globalDictMaxRetainedBytes, + Map editions ) { + /// Defensively copies `editions` into an immutable map. + public WriteOptions { + editions = Map.copyOf(editions); + } + /// Default aggregate retention budget (2 GB) for the buffered per-chunk code arrays of global /// -dictionary candidate columns. Raised from 256 MB when buffering became cardinality-bounded /// (ADR 0021), then from 1 GB (#303): a wide, high-cardinality file (NYC 311, ~30 admitted string @@ -47,11 +67,17 @@ public record WriteOptions( /// [#withGlobalDictMaxRetainedBytes(long)]. private static final long DEFAULT_GLOBAL_DICT_MAX_RETAINED_BYTES = 2L * 1024 * 1024 * 1024; - /// Default options: global dictionary encoding enabled, no cascading compression, Zstd disabled. + /// The default edition guard: only the latest frozen `core` edition enabled. See the `editions` + /// parameter's javadoc above for the safety rationale. + private static final Map DEFAULT_EDITIONS = Map.of(EditionFamily.CORE, Editions.CORE_2026_07_0); + + /// Default options: global dictionary encoding enabled, no cascading compression, Zstd disabled, + /// edition guard targeting the latest frozen `core` edition. /// /// @return default `WriteOptions` public static WriteOptions defaults() { - return new WriteOptions(65_536, true, 0.90, 0, true, false, DEFAULT_GLOBAL_DICT_MAX_RETAINED_BYTES); + return new WriteOptions(65_536, true, 0.90, 0, true, false, DEFAULT_GLOBAL_DICT_MAX_RETAINED_BYTES, + DEFAULT_EDITIONS); } /// Enable cascading compression with up to `depth` recursive levels. @@ -60,7 +86,8 @@ public static WriteOptions defaults() { /// @param depth maximum cascade depth /// @return `WriteOptions` with cascading enabled at the given depth public static WriteOptions cascading(int depth) { - return new WriteOptions(65_536, true, 0.90, depth, true, false, DEFAULT_GLOBAL_DICT_MAX_RETAINED_BYTES); + return new WriteOptions(65_536, true, 0.90, depth, true, false, DEFAULT_GLOBAL_DICT_MAX_RETAINED_BYTES, + DEFAULT_EDITIONS); } /// Returns a copy of these options with zone-map statistics set to `enabled`. @@ -69,7 +96,7 @@ public static WriteOptions cascading(int depth) { /// @return a new `WriteOptions` with the zone-map flag updated public WriteOptions withZoneMaps(boolean enabled) { return new WriteOptions(chunkSize, enabled, compressionRatioThreshold, allowedCascading, globalDict, enableZstd, - globalDictMaxRetainedBytes); + globalDictMaxRetainedBytes, editions); } /// Returns a copy of these options with global dictionary encoding set to `enabled`. @@ -78,7 +105,7 @@ public WriteOptions withZoneMaps(boolean enabled) { /// @return a new `WriteOptions` with the global dict flag updated public WriteOptions withGlobalDict(boolean enabled) { return new WriteOptions(chunkSize, enableZoneMaps, compressionRatioThreshold, allowedCascading, enabled, enableZstd, - globalDictMaxRetainedBytes); + globalDictMaxRetainedBytes, editions); } /// Returns a copy of these options with Zstandard compression set to `enabled`. @@ -94,7 +121,7 @@ public WriteOptions withGlobalDict(boolean enabled) { /// @return a new `WriteOptions` with the Zstd flag updated public WriteOptions withZstd(boolean enabled) { return new WriteOptions(chunkSize, enableZoneMaps, compressionRatioThreshold, allowedCascading, globalDict, enabled, - globalDictMaxRetainedBytes); + globalDictMaxRetainedBytes, editions); } /// Returns a copy of these options with the global-dictionary retention budget set to `budgetBytes`. @@ -109,6 +136,35 @@ public WriteOptions withZstd(boolean enabled) { /// @return a new `WriteOptions` with the global-dict retention budget updated public WriteOptions withGlobalDictMaxRetainedBytes(long budgetBytes) { return new WriteOptions(chunkSize, enableZoneMaps, compressionRatioThreshold, allowedCascading, globalDict, - enableZstd, budgetBytes); + enableZstd, budgetBytes, editions); + } + + /// Returns a copy of these options with `edition` enabled, replacing any edition already + /// enabled for `edition.id().family()`. Enabling an edition from a different family than any + /// currently configured adds to, rather than replaces, the enabled set — a writer may target at + /// most one edition per family, but multiple families at once (e.g. `core` and `unstable` + /// simultaneously). + /// + /// Encoding an id outside the union of every currently-enabled edition's cumulative members + /// fails the write immediately (see [io.github.dfa1.vortex.writer.VortexWriter]) — the edition + /// guarantee is checked at write time, never persisted into the file itself. + /// + /// @param edition the edition to enable + /// @return a new `WriteOptions` with `edition` enabled for its family + public WriteOptions withEdition(Edition edition) { + Map updated = new HashMap<>(editions); + updated.put(edition.id().family(), edition); + return new WriteOptions(chunkSize, enableZoneMaps, compressionRatioThreshold, allowedCascading, globalDict, + enableZstd, globalDictMaxRetainedBytes, updated); + } + + /// Returns a copy of these options with every enabled edition cleared — the explicit escape + /// hatch. With no edition enabled, the writer may emit any registered encoding, including + /// third-party or experimental ones, at the cost of the edition read-compatibility guarantee. + /// + /// @return a new `WriteOptions` with no edition guard at all + public WriteOptions withoutEditionGuard() { + return new WriteOptions(chunkSize, enableZoneMaps, compressionRatioThreshold, allowedCascading, globalDict, + enableZstd, globalDictMaxRetainedBytes, Map.of()); } } diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/EncodeContext.java b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/EncodeContext.java index 719d2bd6..ef4b3f57 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/encode/EncodeContext.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/encode/EncodeContext.java @@ -47,7 +47,18 @@ public record EncodeContext( /// @param registry write registry for encoder and extension lookup /// @return a new [EncodeContext] ready for non-cascading encoding public static EncodeContext of(Arena arena, WriteRegistry registry) { - return new EncodeContext(arena, registry, 0, Set.of(), 42L, 4096, 0.05); + return of(arena, registry, Set.of()); + } + + /// Creates a non-cascading context with an initial exclusion set (e.g. encoding ids outside a + /// [io.github.dfa1.vortex.writer.WriteOptions] edition guard), default sampling. + /// + /// @param arena the arena to allocate encode output buffers from + /// @param registry write registry for encoder and extension lookup + /// @param initialExcluded encoding ids excluded from consideration from the start + /// @return a new [EncodeContext] ready for non-cascading encoding + public static EncodeContext of(Arena arena, WriteRegistry registry, Set initialExcluded) { + return new EncodeContext(arena, registry, 0, Set.copyOf(initialExcluded), 42L, 4096, 0.05); } /// Creates a cascading context with the given depth and default sampling parameters. @@ -57,7 +68,24 @@ public static EncodeContext of(Arena arena, WriteRegistry registry) { /// @param registry write registry for encoder and extension lookup /// @return a new [EncodeContext] ready for cascading compression public static EncodeContext ofDepth(int depth, Arena arena, WriteRegistry registry) { - return new EncodeContext(arena, registry, depth, Set.of(), 42L, 4096, 0.05); + return ofDepth(depth, arena, registry, Set.of()); + } + + /// Creates a cascading context with an initial exclusion set (e.g. encoding ids outside a + /// [io.github.dfa1.vortex.writer.WriteOptions] edition guard), default sampling. + /// + /// [CascadingCompressor] already consults [#excluded] at every candidate-selection site + /// (including nested competitions like a masked column's validity-bitmap cascade), so seeding + /// it here makes the edition guard a graceful "skip ineligible candidates and fall back to the + /// best remaining one," not a hard failure after the fact. + /// + /// @param depth maximum allowed cascade depth + /// @param arena the arena to allocate encode output buffers from + /// @param registry write registry for encoder and extension lookup + /// @param initialExcluded encoding ids excluded from consideration from the start + /// @return a new [EncodeContext] ready for cascading compression + public static EncodeContext ofDepth(int depth, Arena arena, WriteRegistry registry, Set initialExcluded) { + return new EncodeContext(arena, registry, depth, Set.copyOf(initialExcluded), 42L, 4096, 0.05); } /// Returns a copy of this context with the cascade depth decremented by one. diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/DeltaEncodingTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/DeltaEncodingTest.java index e03cfa4b..9799c8b9 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/DeltaEncodingTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/DeltaEncodingTest.java @@ -41,7 +41,7 @@ void roundTrip_monotonicIncreasing(@TempDir Path tmp) throws IOException { long[] data = {100L, 105L, 110L, 115L, 120L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -60,7 +60,7 @@ void roundTrip_monotonicDecreasing(@TempDir Path tmp) throws IOException { long[] data = {1000L, 990L, 975L, 950L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -79,7 +79,7 @@ void roundTrip_allSameValue(@TempDir Path tmp) throws IOException { long[] data = {42L, 42L, 42L, 42L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -98,7 +98,7 @@ void roundTrip_singleElement(@TempDir Path tmp) throws IOException { long[] data = {99L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -117,7 +117,7 @@ void roundTrip_mixedDeltas(@TempDir Path tmp) throws IOException { long[] data = {0L, 5L, 7L, 12L, 11L, 15L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -139,7 +139,7 @@ void roundTrip_multipleChunks(@TempDir Path tmp) throws IOException { long[] chunk2 = {2000L, 2005L, 2010L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), chunk1)); @@ -164,7 +164,7 @@ void roundTrip_sequentialTimestamps(int n, @TempDir Path tmp) throws IOException } try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/NullCountPruningTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/NullCountPruningTest.java index 19bd3b1a..2bb3685e 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/NullCountPruningTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/NullCountPruningTest.java @@ -33,7 +33,7 @@ class NullCountPruningTest { // sizes and null patterns: 3 rows / 0 nulls, 2 rows / 1 null, 4 rows / all null. private Path write() throws IOException { Path file = tmp.resolve("nulls.vtx"); - WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, SCHEMA, opts)) { sut.writeChunk(Map.of(ColumnName.of("v"), new NullableData( diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java index 1db6b10e..b7d961cc 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java @@ -60,7 +60,7 @@ void writeSegments_are64ByteAligned(@TempDir Path tmp) throws IOException { // VortexWriter pads before each segment so every buffer starts 64-aligned (Arrow-compatible); // a broken pad — wrong modulus arithmetic or a skipped writePadding — leaves a segment offset // off a 64-byte boundary. - WriteOptions opts = new WriteOptions(3, false, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(3, false, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("aligned.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, SCHEMA, opts)) { diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/WriterEditionGuardTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/WriterEditionGuardTest.java new file mode 100644 index 00000000..c482738b --- /dev/null +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/WriterEditionGuardTest.java @@ -0,0 +1,102 @@ +package io.github.dfa1.vortex.writer; + +import io.github.dfa1.vortex.core.error.VortexException; +import io.github.dfa1.vortex.core.model.ColumnName; +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.Editions; +import io.github.dfa1.vortex.reader.ReadRegistry; +import io.github.dfa1.vortex.reader.VortexReader; +import io.github.dfa1.vortex.writer.encode.DeltaEncodingEncoder; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.nio.channels.FileChannel; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.List; +import java.util.Map; + +import static io.github.dfa1.vortex.writer.VortexReads.readAllLongs; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/// End-to-end tests for the [WriteOptions#editions()] guard (issue #301): [VortexWriter]'s +/// backstop check in `registerEncodingIds`, which fires for selection paths that don't consult +/// [io.github.dfa1.vortex.writer.encode.EncodeContext#excluded()] at all — e.g. a forced, +/// single-candidate explicit encoder list, as used here. The graceful-fallback filtering behavior +/// for cost-based competitions is covered separately by +/// `io.github.dfa1.vortex.writer.encode.MaskedValidityCascadeEditionExclusionTest`. +class WriterEditionGuardTest { + + private static final DType.Struct I64_SCHEMA = new DType.Struct( + List.of(ColumnName.of("ts")), + List.of(DType.I64), + false); + + private static ReadRegistry deltaRegistry() { + return ReadRegistry.builder() + .register(new io.github.dfa1.vortex.reader.decode.DeltaEncodingDecoder()) + .register(new io.github.dfa1.vortex.reader.decode.PrimitiveEncodingDecoder()) + .build(); + } + + @Test + void defaultGuard_forcedOutOfEditionEncoder_throwsNamingIdAndEdition(@TempDir Path tmp) throws IOException { + // Given — fastlanes.delta is unstable-family, outside the default core2026.07.0 guard; + // the explicit single-encoder list forces findEncoder's first-match dispatch straight to + // it, bypassing CascadingCompressor's exclusion-aware competition entirely + Path file = tmp.resolve("delta_guarded.vtx"); + long[] data = {100L, 105L, 110L, 115L, 120L}; + + // When / Then + try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults(), + List.of(new DeltaEncodingEncoder()))) { + assertThatThrownBy(() -> sut.writeChunk(Map.of(ColumnName.of("ts"), data))) + .isInstanceOf(VortexException.class) + .hasMessageContaining("fastlanes.delta") + .hasMessageContaining("core2026.07.0") + .hasMessageContaining("withoutEditionGuard"); + } + } + + @Test + void withoutEditionGuard_allowsTheSameForcedEncoder(@TempDir Path tmp) throws IOException { + // Given — the identical setup, but with the guard explicitly cleared + Path file = tmp.resolve("delta_unguarded.vtx"); + long[] data = {100L, 105L, 110L, 115L, 120L}; + + // When + try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + List.of(new DeltaEncodingEncoder()))) { + sut.writeChunk(Map.of(ColumnName.of("ts"), data)); + } + + // Then + try (var vf = VortexReader.open(file, deltaRegistry())) { + assertThat(readAllLongs(vf, "ts")).containsExactly(data); + } + } + + @Test + void withEdition_enablingUnstable_allowsTheForcedEncoder(@TempDir Path tmp) throws IOException { + // Given — instead of clearing the guard entirely, explicitly opt into the unstable + // edition that covers fastlanes.delta + Path file = tmp.resolve("delta_unstable.vtx"); + long[] data = {100L, 105L, 110L, 115L, 120L}; + WriteOptions options = WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0); + + // When + try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + var sut = VortexWriter.create(ch, I64_SCHEMA, options, List.of(new DeltaEncodingEncoder()))) { + sut.writeChunk(Map.of(ColumnName.of("ts"), data)); + } + + // Then + try (var vf = VortexReader.open(file, deltaRegistry())) { + assertThat(readAllLongs(vf, "ts")).containsExactly(data); + } + } +} diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/WriterZoneMapTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/WriterZoneMapTest.java index feebc855..aa3bf7fd 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/WriterZoneMapTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/WriterZoneMapTest.java @@ -40,7 +40,7 @@ class WriterZoneMapTest { // Three zones of four rows: [0..3], [4..7], [8..11]. private static Path write(Path tmp, boolean zoneMaps) throws IOException { - WriteOptions opts = new WriteOptions(4, zoneMaps, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(4, zoneMaps, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("zoned-" + zoneMaps + ".vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, SCHEMA, opts)) { @@ -139,7 +139,7 @@ void zoneMaps_nullableColumn_recordsPerZoneNullCount(@TempDir Path tmp) throws I // zone 1 = [null, null] DType.Struct schema = new DType.Struct( List.of(ColumnName.of("v")), List.of(new DType.Primitive(PType.I64, true)), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("nullable.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -176,7 +176,7 @@ void everyPrimitiveType_emitsZonedLayout(PType ptype, @TempDir Path tmp) throws // every one gets a vortex.stats layout — exercising each per-ptype stat-column arm. DType.Struct schema = new DType.Struct( List.of(ColumnName.of("v")), List.of(new DType.Primitive(ptype, false)), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("ptype-" + ptype + ".vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -196,7 +196,7 @@ void everyPrimitiveType_emitsZonedLayout(PType ptype, @TempDir Path tmp) throws void noChunks_emitsNoZoneMap(@TempDir Path tmp) throws IOException { // Given a file closed without any writeChunk: the column has no chunks, so flushZoneMaps // skips it (the empty-chunks guard) and emits no zone-map. - WriteOptions opts = new WriteOptions(4, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(4, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("empty.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, SCHEMA, opts)) { @@ -217,7 +217,7 @@ void chunkWithoutStats_emitsNullCountOnlyZoneMap(@TempDir Path tmp) throws IOExc // — SUM is independent (the empty zone's sum is simply null). DType.Struct schema = new DType.Struct( List.of(ColumnName.of("v")), List.of(DType.I64), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("partial.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -241,7 +241,7 @@ void utf8Column_emitsStringMinMaxZoneMap(@TempDir Path tmp) throws IOException { // the zone-map carries MAX+MIN+NULL_COUNT (string min/max), not null_count alone. DType.Struct schema = new DType.Struct( List.of(ColumnName.of("s")), List.of(DType.UTF8), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("utf8.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -279,7 +279,7 @@ void extensionColumn_emitsStorageMinMaxZoneMap(@TempDir Path tmp) throws IOExcep DType ext = new DType.Extension( "test.ext", DType.I64, null, false); DType.Struct schema = new DType.Struct(List.of(ColumnName.of("t")), List.of(ext), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("ext.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -316,7 +316,7 @@ void dictColumn_emitsStringMinMaxZoneMapWrappingDict(@TempDir Path tmp) throws I // zone 1 = a..c → MAX+MIN+NULL_COUNT, with the column wrapped as vortex.stats over the dict. DType.Struct schema = new DType.Struct( List.of(ColumnName.of("s")), List.of(DType.UTF8), false); - WriteOptions opts = new WriteOptions(6, true, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(6, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("dict.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -353,7 +353,7 @@ void primitiveDictColumn_emitsNumericMinMaxZoneMapWrappingDict(@TempDir Path tmp // Zone-map min/max are computed on the logical I64 values: zone 0 = 1..2, zone 1 = 1..3. DType.Struct schema = new DType.Struct( List.of(ColumnName.of("v")), List.of(DType.I64), false); - WriteOptions opts = new WriteOptions(6, true, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(6, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("primdict.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -393,7 +393,7 @@ void zoneMaps_f64StatsPayloadDecodesPerZoneMinMaxSum(@TempDir Path tmp) throws I // a decode that returned 0.0 or read the wrong scalar field would slip through. Fractional // .5 values also make a truncating (int) decode observable. DType.Struct schema = new DType.Struct(List.of(ColumnName.of("v")), List.of(DType.F64), false); - WriteOptions opts = new WriteOptions(4, true, 0.90, 0, true, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(4, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("zoned-f64.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -434,7 +434,7 @@ void zoneMaps_perTypeStatsDecodePerZoneMinMax(PType ptype, @TempDir Path tmp) th // F32 statColumn arms (and the f32 scalar field read in scalarDouble), which the I64/F64 // value tests never reach. DType.Struct schema = new DType.Struct(List.of(ColumnName.of("v")), List.of(new DType.Primitive(ptype, false)), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); Path file = tmp.resolve("ptype-stats-" + ptype + ".vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/encode/MaskedValidityCascadeEditionExclusionTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/encode/MaskedValidityCascadeEditionExclusionTest.java new file mode 100644 index 00000000..8dec867b --- /dev/null +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/encode/MaskedValidityCascadeEditionExclusionTest.java @@ -0,0 +1,86 @@ +package io.github.dfa1.vortex.writer.encode; + +import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.EncodingId; +import io.github.dfa1.vortex.core.model.PType; +import io.github.dfa1.vortex.writer.WriteRegistry; +import org.junit.jupiter.api.Test; + +import java.lang.foreign.Arena; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; + +/// Verifies the exact mechanism [io.github.dfa1.vortex.writer.VortexWriter]'s edition guard +/// (issue #301) relies on: seeding [EncodeContext]'s initial `excluded` set makes +/// [CascadingCompressor] skip an out-of-edition candidate during its cost-based competition and +/// fall back to the best remaining one, instead of the guard only discovering the violation after +/// encoding completes. +/// +/// This generalizes to any nested candidate list built the same way, including +/// [SparseEncodingEncoder]'s `INDEX_CASCADE_CANDIDATES` — used by [MaskedEncodingEncoder]'s +/// nested validity-mask cascade to compress a `vortex.sparse` patch-index array, entirely +/// independent of any top-level cascade codec list, which is what makes an `unstable`-family +/// encoding structurally reachable from a plain nullable-column write, not just an explicit +/// custom encoder list. Uses `vortex.constant` vs. `vortex.primitive` (not `fastlanes.delta`) as +/// the competing pair: `DeltaEncodingEncoder` stores its transposed bases and deltas at full +/// width with no further cascading, so it never actually wins a real size competition against raw +/// storage in the current implementation — constant-vs-primitive gives a reliable, easily +/// constructed win margin instead. See adr/0023-vortex-editions-adoption.md. +class MaskedValidityCascadeEditionExclusionTest { + + private static Set collectEncodingIds(EncodeNode node) { + Set ids = new LinkedHashSet<>(); + collectEncodingIds(node, ids); + return ids; + } + + private static void collectEncodingIds(EncodeNode node, Set into) { + into.add(node.encodingId()); + for (EncodeNode child : node.children()) { + collectEncodingIds(child, into); + } + } + + private static CascadingCompressor compressor() { + return new CascadingCompressor(List.of(new ConstantEncodingEncoder(), new PrimitiveEncodingEncoder())); + } + + @Test + void unrestrictedContext_selectsConstantOverRawPrimitive() { + // Given — an all-same-value array, cascading enabled, no exclusions + long[] values = new long[4000]; + java.util.Arrays.fill(values, 7L); + EncodeContext ctx = EncodeContext.ofDepth(1, Arena.ofAuto(), WriteRegistry.loadAll()); + + // When + EncodeResult result = compressor().encode(new DType.Primitive(PType.I64, false), values, ctx); + + // Then — confirms the data pattern genuinely makes vortex.constant win a fair competition + // against vortex.primitive, so the next test's exclusion is a real effect, not vacuous + // (vortex.constant was the only other candidate to fall back from) + assertThat(collectEncodingIds(result.rootNode())).contains(EncodingId.VORTEX_CONSTANT); + } + + @Test + void editionExcludedContext_fallsBackInsteadOfSelectingConstant() { + // Given — the exact same all-same-value array, but vortex.constant seeded into the + // initial exclusion set, mirroring how VortexWriter seeds editionExcluded when the + // configured edition doesn't cover it (see VortexWriter#editionExcluded) + long[] values = new long[4000]; + java.util.Arrays.fill(values, 7L); + EncodeContext ctx = EncodeContext.ofDepth( + 1, Arena.ofAuto(), WriteRegistry.loadAll(), Set.of(EncodingId.VORTEX_CONSTANT)); + + // When + EncodeResult result = compressor().encode(new DType.Primitive(PType.I64, false), values, ctx); + + // Then — gracefully falls back to the only remaining candidate; never surfaces a + // violation here, let alone throws + assertThat(collectEncodingIds(result.rootNode())) + .doesNotContain(EncodingId.VORTEX_CONSTANT) + .contains(EncodingId.VORTEX_PRIMITIVE); + } +} From b4ed38e58ee0b2edf17316273665453c47ce94fe Mon Sep 17 00:00:00 2001 From: Davide Angelocola Date: Mon, 27 Jul 2026 07:31:56 +0200 Subject: [PATCH 2/2] fix(core): use YearMonth and typed EncodingId constants in Editions Two review comments from the PR: - EditionId(EditionFamily, int year, int month, int version) -> EditionId(EditionFamily, YearMonth cutMonth, int version). java.time.YearMonth already validates the month range (throwing DateTimeException, arguably more idiomatic than a hand-rolled IllegalArgumentException for this exact domain) and is Comparable, so isAtOrBefore simplifies to a single compareTo plus the version tie-break. Drops the separate 4-digit-year guard - belt-and-braces validation on a value only ever constructed from 8 hardcoded catalog literals, not from untrusted input. - Editions' catalog now references the real EncodingId.WellKnown constants (EncodingId.VORTEX_ALP, ...) directly instead of raw wire strings parsed at runtime via EncodingId.parse(...). Matches this project's own "strings at the boundary, types inside" principle - these ids come from a hardcoded catalog, not wire input, so there was no boundary to parse a string at; using the constants also catches a typo'd id at compile time instead of it silently becoming an unintended EncodingId.Custom. Only the handful of unstable ids with no WellKnown constant yet still use EncodingId.Custom. Applied the same fix to EditionsTest's golden-set expected values. Also reverted EditionFamily#toString() (a mid-session addition) back to the enum default: overriding toString() to return something other than the declared constant name is surprising and breaks valueOf(family.toString()). EditionId#toString() now lowercases the family name explicitly at the one call site that needs the wire form. ADR 0023 and docs/reference.md updated to match. Co-Authored-By: Claude Sonnet 5 --- adr/0023-vortex-editions-adoption.md | 14 ++- .../dfa1/vortex/core/model/EditionId.java | 46 +++---- .../dfa1/vortex/core/model/Editions.java | 78 ++++++------ .../dfa1/vortex/core/model/EditionIdTest.java | 60 +++------ .../dfa1/vortex/core/model/EditionsTest.java | 115 +++++++++--------- docs/reference.md | 2 +- 6 files changed, 136 insertions(+), 179 deletions(-) diff --git a/adr/0023-vortex-editions-adoption.md b/adr/0023-vortex-editions-adoption.md index ee9176b8..cd4fcb45 100644 --- a/adr/0023-vortex-editions-adoption.md +++ b/adr/0023-vortex-editions-adoption.md @@ -40,12 +40,18 @@ faithfully rather than being truncated to what is implemented today. because any Rust crate can register a declaration into the session. vortex-java's registries are explicit and closed (no `ServiceLoader` — the same design decision already made for `ReadRegistry`/`WriteRegistry`), so this collapses to three `core.model` types: - - `EditionId(EditionFamily family, int year, int month, int version)` — `toString()` matches - Rust's `Display` exactly (`"core2025.05.0"`, month zero-padded, version not); - `isAtOrBefore(EditionId)` orders editions within a family (cross-family is always `false`). + - `EditionId(EditionFamily family, YearMonth cutMonth, int version)` — reuses + `java.time.YearMonth` instead of two raw ints (validates the month range for free, and is + `Comparable`, simplifying `isAtOrBefore`); `toString()` matches Rust's `Display` exactly + (`"core2025.05.0"`, month zero-padded, version not); `isAtOrBefore(EditionId)` orders + editions within a family (cross-family is always `false`). - `Edition(EditionId id, Optional minVortexVersion, Set added)` — folds Rust's separate `Edition`/`EditionDeclaration` into one type; `added` is only what joins *at - this exact edition*, matching Rust's field exactly (not cumulative). + this exact edition*, matching Rust's field exactly (not cumulative). Populated with the real + `EncodingId.WellKnown` constants wherever one exists ("strings at the boundary, types + inside" — these come from a hardcoded catalog, not wire input, so there is no boundary to + parse a string at), falling back to `EncodingId.Custom(rawId)` only for the handful of + `unstable` ids with no `WellKnown` constant yet. - `Editions` — a final utility class holding the 8 catalog constants, `ALL` (declaration order), `cumulativeMembers(Edition)` (union of `added()` for every same-family edition at or before the given one, seeded with the argument's own `added()` first), and diff --git a/core/src/main/java/io/github/dfa1/vortex/core/model/EditionId.java b/core/src/main/java/io/github/dfa1/vortex/core/model/EditionId.java index a263c630..f23b6b9e 100644 --- a/core/src/main/java/io/github/dfa1/vortex/core/model/EditionId.java +++ b/core/src/main/java/io/github/dfa1/vortex/core/model/EditionId.java @@ -1,35 +1,25 @@ package io.github.dfa1.vortex.core.model; +import java.time.YearMonth; import java.util.Locale; +import java.util.Objects; /// Identifier of a Vortex [edition](https://github.com/vortex-data/vortex/blob/develop/docs/specs/editions.md), /// e.g. `core2026.07.0`. /// /// `family` names an independently versioned, additive group of encodings (`core` is the set the -/// default writer emits); the date components record when the edition was frozen and order -/// editions chronologically *within* a family — there is no ordering across families. +/// default writer emits); `cutMonth` records when the edition was frozen and orders editions +/// chronologically *within* a family — there is no ordering across families. /// -/// @param family the edition family -/// @param year the four-digit year the edition was cut -/// @param month the month the edition was cut (1-12) -/// @param version distinguishes editions cut in the same month; normally `0` -public record EditionId(EditionFamily family, int year, int month, int version) { +/// @param family the edition family +/// @param cutMonth the year and month the edition was cut +/// @param version distinguishes editions cut in the same month; normally `0` +public record EditionId(EditionFamily family, YearMonth cutMonth, int version) { - /// Validates the identifier's form: a four-digit year and a month in 1-12 (`family` needs no - /// validation of its own — [EditionFamily] is a closed enum). Editions are a client-side - /// write/read policy, never derived from untrusted file bytes, so a malformed value here is a - /// programmer error — hence [IllegalArgumentException], not - /// [io.github.dfa1.vortex.core.error.VortexException] (which is reserved for untrusted-input - /// parsing failures). - /// - /// @throws IllegalArgumentException if `year` or `month` is malformed + /// @throws NullPointerException if `family` or `cutMonth` is `null` public EditionId { - if (year < 1000 || year > 9999) { - throw new IllegalArgumentException("edition year must be four digits, got: " + year); - } - if (month < 1 || month > 12) { - throw new IllegalArgumentException("edition month must be in 1-12, got: " + month); - } + Objects.requireNonNull(family, "family"); + Objects.requireNonNull(cutMonth, "cutMonth"); } /// Returns `true` if `this` is the same edition as `other` or an earlier edition of the same @@ -38,20 +28,16 @@ public record EditionId(EditionFamily family, int year, int month, int version) /// @param other the edition id to compare against /// @return `true` if `this` is at or before `other` within the same family public boolean isAtOrBefore(EditionId other) { - if (!family.equals(other.family)) { + if (family != other.family) { return false; } - if (year != other.year) { - return year < other.year; - } - if (month != other.month) { - return month < other.month; - } - return version <= other.version; + int cmp = cutMonth.compareTo(other.cutMonth); + return cmp != 0 ? cmp < 0 : version <= other.version; } @Override public String toString() { - return "%s%d.%02d.%d".formatted(family.name().toLowerCase(Locale.ROOT), year, month, version); + return "%s%d.%02d.%d".formatted( + family.name().toLowerCase(Locale.ROOT), cutMonth.getYear(), cutMonth.getMonthValue(), version); } } diff --git a/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java b/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java index 7ad6d8e5..725a3db4 100644 --- a/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java +++ b/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java @@ -1,5 +1,6 @@ package io.github.dfa1.vortex.core.model; +import java.time.YearMonth; import java.util.LinkedHashSet; import java.util.List; import java.util.Optional; @@ -17,71 +18,76 @@ /// the union of everything it and every earlier edition of the same family added; see /// [#cumulativeMembers(Edition)]. /// -/// vortex-java implements every `core`-family encoding through `core2026.07.0`. Of `unstable`, it -/// implements only `fastlanes.delta` and `vortex.patched` — the remaining ids -/// (`vortex.zstd_buffers`, `vortex.parquet.variant`, the `vortex.tensor.*` family, -/// `vortex.onpair`) have no [EncodingId.WellKnown] constant yet and resolve to -/// [EncodingId.Custom] via [EncodingId#parse(String)], which the catalog stores uniformly with -/// the implemented ids — this catalog mirrors upstream faithfully rather than being truncated to -/// what is implemented today. +/// vortex-java implements every `core`-family encoding through `core2026.07.0`, referenced below +/// by their [EncodingId.WellKnown] constants. Of `unstable`, it implements only `fastlanes.delta` +/// and `vortex.patched` — the remaining ids (`vortex.zstd_buffers`, `vortex.parquet.variant`, the +/// `vortex.tensor.*` family, `vortex.onpair`) have no `WellKnown` constant yet, so they are named +/// as [EncodingId.Custom] instead; the catalog stores both uniformly and mirrors upstream +/// faithfully rather than being truncated to what is implemented today. public final class Editions { /// The baseline `core` edition: stable encodings writable by Vortex 0.36.0. public static final Edition CORE_2025_05_0 = new Edition( - new EditionId(EditionFamily.CORE, 2025, 5, 0), + new EditionId(EditionFamily.CORE, YearMonth.of(2025, 5), 0), Optional.of("0.36.0"), - idSet( - "fastlanes.bitpacked", "fastlanes.for", - "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", - "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", - "vortex.dict", "vortex.ext", "vortex.fsst", "vortex.list", "vortex.null", - "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", - "vortex.varbinview", "vortex.zigzag")); + Set.of( + EncodingId.FASTLANES_BITPACKED, EncodingId.FASTLANES_FOR, + EncodingId.VORTEX_ALP, EncodingId.VORTEX_ALPRD, EncodingId.VORTEX_BOOL, + EncodingId.VORTEX_BYTEBOOL, EncodingId.VORTEX_CHUNKED, EncodingId.VORTEX_CONSTANT, + EncodingId.VORTEX_DATETIMEPARTS, EncodingId.VORTEX_DECIMAL, + EncodingId.VORTEX_DECIMAL_BYTE_PARTS, EncodingId.VORTEX_DICT, EncodingId.VORTEX_EXT, + EncodingId.VORTEX_FSST, EncodingId.VORTEX_LIST, EncodingId.VORTEX_NULL, + EncodingId.VORTEX_PRIMITIVE, EncodingId.VORTEX_RUNEND, EncodingId.VORTEX_SPARSE, + EncodingId.VORTEX_STRUCT, EncodingId.VORTEX_VARBIN, EncodingId.VORTEX_VARBINVIEW, + EncodingId.VORTEX_ZIGZAG)); /// The `core` edition adding stable encodings released through June 2025. public static final Edition CORE_2025_06_0 = new Edition( - new EditionId(EditionFamily.CORE, 2025, 6, 0), + new EditionId(EditionFamily.CORE, YearMonth.of(2025, 6), 0), Optional.of("0.40.0"), - idSet("vortex.pco", "vortex.sequence", "vortex.zstd")); + Set.of(EncodingId.VORTEX_PCO, EncodingId.VORTEX_SEQUENCE, EncodingId.VORTEX_ZSTD)); /// The `core` edition adding stable encodings released through October 2025. public static final Edition CORE_2025_10_0 = new Edition( - new EditionId(EditionFamily.CORE, 2025, 10, 0), + new EditionId(EditionFamily.CORE, YearMonth.of(2025, 10), 0), Optional.of("0.54.0"), - idSet("fastlanes.rle", "vortex.fixed_size_list", "vortex.listview", "vortex.masked")); + Set.of(EncodingId.FASTLANES_RLE, EncodingId.VORTEX_FIXED_SIZE_LIST, + EncodingId.VORTEX_LISTVIEW, EncodingId.VORTEX_MASKED)); /// The `core` edition adding stable encodings released through July 2026. public static final Edition CORE_2026_07_0 = new Edition( - new EditionId(EditionFamily.CORE, 2026, 7, 0), + new EditionId(EditionFamily.CORE, YearMonth.of(2026, 7), 0), Optional.of("0.65.0"), - idSet("vortex.variant")); + Set.of(EncodingId.VORTEX_VARIANT)); /// The May 2025 draft edition of the `unstable` family. public static final Edition UNSTABLE_2025_05_0 = new Edition( - new EditionId(EditionFamily.UNSTABLE, 2025, 5, 0), + new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2025, 5), 0), Optional.empty(), - idSet("fastlanes.delta")); + Set.of(EncodingId.FASTLANES_DELTA)); /// The February 2026 draft edition of the `unstable` family. public static final Edition UNSTABLE_2026_02_0 = new Edition( - new EditionId(EditionFamily.UNSTABLE, 2026, 2, 0), + new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2026, 2), 0), Optional.empty(), - idSet("vortex.zstd_buffers")); + Set.of(new EncodingId.Custom("vortex.zstd_buffers"))); /// The April 2026 draft edition of the `unstable` family. public static final Edition UNSTABLE_2026_04_0 = new Edition( - new EditionId(EditionFamily.UNSTABLE, 2026, 4, 0), + new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2026, 4), 0), Optional.empty(), - idSet( - "vortex.parquet.variant", "vortex.patched", - "vortex.tensor.cosine_similarity", "vortex.tensor.inner_product", - "vortex.tensor.l2_denorm", "vortex.tensor.l2_norm")); + Set.of( + new EncodingId.Custom("vortex.parquet.variant"), EncodingId.VORTEX_PATCHED, + new EncodingId.Custom("vortex.tensor.cosine_similarity"), + new EncodingId.Custom("vortex.tensor.inner_product"), + new EncodingId.Custom("vortex.tensor.l2_denorm"), + new EncodingId.Custom("vortex.tensor.l2_norm"))); /// The June 2026 draft edition of the `unstable` family. public static final Edition UNSTABLE_2026_06_0 = new Edition( - new EditionId(EditionFamily.UNSTABLE, 2026, 6, 0), + new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2026, 6), 0), Optional.empty(), - idSet("vortex.onpair")); + Set.of(new EncodingId.Custom("vortex.onpair"))); /// Every declared edition, in the order above. Order matters: [#owningEdition(EncodingId)] /// returns the first entry whose `added` set contains the queried id. @@ -128,12 +134,4 @@ public static Optional owningEdition(EncodingId id) { } return Optional.empty(); } - - private static Set idSet(String... rawIds) { - Set ids = new LinkedHashSet<>(rawIds.length); - for (String rawId : rawIds) { - ids.add(EncodingId.parse(rawId)); - } - return Set.copyOf(ids); - } } diff --git a/core/src/test/java/io/github/dfa1/vortex/core/model/EditionIdTest.java b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionIdTest.java index 51c2f4cb..45a977c2 100644 --- a/core/src/test/java/io/github/dfa1/vortex/core/model/EditionIdTest.java +++ b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionIdTest.java @@ -2,11 +2,10 @@ import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; -import org.junit.jupiter.params.ParameterizedTest; -import org.junit.jupiter.params.provider.ValueSource; + +import java.time.YearMonth; import static org.assertj.core.api.Assertions.assertThat; -import static org.assertj.core.api.Assertions.assertThatThrownBy; class EditionIdTest { @@ -16,7 +15,7 @@ class ToString { @Test void toString_zeroPadsMonthButNotVersion() { // Given — month 5 must render as "05", version 0 must render as "0" (not "00") - EditionId sut = new EditionId(EditionFamily.CORE, 2025, 5, 0); + EditionId sut = new EditionId(EditionFamily.CORE, YearMonth.of(2025, 5), 0); // When String result = sut.toString(); @@ -28,7 +27,7 @@ void toString_zeroPadsMonthButNotVersion() { @Test void toString_doubleDigitMonthAndNonZeroVersion() { // Given - EditionId sut = new EditionId(EditionFamily.UNSTABLE, 2026, 11, 2); + EditionId sut = new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2026, 11), 2); // When String result = sut.toString(); @@ -44,8 +43,8 @@ class IsAtOrBefore { @Test void isAtOrBefore_earlierYear_isTrue() { // Given - EditionId earlier = new EditionId(EditionFamily.CORE, 2025, 5, 0); - EditionId later = new EditionId(EditionFamily.CORE, 2026, 7, 0); + EditionId earlier = new EditionId(EditionFamily.CORE, YearMonth.of(2025, 5), 0); + EditionId later = new EditionId(EditionFamily.CORE, YearMonth.of(2026, 7), 0); // When boolean result = earlier.isAtOrBefore(later); @@ -57,7 +56,7 @@ void isAtOrBefore_earlierYear_isTrue() { @Test void isAtOrBefore_sameEdition_isTrue() { // Given - EditionId sut = new EditionId(EditionFamily.CORE, 2025, 5, 0); + EditionId sut = new EditionId(EditionFamily.CORE, YearMonth.of(2025, 5), 0); // When boolean result = sut.isAtOrBefore(sut); @@ -69,8 +68,8 @@ void isAtOrBefore_sameEdition_isTrue() { @Test void isAtOrBefore_laterEdition_isFalse() { // Given - EditionId later = new EditionId(EditionFamily.CORE, 2026, 7, 0); - EditionId earlier = new EditionId(EditionFamily.CORE, 2025, 5, 0); + EditionId later = new EditionId(EditionFamily.CORE, YearMonth.of(2026, 7), 0); + EditionId earlier = new EditionId(EditionFamily.CORE, YearMonth.of(2025, 5), 0); // When boolean result = later.isAtOrBefore(earlier); @@ -82,8 +81,8 @@ void isAtOrBefore_laterEdition_isFalse() { @Test void isAtOrBefore_sameYearEarlierMonth_isTrue() { // Given - EditionId sut = new EditionId(EditionFamily.CORE, 2025, 5, 0); - EditionId other = new EditionId(EditionFamily.CORE, 2025, 6, 0); + EditionId sut = new EditionId(EditionFamily.CORE, YearMonth.of(2025, 5), 0); + EditionId other = new EditionId(EditionFamily.CORE, YearMonth.of(2025, 6), 0); // When / Then assertThat(sut.isAtOrBefore(other)).isTrue(); @@ -92,8 +91,8 @@ void isAtOrBefore_sameYearEarlierMonth_isTrue() { @Test void isAtOrBefore_sameYearMonthEarlierVersion_isTrue() { // Given - EditionId sut = new EditionId(EditionFamily.CORE, 2025, 5, 0); - EditionId other = new EditionId(EditionFamily.CORE, 2025, 5, 1); + EditionId sut = new EditionId(EditionFamily.CORE, YearMonth.of(2025, 5), 0); + EditionId other = new EditionId(EditionFamily.CORE, YearMonth.of(2025, 5), 1); // When / Then assertThat(sut.isAtOrBefore(other)).isTrue(); @@ -103,41 +102,12 @@ void isAtOrBefore_sameYearMonthEarlierVersion_isTrue() { void isAtOrBefore_differentFamily_isAlwaysFalse() { // Given — unstable2025.05.0 is chronologically "earlier" than core2026.07.0, but // families are never ordered against each other - EditionId unstable = new EditionId(EditionFamily.UNSTABLE, 2025, 5, 0); - EditionId core = new EditionId(EditionFamily.CORE, 2026, 7, 0); + EditionId unstable = new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2025, 5), 0); + EditionId core = new EditionId(EditionFamily.CORE, YearMonth.of(2026, 7), 0); // When / Then assertThat(unstable.isAtOrBefore(core)).isFalse(); assertThat(core.isAtOrBefore(unstable)).isFalse(); } } - - @Nested - class Validation { - - @ParameterizedTest - @ValueSource(ints = {0, 99, 999, 10000}) - void constructor_nonFourDigitYear_throwsIllegalArgumentException(int year) { - // Given / When / Then - assertThatThrownBy(() -> new EditionId(EditionFamily.CORE, year, 5, 0)) - .isInstanceOf(IllegalArgumentException.class) - .hasMessageContaining("year"); - } - - @ParameterizedTest - @ValueSource(ints = {0, 13, -1}) - void constructor_monthOutOfRange_throwsIllegalArgumentException(int month) { - // Given / When / Then - assertThatThrownBy(() -> new EditionId(EditionFamily.CORE, 2025, month, 0)) - .isInstanceOf(IllegalArgumentException.class) - .hasMessageContaining("month"); - } - - @Test - void constructor_validValues_doesNotThrow() { - // Given / When / Then - assertThat(new EditionId(EditionFamily.CORE, 2025, 1, 0)).isNotNull(); - assertThat(new EditionId(EditionFamily.UNSTABLE, 9999, 12, 99)).isNotNull(); - } - } } diff --git a/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java index f72b9221..2236c463 100644 --- a/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java +++ b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java @@ -3,10 +3,10 @@ import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; +import java.time.YearMonth; +import java.util.LinkedHashSet; import java.util.Optional; import java.util.Set; -import java.util.stream.Collectors; -import java.util.stream.Stream; import static org.assertj.core.api.Assertions.assertThat; @@ -16,6 +16,26 @@ /// own `validate_edition` test harness. class EditionsTest { + // core2025.05.0's 23-member baseline, referenced by the typed EncodingId.WellKnown constants + // (not the wire strings) so a typo here fails to compile instead of silently becoming an + // unintended EncodingId.Custom. + private static final Set CORE_2025_05_0_BASELINE = Set.of( + EncodingId.FASTLANES_BITPACKED, EncodingId.FASTLANES_FOR, + EncodingId.VORTEX_ALP, EncodingId.VORTEX_ALPRD, EncodingId.VORTEX_BOOL, + EncodingId.VORTEX_BYTEBOOL, EncodingId.VORTEX_CHUNKED, EncodingId.VORTEX_CONSTANT, + EncodingId.VORTEX_DATETIMEPARTS, EncodingId.VORTEX_DECIMAL, + EncodingId.VORTEX_DECIMAL_BYTE_PARTS, EncodingId.VORTEX_DICT, EncodingId.VORTEX_EXT, + EncodingId.VORTEX_FSST, EncodingId.VORTEX_LIST, EncodingId.VORTEX_NULL, + EncodingId.VORTEX_PRIMITIVE, EncodingId.VORTEX_RUNEND, EncodingId.VORTEX_SPARSE, + EncodingId.VORTEX_STRUCT, EncodingId.VORTEX_VARBIN, EncodingId.VORTEX_VARBINVIEW, + EncodingId.VORTEX_ZIGZAG); + + private static Set union(Set a, EncodingId... more) { + Set result = new LinkedHashSet<>(a); + result.addAll(Set.of(more)); + return Set.copyOf(result); + } + @Nested class CumulativeMembers { @@ -26,13 +46,7 @@ void core2025_05_0_isExactlyItsOwnBaseline() { Set result = Editions.cumulativeMembers(Editions.CORE_2025_05_0); // Then - assertThat(result).isEqualTo(ids( - "fastlanes.bitpacked", "fastlanes.for", - "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", - "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", - "vortex.dict", "vortex.ext", "vortex.fsst", "vortex.list", "vortex.null", - "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", - "vortex.varbinview", "vortex.zigzag")); + assertThat(result).isEqualTo(CORE_2025_05_0_BASELINE); } @Test @@ -42,14 +56,8 @@ void core2025_06_0_addsToTheBaseline() { Set result = Editions.cumulativeMembers(Editions.CORE_2025_06_0); // Then — baseline 23 plus pco/sequence/zstd - assertThat(result).isEqualTo(ids( - "fastlanes.bitpacked", "fastlanes.for", - "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", - "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", - "vortex.dict", "vortex.ext", "vortex.fsst", "vortex.list", "vortex.null", - "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", - "vortex.varbinview", "vortex.zigzag", - "vortex.pco", "vortex.sequence", "vortex.zstd")); + assertThat(result).isEqualTo(union(CORE_2025_05_0_BASELINE, + EncodingId.VORTEX_PCO, EncodingId.VORTEX_SEQUENCE, EncodingId.VORTEX_ZSTD)); } @Test @@ -59,15 +67,10 @@ void core2025_10_0_addsToPreviousCore() { Set result = Editions.cumulativeMembers(Editions.CORE_2025_10_0); // Then — the 26 above plus rle/fixed_size_list/listview/masked - assertThat(result).isEqualTo(ids( - "fastlanes.bitpacked", "fastlanes.for", "fastlanes.rle", - "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", - "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", - "vortex.dict", "vortex.ext", "vortex.fixed_size_list", "vortex.fsst", "vortex.list", - "vortex.listview", "vortex.masked", "vortex.null", - "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", - "vortex.varbinview", "vortex.zigzag", - "vortex.pco", "vortex.sequence", "vortex.zstd")); + assertThat(result).isEqualTo(union(CORE_2025_05_0_BASELINE, + EncodingId.VORTEX_PCO, EncodingId.VORTEX_SEQUENCE, EncodingId.VORTEX_ZSTD, + EncodingId.FASTLANES_RLE, EncodingId.VORTEX_FIXED_SIZE_LIST, + EncodingId.VORTEX_LISTVIEW, EncodingId.VORTEX_MASKED)); } @Test @@ -77,17 +80,11 @@ void core2026_07_0_isTheFullCoreSet() { Set result = Editions.cumulativeMembers(Editions.CORE_2026_07_0); // Then — every core encoding through vortex.variant; 31 total - assertThat(result).hasSize(31) - .isEqualTo(ids( - "fastlanes.bitpacked", "fastlanes.for", "fastlanes.rle", - "vortex.alp", "vortex.alprd", "vortex.bool", "vortex.bytebool", "vortex.chunked", - "vortex.constant", "vortex.datetimeparts", "vortex.decimal", "vortex.decimal_byte_parts", - "vortex.dict", "vortex.ext", "vortex.fixed_size_list", "vortex.fsst", "vortex.list", - "vortex.listview", "vortex.masked", "vortex.null", - "vortex.primitive", "vortex.runend", "vortex.sparse", "vortex.struct", "vortex.varbin", - "vortex.varbinview", "vortex.zigzag", - "vortex.pco", "vortex.sequence", "vortex.zstd", - "vortex.variant")); + assertThat(result).hasSize(31).isEqualTo(union(CORE_2025_05_0_BASELINE, + EncodingId.VORTEX_PCO, EncodingId.VORTEX_SEQUENCE, EncodingId.VORTEX_ZSTD, + EncodingId.FASTLANES_RLE, EncodingId.VORTEX_FIXED_SIZE_LIST, + EncodingId.VORTEX_LISTVIEW, EncodingId.VORTEX_MASKED, + EncodingId.VORTEX_VARIANT)); } @Test @@ -97,7 +94,7 @@ void unstable2025_05_0_isExactlyItsOwnAddition() { Set result = Editions.cumulativeMembers(Editions.UNSTABLE_2025_05_0); // Then - assertThat(result).isEqualTo(ids("fastlanes.delta")); + assertThat(result).containsExactly(EncodingId.FASTLANES_DELTA); } @Test @@ -107,7 +104,8 @@ void unstable2026_02_0_addsToThePreviousUnstable() { Set result = Editions.cumulativeMembers(Editions.UNSTABLE_2026_02_0); // Then - assertThat(result).isEqualTo(ids("fastlanes.delta", "vortex.zstd_buffers")); + assertThat(result).containsExactlyInAnyOrder( + EncodingId.FASTLANES_DELTA, new EncodingId.Custom("vortex.zstd_buffers")); } @Test @@ -117,11 +115,13 @@ void unstable2026_04_0_addsToPreviousUnstable() { Set result = Editions.cumulativeMembers(Editions.UNSTABLE_2026_04_0); // Then - assertThat(result).isEqualTo(ids( - "fastlanes.delta", "vortex.zstd_buffers", - "vortex.parquet.variant", "vortex.patched", - "vortex.tensor.cosine_similarity", "vortex.tensor.inner_product", - "vortex.tensor.l2_denorm", "vortex.tensor.l2_norm")); + assertThat(result).containsExactlyInAnyOrder( + EncodingId.FASTLANES_DELTA, new EncodingId.Custom("vortex.zstd_buffers"), + new EncodingId.Custom("vortex.parquet.variant"), EncodingId.VORTEX_PATCHED, + new EncodingId.Custom("vortex.tensor.cosine_similarity"), + new EncodingId.Custom("vortex.tensor.inner_product"), + new EncodingId.Custom("vortex.tensor.l2_denorm"), + new EncodingId.Custom("vortex.tensor.l2_norm")); } @Test @@ -131,13 +131,14 @@ void unstable2026_06_0_isTheFullUnstableSet() { Set result = Editions.cumulativeMembers(Editions.UNSTABLE_2026_06_0); // Then - assertThat(result).hasSize(9) - .isEqualTo(ids( - "fastlanes.delta", "vortex.zstd_buffers", - "vortex.parquet.variant", "vortex.patched", - "vortex.tensor.cosine_similarity", "vortex.tensor.inner_product", - "vortex.tensor.l2_denorm", "vortex.tensor.l2_norm", - "vortex.onpair")); + assertThat(result).hasSize(9).containsExactlyInAnyOrder( + EncodingId.FASTLANES_DELTA, new EncodingId.Custom("vortex.zstd_buffers"), + new EncodingId.Custom("vortex.parquet.variant"), EncodingId.VORTEX_PATCHED, + new EncodingId.Custom("vortex.tensor.cosine_similarity"), + new EncodingId.Custom("vortex.tensor.inner_product"), + new EncodingId.Custom("vortex.tensor.l2_denorm"), + new EncodingId.Custom("vortex.tensor.l2_norm"), + new EncodingId.Custom("vortex.onpair")); } @Test @@ -147,23 +148,19 @@ void editionNotInAll_cumulativeSeedsFromItsOwnAddedRatherThanRequiringIdentityIn // in production), so this exercises cumulativeMembers' seeding logic directly rather // than a reachable real-world scenario. Edition hypothetical = new Edition( - new EditionId(EditionFamily.CORE, 2099, 1, 0), Optional.of("9.0.0"), - Set.of(EncodingId.parse("vortex.future"))); + new EditionId(EditionFamily.CORE, YearMonth.of(2099, 1), 0), Optional.of("9.0.0"), + Set.of(new EncodingId.Custom("vortex.future"))); // When Set result = Editions.cumulativeMembers(hypothetical); // Then — its own addition, plus one member from every earlier core edition assertThat(result).contains( - EncodingId.parse("vortex.future"), // its own addition + new EncodingId.Custom("vortex.future"), // its own addition EncodingId.VORTEX_PRIMITIVE, // core2025.05.0 EncodingId.VORTEX_ZSTD, // core2025.06.0 EncodingId.VORTEX_MASKED, // core2025.10.0 - EncodingId.VORTEX_VARIANT); // core2026.07.0 - } - - private Set ids(String... rawIds) { - return Stream.of(rawIds).map(EncodingId::parse).collect(Collectors.toUnmodifiableSet()); + EncodingId.VORTEX_VARIANT); // core2026.07.0 } } diff --git a/docs/reference.md b/docs/reference.md index 114f6fd2..0faa352c 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -64,7 +64,7 @@ shortcut returning a nullable copy), `withNullable(boolean)`, `DType.Struct.fiel | `LayoutId` | `sealed interface` — `WellKnown` enum + `Custom` record | Layout identity (separate namespace from encodings; `vortex.flat` is layout-only); both zoned aliases `vortex.zoned`/`vortex.stats` | | `ColumnName` | `record ColumnName(String value)` | Validated column name: non-blank, no control characters. `ColumnName.violation(String)` is the policy chokepoint shared by builder, writer, and file parser | | `EditionFamily` | `enum` — `CORE`, `UNSTABLE` | Closed (unlike `EncodingId`/`LayoutId`): an edition family is a cross-implementation compatibility promise, so a custom family carries no real guarantee — see [Editions](#editions) | -| `EditionId` | `record EditionId(EditionFamily family, int year, int month, int version)` | e.g. `core2025.05.0`; `isAtOrBefore` orders editions within a family only | +| `EditionId` | `record EditionId(EditionFamily family, YearMonth cutMonth, int version)` | e.g. `core2025.05.0`; `isAtOrBefore` orders editions within a family only | | `Edition` | `record Edition(EditionId id, Optional minVortexVersion, Set added)` | Only `Editions`'s 8 catalog constants should be constructed (API contract, not compiler-enforced — see ADR 0023) | ## Reader API