[feat] tiled fixed size list array (PDX-style) by lwwmanning · Pull Request #9118 · vortex-data/vortex

lwwmanning · 2026-07-31T14:50:32Z

Rationale for this change

Many vector workloads, including vector indices, benefit from storing fixed-size primitive lists in a two-dimensional, PDX-style layout so kernels can process tiles directly instead of transposing rows during scoring. This adds an experimental tiled fixed-size-list encoding behind unstable_encodings; it does not carry a stable wire-format guarantee.

What changes are included in this PR?

Adds the vortex-tiled-fsl crate with configurable row and dimension tiles, checked geometry, scalar access, canonical take fallback, file round trips, benchmarks, and differential fuzzing against canonical fixed-size lists. Full-width slices are O(1) row views at the TiledFSL layer and retain at most the selected rows plus two boundary tiles; a local slice-reduce benchmark measured 42.05 ns at 1,024 rows and 43.04 ns at 1,000,000 rows. Aligned multi-slab slices use one PiecewiseSequenceArray run per dimension slab. Unaligned multi-slab slices remain lazy and, when executed, issue a bounded slice to each child slab before canonical assembly; child-internal work remains encoding-dependent.

Element validity remains encapsulated by the nullable primitive elements child, while TiledFSL owns only outer-row validity. Serialized row views carry their logical offset and retained backing extent, including nullable FastLanes-bitpacked children. Tensor and TiledFSL registration remain available without the files feature and initialize only once in default sessions.

The default compressor does not select or preserve this encoding automatically. Callers that persist tiled arrays must choose an appropriate layout strategy for the relevant field. Coverage includes canonical conformance, malformed serde inputs, file round trips, deterministic stateful fuzzing, a 10,000-run libFuzzer pass, and structural scaling checks.

What APIs are changed? Are there any user-facing changes?

vortex::encodings::tiled_fsl is available behind unstable_encodings, and the unstable edition permits vortex.tiled_fsl. The experimental API exposes tiled array construction, TileGeometry, tile-bound iteration, encoding from primitive fixed-size lists, and access to the physical child and row-view metadata.

🤖 Generated with Codex

Signed-off-by: Will Manning <will@willmanning.io>

…write Signed-off-by: Will Manning <will@willmanning.io>

Signed-off-by: Will Manning <will@willmanning.io>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d4dfc88bfc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-03T00:25:51Z

+                        if let (Some(source_validity), Some(output_validity)) =
+                            (&source_validity, &mut output_validity)
+                        {
+                            output_validity.set_to(physical, source_validity.value(canonical));


Avoid bit-at-a-time validity transposition

When nullable primitive elements resolve to Mask::Values, encoding performs a BitBuffer::value lookup and branch for every scalar; the inverse path repeats the same pattern at line 179. For large vector arrays this makes validity transposition a bit-at-a-time hot loop and discards the bitmap word-level fast paths, so transpose validity by u64 chunks with all-set/all-unset handling (or iterate set indices into an unset destination) and cover the nullable case in the transpose benchmark.

AGENTS.md reference: AGENTS.md:L185-L190

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-08-03T00:25:51Z

+
+pub const UNSTABLE_2026_07_0: EditionId = EditionId::new("unstable", 2026, 7, 0);
+


Document the new public edition declarations

Add API documentation for UNSTABLE_2026_07_0 and DECLARATION, as well as module-level documentation for this publicly exported edition module. Every neighboring unstable-edition module documents these public definitions, while this new module exposes them without any doc comments, contrary to the repository requirement for every new public API definition.

AGENTS.md reference: AGENTS.md:L146-L146

Useful? React with 👍 / 👎.

Signed-off-by: Will Manning <will@willmanning.io>

robert3005 · 2026-08-07T11:00:48Z

+}
+
+#[expect(clippy::result_large_err)]
+fn fuzz<T>(result: VortexResult<T>) -> VortexFuzzResult<T> {


this function is terrible, it's just a map_err, inline it everywhere and a lot of weird code will go away imho

robert3005 · 2026-08-07T11:01:42Z

+                        tiled.encoding_id()
+                    );
+                    Ok(())
+                })())?;


We run reduce operations so I don't think you will have a slice here always, looks like this sometimes will end up with just the tiled array

robert3005 · 2026-08-07T16:09:41Z

+use vortex_fuzz::run_tiled_fsl;
+
+fuzz_target!(|input: FuzzTiledFsl| -> Corpus {
+    match run_tiled_fsl(input) {


I think you can unwrap the error here and use vortexresult throughout the module. I only have VortexFuzzResult so that we have the necessary debug information for fuzzing

robert3005 · 2026-08-07T16:12:16Z

+            .collect::<VortexResult<Buffer<u64>>>()?;
+        let row = array
+            .elements()
+            .take(PrimitiveArray::from_iter(indices).into_array())?


This should not need from_iter, you already have a Buffer

robert3005 · 2026-08-07T16:12:51Z

+                Mask::Values(validity) => values
+                    .iter()
+                    .copied()
+                    .zip(validity.bit_buffer().iter())


bit_buffer().for_each_set_index?

robert3005 · 2026-08-07T16:13:21Z

+        }
+        let elements = elements.finish();
+        let decoded = decode_visible_elements(
+            elements.as_::<Primitive>(),


how do we know this is primitive?

robert3005 · 2026-08-07T16:13:36Z

+            let span_elements = array
+                .elements()
+                .slice(span)?
+                .execute::<PrimitiveArray>(ctx)?;


this is weird

robert3005 · 2026-08-07T16:14:10Z

+    let mut starts = Vec::<u64>::with_capacity(dimension_slab_count);
+    let mut lengths = Vec::<u64>::with_capacity(dimension_slab_count);
+
+    for span in spans {
+        starts.push(u64::try_from(span.start)?);
+        lengths.push(u64::try_from(span.len())?);
+    }


build buffers directly

robert3005 · 2026-08-07T16:23:47Z

+        let validity = self
+            .as_ref()
+            .slots()
+            .get(TiledFixedSizeListSlots::VALIDITY)


You will get TiledFixedSizeListSlotsExt which will give you a method so you can make this be just self.validity(). See StructArray

lwwmanning added 30 commits July 30, 2026 18:08

docs: design tiled fixed-size-list encoding

f6f63e8

Signed-off-by: Will Manning <will@willmanning.io>

docs: derive tiled list dtype from parts

5ce6464

Signed-off-by: Will Manning <will@willmanning.io>

docs: expose tiled physical ranges

f2b5482

Signed-off-by: Will Manning <will@willmanning.io>

docs: plan tiled fixed-size-list implementation

5460aaa

Signed-off-by: Will Manning <will@willmanning.io>

docs: plan tiled list differential fuzzing

51b9483

Signed-off-by: Will Manning <will@willmanning.io>

docs: harden tiled list implementation plan

515c07d

Signed-off-by: Will Manning <will@willmanning.io>

docs: make tiled list plan executable

104c039

Signed-off-by: Will Manning <will@willmanning.io>

feat: add tiled fixed-size-list geometry

b34c1b2

Signed-off-by: Will Manning <will@willmanning.io>

feat: add tiled list array metadata

b79bd81

Signed-off-by: Will Manning <will@willmanning.io>

feat: transpose tiled fixed-size lists

6842de3

Signed-off-by: Will Manning <will@willmanning.io>

fix: harden tiled list scalar execution

9097e9f

Signed-off-by: Will Manning <will@willmanning.io>

feat: expose tiled list ranges

72355ed

Signed-off-by: Will Manning <will@willmanning.io>

fix: validate tiled geometry widths

fd642d9

Signed-off-by: Will Manning <will@willmanning.io>

feat: preserve tiled lists through slice

1d2a2b2

Signed-off-by: Will Manning <will@willmanning.io>

fix: reject overflowing tiled row gathers

fed9a5b

Signed-off-by: Will Manning <will@willmanning.io>

feat: preserve tiled lists through take

a86b593

Signed-off-by: Will Manning <will@willmanning.io>

fix: validate tiled take mask length

f83bcac

Signed-off-by: Will Manning <will@willmanning.io>

test: cover bitpacked tiled list children

ba2eebb

Signed-off-by: Will Manning <will@willmanning.io>

test: verify bitpacked tiled row values

c109d1d

Signed-off-by: Will Manning <will@willmanning.io>

feat: register tiled lists as unstable encoding

4a5eaad

Signed-off-by: Will Manning <will@willmanning.io>

test: check tiled lists against canonical FSL

080788a

Signed-off-by: Will Manning <will@willmanning.io>

test: make tiled conformance oracle independent

37f6e88

Signed-off-by: Will Manning <will@willmanning.io>

test: fuzz tiled lists against canonical FSL

4f075c7

Signed-off-by: Will Manning <will@willmanning.io>

chore: keep task reports out of source

1041cf9

Signed-off-by: Will Manning <will@willmanning.io>

ci: schedule tiled list fuzzing

22b4022

Signed-off-by: Will Manning <will@willmanning.io>

bench: measure tiled fixed-size lists

14ccd31

Signed-off-by: Will Manning <will@willmanning.io>

bench: distinguish tiled benchmark rows

3df4777

Signed-off-by: Will Manning <will@willmanning.io>

fix: satisfy tiled list source licensing

1f22f6a

Signed-off-by: Will Manning <will@willmanning.io>

docs: clarify tiled empty slice semantics

0307663

Signed-off-by: Will Manning <will@willmanning.io>

test: cover empty tiled slice composition

de9cf8f

Signed-off-by: Will Manning <will@willmanning.io>

lwwmanning added 17 commits August 1, 2026 15:08

fix: bound multi-slab tiled FSL slices

4e19483

Signed-off-by: Will Manning <will@willmanning.io>

fix: push down tiled FSL slab slices

58b3e64

Signed-off-by: Will Manning <will@willmanning.io>

feat: serialize tiled FSL row views

a81f6b8

Signed-off-by: Will Manning <will@willmanning.io>

test: fuzz tiled FSL row views

e2a64cf

Signed-off-by: Will Manning <will@willmanning.io>

test: validate reconstructed tiled FSL trees

674f292

Signed-off-by: Will Manning <will@willmanning.io>

test: deduplicate tiled FSL serde coverage

9b3aa32

Signed-off-by: Will Manning <will@willmanning.io>

bench: pin tiled FSL slice scaling

b09f8cd

Signed-off-by: Will Manning <will@willmanning.io>

test: tighten tiled FSL slice scaling check

a6a5262

Signed-off-by: Will Manning <will@willmanning.io>

test: consolidate tiled FSL coverage

4ec7dea

Signed-off-by: Will Manning <will@willmanning.io>

test: restore oversized tiled slice regression

42f2bfb

Signed-off-by: Will Manning <will@willmanning.io>

chore: clean up tiled FSL geometry

a59a3be

Signed-off-by: Will Manning <will@willmanning.io>

Merge remote-tracking branch 'origin/develop' into codex/tiled-dco-re…

503fbc5

…write Signed-off-by: Will Manning <will@willmanning.io>

fix: reject mismatched tiled FSL validity

677f87f

Signed-off-by: Will Manning <will@willmanning.io>

fix: validate tiled FSL row views in fuzz oracle

89d16e9

Signed-off-by: Will Manning <will@willmanning.io>

fix: avoid duplicate unstable initialization

ce41e40

Signed-off-by: Will Manning <will@willmanning.io>

fix(tiled-fsl): harden row view invariants

370b989

Signed-off-by: Will Manning <will@willmanning.io>

feat(tiled-fsl): preserve encoding through masks

d4dfc88

Signed-off-by: Will Manning <will@willmanning.io>

lwwmanning requested a review from connortsui20 August 1, 2026 21:30

lwwmanning marked this pull request as ready for review August 3, 2026 00:20

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

fix(tiled-fsl): satisfy unwrap lint

4ec4daf

Signed-off-by: Will Manning <will@willmanning.io>

robert3005 reviewed Aug 7, 2026

View reviewed changes

Navigation Menu

Uh oh!

[feat] tiled fixed size list array (PDX-style) - #9118

[feat] tiled fixed size list array (PDX-style)#9118
lwwmanning wants to merge 54 commits into
developfrom
codex/tiled-fixed-size-list

lwwmanning commented Jul 31, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 3, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 3, 2026

Uh oh!

robert3005 Aug 7, 2026

Uh oh!

robert3005 Aug 7, 2026

Uh oh!

robert3005 Aug 7, 2026

Uh oh!

robert3005 Aug 7, 2026

Uh oh!

robert3005 Aug 7, 2026

Uh oh!

robert3005 Aug 7, 2026

Uh oh!

robert3005 Aug 7, 2026

Uh oh!

robert3005 Aug 7, 2026

Uh oh!

robert3005 Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		pub const UNSTABLE_2026_07_0: EditionId = EditionId::new("unstable", 2026, 7, 0);

Uh oh!

Conversation

lwwmanning commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

What APIs are changed? Are there any user-facing changes?

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lwwmanning commented Jul 31, 2026 •

edited

Loading