Skip to content

Add local read-at support for compio retrieval - #9294

Draft
danking wants to merge 7 commits into
developfrom
dk/local-read-at
Draft

Add local read-at support for compio retrieval#9294
danking wants to merge 7 commits into
developfrom
dk/local-read-at

Conversation

@danking

@danking danking commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Authored by Codex.

Summary

Add the Vortex-side APIs needed for Spiral's compio/cyper object-storage client path.

  • Add a local, non-Send read-at interface alongside the existing Send interface.
  • Add OpenOptions::open_read_local so footer discovery can run on a caller-selected local I/O worker.
  • Keep each local file's segment retrieval driver pinned to the same LocalIoWorker; the driver's cyper/compio state machine never moves between threads.
  • Preserve Vortex's existing segment request queue and completion callbacks as the retrieval boundary.
  • Keep scan tasks themselves Send. Query workers await scan tasks and perform pruning, decode, and projection; only segment retrieval executes on the compio worker.
  • Remove the earlier scan-level local task API after review showed it placed CPU/query processing on I/O event-loop threads.

This is the Vortex dependency for Spiral draft PR https://github.com/spiraldb/spiraldb/pull/6406.

Design

The structure follows the retrieval/query-worker split in sections 3 and 4 of Exploiting Cloud Object Storage: a small number of retrieval event loops own many state-machine requests, while query workers process completed morsels. File affinity is intentional because compio/cyper futures are non-Send; hostname affinity is not used.

Validation

  • cargo +nightly-2026-06-29 check -p vortex-io -p vortex-layout -p vortex-file
  • Spiral's strict integration test opens a generated 4,096-row Vortex file through open_read_local, materializes every row, verifies only local provider methods were called, and verifies scan map/decode work did not execute on vortex-io-* threads.

This remains draft while the dependent Spiral branch is measured on Linux io_uring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant