Skip to content

sqlite: add virtual table support via createModule() - #65787

Open
TrevorBurnham wants to merge 1 commit into
nodejs:mainfrom
TrevorBurnham:sqlite-vtab-create-module
Open

sqlite: add virtual table support via createModule()#65787
TrevorBurnham wants to merge 1 commit into
nodejs:mainfrom
TrevorBurnham:sqlite-vtab-create-module

Conversation

@TrevorBurnham

@TrevorBurnham TrevorBurnham commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #61539

This PR is a continuation of #61544 by @byteforge38, which became inactive.

It exposes SQLite's virtual table API through a new database.createModule(name, options) method, wrapping sqlite3_create_module_v2().

This PR enables read-only virtual tables backed by JavaScript data sources. A registered module can be used two ways:

  • Eponymous table: query the module name directly (SELECT * FROM module_name).
  • Named virtual table: CREATE VIRTUAL TABLE t USING module_name.

Hidden columns pass parameters via table-valued function syntax (SELECT * FROM module_name(param1, param2)).

options accepts columns, rows, directOnly, and useBigIntArguments.

Note: column affinity

A virtual table doesn't apply column affinity to the values xColumn returns, which makes declared types behave differently than they do on an ordinary table:

// real table: affinity converts the bound double
db.exec('CREATE TABLE t(v INTEGER)');
db.prepare('INSERT INTO t VALUES (?)').run(7);
db.prepare('SELECT typeof(v) FROM t').get();   // 'integer'

// virtual table: no affinity applied
db.createModule('gs', { columns: [{ name: 'v', type: 'INTEGER' }], *rows() { yield [7]; } });
db.prepare('SELECT typeof(v) FROM gs').get();  // 'real'

This follows from the conversion rules discussed in #63826: A number is bound as REAL, a bigint as INTEGER. Value-based coercion would make it impossible to yield 666.0 into a REAL column, and bigint already gives callers explicit control.

Coercing to the declared type would be a different mechanism, driven by explicit intent rather than by guessing from the value, and it would leave REAL columns alone. But SQLite core doesn't do this for virtual tables (generate_series returns integers because it calls sqlite3_result_int64, not via affinity), and it adds per-cell cost.

I've documented the current behavior rather than changing it, since coercing to the declared type is behavior SQLite core doesn't have for virtual tables.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/sqlite

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. sqlite Issues and PRs related to the SQLite subsystem. labels Sep 4, 2026
@TrevorBurnham
TrevorBurnham force-pushed the sqlite-vtab-create-module branch 2 times, most recently from 7862ab0 to 69f939a Compare September 4, 2026 19:28
Expose SQLite's virtual table API through a new
`database.createModule(name, options)` method, wrapping
`sqlite3_create_module_v2()`. This enables read-only virtual tables
backed by JavaScript data sources, usable either as an eponymous table
(`SELECT * FROM module_name`) or via `CREATE VIRTUAL TABLE t USING
module_name`. Hidden columns pass parameters using table-valued
function syntax (`SELECT * FROM module_name(param1, param2)`).

`options` accepts `columns`, `rows`, `directOnly`, and
`useBigIntArguments`. Column types are validated against INTEGER,
TEXT, REAL, BLOB, and ANY, and column names are quoted when building
the `sqlite3_declare_vtab()` schema.

Rebased from nodejs#61544, which was
opened by byteforge38 and became inactive. Changes on top of that
work:

- xColumn reports the value each hidden column was constrained to,
  rather than NULL. SQLite treats xBestIndex's `omit` as a hint, so it
  may recheck a constraint it already handed to xFilter; against NULL
  that recheck rejected every row, and `gs(1, 3) WHERE start = 1`
  returned no rows.
- xBestIndex lowers estimatedCost as it consumes constraints. With a
  constant cost the planner was free to pick the unconstrained plan and
  recheck afterwards, so a correlated parameter such as
  `FROM t, gs(t.a, t.a + 1)` also returned no rows.
- Violations of the iteration protocol report a SQLite error instead of
  calling PropagateJSError with no JavaScript exception pending. That
  left `.all()` returning undefined and `exec()` reporting success.
- xBestIndex passes the constrained hidden-column indices to xFilter
  through idxStr rather than an int bitmask, which previously aliased
  for parameter indices at or above the width of an int.
- xFilter, xNext, and xColumn take a CallbackDepthGuard. Without it
  close() from inside rows(), an iterator's next(), or a row getter
  finalized the statement that SQLite was still stepping, crashing the
  process.
- xClose calls the iterator's return() method so generator `finally`
  blocks run when SQLite stops stepping early, as it does for LIMIT or
  a `break` out of a for...of loop. It is skipped while tearing down
  from ~StatementSync or ~DatabaseSync, which run from garbage
  collection callbacks where JavaScript cannot be executed; an
  abandoned generator does not run `finally` in JavaScript either. It
  is also skipped when an error is already pending, so that error still
  reaches the caller.
- VirtualTableModule holds a BaseObjectWeakPtr<DatabaseSync> to match
  UserDefinedFunction instead of a raw pointer.
- createModule() rejects being called from an authorizer callback.
- Documents that values yielded by rows() follow the usual conversion
  rules, so a number is stored as REAL and a BigInt as INTEGER even
  when a column declares INTEGER, since virtual tables do not apply
  column affinity to the values they return.

Refs: nodejs#61544
Refs: nodejs#63826
Fixes: nodejs#61539

Co-authored-by: byteforge38 <stormcraft318@gmail.com>
Signed-off-by: Trevor Burnham <trevorburnham@gmail.com>
Assisted-by: Claude Opus 5
@TrevorBurnham
TrevorBurnham force-pushed the sqlite-vtab-create-module branch from 69f939a to 89b3a3a Compare September 4, 2026 21:04
@TrevorBurnham
TrevorBurnham marked this pull request as ready for review September 4, 2026 23:17
@codecov

codecov Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.25000% with 87 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.12%. Comparing base (68e0ce0) to head (89b3a3a).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
src/node_sqlite.cc 80.96% 44 Missing and 43 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #65787      +/-   ##
==========================================
- Coverage   90.13%   90.12%   -0.02%     
==========================================
  Files         769      769              
  Lines      261645   262109     +464     
  Branches    49671    49755      +84     
==========================================
+ Hits       235831   236217     +386     
- Misses      16845    16856      +11     
- Partials     8969     9036      +67     
Files with missing lines Coverage Δ
src/node_sqlite.h 87.27% <100.00%> (+0.86%) ⬆️
src/node_sqlite.cc 82.04% <80.96%> (-0.20%) ⬇️

... and 36 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. sqlite Issues and PRs related to the SQLite subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose SQLite virtual table API

2 participants