From 89b3a3a8d083010aa1db9ae0b57d4f8de85db36d Mon Sep 17 00:00:00 2001 From: Trevor Burnham Date: Fri, 4 Sep 2026 09:47:57 -0400 Subject: [PATCH] sqlite: add virtual table support via createModule() Expose SQLite's virtual table API through a new `database.createModule(name, options)` method, wrapping `sqlite3_create_module_v2()`. This enables read-only virtual tables backed by JavaScript data sources, usable either as an eponymous table (`SELECT * FROM module_name`) or via `CREATE VIRTUAL TABLE t USING module_name`. Hidden columns pass parameters using table-valued function syntax (`SELECT * FROM module_name(param1, param2)`). `options` accepts `columns`, `rows`, `directOnly`, and `useBigIntArguments`. Column types are validated against INTEGER, TEXT, REAL, BLOB, and ANY, and column names are quoted when building the `sqlite3_declare_vtab()` schema. Rebased from https://github.com/nodejs/node/pull/61544, which was opened by byteforge38 and became inactive. Changes on top of that work: - xColumn reports the value each hidden column was constrained to, rather than NULL. SQLite treats xBestIndex's `omit` as a hint, so it may recheck a constraint it already handed to xFilter; against NULL that recheck rejected every row, and `gs(1, 3) WHERE start = 1` returned no rows. - xBestIndex lowers estimatedCost as it consumes constraints. With a constant cost the planner was free to pick the unconstrained plan and recheck afterwards, so a correlated parameter such as `FROM t, gs(t.a, t.a + 1)` also returned no rows. - Violations of the iteration protocol report a SQLite error instead of calling PropagateJSError with no JavaScript exception pending. That left `.all()` returning undefined and `exec()` reporting success. - xBestIndex passes the constrained hidden-column indices to xFilter through idxStr rather than an int bitmask, which previously aliased for parameter indices at or above the width of an int. - xFilter, xNext, and xColumn take a CallbackDepthGuard. Without it close() from inside rows(), an iterator's next(), or a row getter finalized the statement that SQLite was still stepping, crashing the process. - xClose calls the iterator's return() method so generator `finally` blocks run when SQLite stops stepping early, as it does for LIMIT or a `break` out of a for...of loop. It is skipped while tearing down from ~StatementSync or ~DatabaseSync, which run from garbage collection callbacks where JavaScript cannot be executed; an abandoned generator does not run `finally` in JavaScript either. It is also skipped when an error is already pending, so that error still reaches the caller. - VirtualTableModule holds a BaseObjectWeakPtr to match UserDefinedFunction instead of a raw pointer. - createModule() rejects being called from an authorizer callback. - Documents that values yielded by rows() follow the usual conversion rules, so a number is stored as REAL and a BigInt as INTEGER even when a column declares INTEGER, since virtual tables do not apply column affinity to the values they return. Refs: https://github.com/nodejs/node/pull/61544 Refs: https://github.com/nodejs/node/issues/63826 Fixes: https://github.com/nodejs/node/issues/61539 Co-authored-by: byteforge38 Signed-off-by: Trevor Burnham Assisted-by: Claude Opus 5 --- doc/api/sqlite.md | 109 +++ src/node_sqlite.cc | 713 +++++++++++++++++++ src/node_sqlite.h | 108 +++ test/parallel/test-sqlite-virtual-table.js | 761 +++++++++++++++++++++ 4 files changed, 1691 insertions(+) create mode 100644 test/parallel/test-sqlite-virtual-table.js diff --git a/doc/api/sqlite.md b/doc/api/sqlite.md index 7d021381fc94..a22aa9ba0e11 100644 --- a/doc/api/sqlite.md +++ b/doc/api/sqlite.md @@ -864,6 +864,114 @@ console.log(allUsers); // ] ``` +### `database.createModule(name, options)` + + + +* `name` {string} The name of the virtual table module. This name is used in + `CREATE VIRTUAL TABLE ... USING name` statements and as an eponymous table + name. +* `options` {Object} Module configuration settings. + * `columns` {Array} An array of column definitions. Each element is an object + with the following properties: + * `name` {string} The name of the column. + * `type` {string} The declared type of the column. Must be one of + `'INTEGER'`, `'TEXT'`, `'REAL'`, `'BLOB'`, or `'ANY'`. + * `hidden` {boolean} If `true`, the column is hidden and acts as a + parameter for table-valued function usage. **Default:** `false`. + * `rows` {Function} A function called to produce rows when the virtual table + is queried. The function receives values for hidden columns (parameters) as + arguments, in the order they are defined. Must return an iterable (such as + an array or generator) where each element is an array of column values. + * `directOnly` {boolean} If `true`, the virtual table can only be used in + top-level SQL statements and cannot be used inside triggers or views. + **Default:** `false`. + * `useBigIntArguments` {boolean} If `true`, integer parameters passed to + `rows` are converted to `BigInt`s. **Default:** `false`. + +Registers a virtual table module with the database. This method is a wrapper +around [`sqlite3_create_module_v2()`][]. Virtual tables allow JavaScript code +to provide the backing data for SQL tables. The registered module can be used +in two ways: + +* **Eponymous table**: Query the module name directly without creating a table + (e.g., `SELECT * FROM module_name`). +* **Named virtual table**: Use `CREATE VIRTUAL TABLE t USING module_name` to + create a persistent virtual table. + +Hidden columns can be used to pass parameters to the `rows` function using +table-valued function syntax (e.g., `SELECT * FROM module_name(param1, param2)`). + +Values yielded by `rows` follow the conversion rules in [Type conversion between +JavaScript and SQLite][]: a {number} is stored as `REAL` and a {bigint} is +stored as `INTEGER`, regardless of the column's declared `type`. Unlike an +ordinary table, a virtual table does not apply column affinity to the values it +returns, so yield a {bigint} when a column needs `INTEGER` storage: + +```js +db.createModule('counter', { + columns: [{ name: 'value', type: 'INTEGER' }], + *rows() { + yield [1]; // typeof(value) is 'real' + yield [2n]; // typeof(value) is 'integer' + }, +}); +``` + +```cjs +const { DatabaseSync } = require('node:sqlite'); + +const db = new DatabaseSync(':memory:'); + +db.createModule('generate_series', { + columns: [ + { name: 'value', type: 'INTEGER' }, + { name: 'start', type: 'INTEGER', hidden: true }, + { name: 'stop', type: 'INTEGER', hidden: true }, + { name: 'step', type: 'INTEGER', hidden: true }, + ], + *rows(start, stop, step) { + start ??= 0; + stop ??= 10; + step ??= 1; + for (let i = start; i <= stop; i += step) { + yield [i]; + } + }, +}); + +console.log(db.prepare('SELECT * FROM generate_series(1, 5, 1)').all()); +// Prints: [ { value: 1 }, { value: 2 }, { value: 3 }, { value: 4 }, { value: 5 } ] +``` + +```mjs +import { DatabaseSync } from 'node:sqlite'; + +const db = new DatabaseSync(':memory:'); + +db.createModule('generate_series', { + columns: [ + { name: 'value', type: 'INTEGER' }, + { name: 'start', type: 'INTEGER', hidden: true }, + { name: 'stop', type: 'INTEGER', hidden: true }, + { name: 'step', type: 'INTEGER', hidden: true }, + ], + *rows(start, stop, step) { + start ??= 0; + stop ??= 10; + step ??= 1; + for (let i = start; i <= stop; i += step) { + yield [i]; + } + }, +}); + +console.log(db.prepare('SELECT * FROM generate_series(1, 5, 1)').all()); +// Prints: [ { value: 1 }, { value: 2 }, { value: 3 }, { value: 4 }, { value: 5 } ] +``` + ### `database.createSession([options])`