From d7dc592ae3c06ec0e7bd28cfdc551abe477dadcb Mon Sep 17 00:00:00 2001 From: "Wid L. Hacker" Date: Sat, 31 Oct 2026 14:41:33 -1000 Subject: [PATCH 1/6] Docs: streamline SQLite requirements; document macOS loadable-extension gap --- docs/index.md | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/docs/index.md b/docs/index.md index 5e2d7851..fde1df52 100644 --- a/docs/index.md +++ b/docs/index.md @@ -16,36 +16,36 @@ This November 2021 poster discusses the context and long-run ambitions: [GenomicSQLite Poster](GenomeInformatics2021Poster.pdf) -Our **[Colab notebook](https://colab.research.google.com/drive/1OlHPOcRQBhDmEnS1wtOdtUGDkcD7LtKx?usp=sharing)** demonstrates key features with Python, one of several language bindings. +Our **[Colab notebook](https://colab.research.google.com/drive/1BND1nYHDg9DLsw5buvaB30BpH_GoYbit)** demonstrates key features with Python, one of several language bindings. **USE AT YOUR OWN RISK:** The extension makes fundamental changes to the database storage layer. While designed to preserve ACID transaction safety, it's young and unlikely to have zero bugs. This project is not associated with the SQLite developers. -## SQLite ≥ 3.31.0 required +## SQLite requirements -**To use the Genomics Extension you might first need to upgrade SQLite itself.** The host program must link [SQLite version 3.31.0 (2020-01-22)](https://www.sqlite.org/releaselog/3_31_0.html) or newer. In your shell, `sqlite3 --version` displays the version installed with your OS, which is probably what your programs use; if in doubt, cause a program to report the result of `SELECT sqlite_version()`. +The host program must support loading extensions and use SQLite 3.31.0 (2020) or newer. Check the version with `SELECT sqlite_version()` in your program. -If this is too old, then upgrade SQLite3 using your preferred binary package manager (e.g. apt, yum, conda, brew), if possible. Otherwise, modify your program's linking step or runtime environment to cause it to use an up-to-date version, for example by setting rpath or `LD_LIBRARY_PATH` to the location of an up-to-date shared library file. Resources: +The macOS system SQLite lacks extension-loading entry points, regardless of version, so programs linked to it cannot load the Genomics Extension. On macOS, use: -* [How To Compile SQLite](https://www.sqlite.org/howtocompile.html) -* [DreamHost Knowledge Base - Installing a custom version of SQLite3](https://help.dreamhost.com/hc/en-us/articles/360028047592-Installing-a-custom-version-of-SQLite3) -* [Rpmfind: libsqlite3](https://rpmfind.net/linux/rpm2html/search.php?query=libsqlite3&submit=Search+...&system=&arch=) -* [Sqlite :: Anaconda Cloud](https://anaconda.org/anaconda/sqlite) -* Homebrew [formula/sqlite](https://formulae.brew.sh/formula/sqlite), [formula-linux/sqlite](https://formulae.brew.sh/formula-linux/sqlite) -* As a last resort for GNU/Linux, our [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases) include a `libsqlite3.so.0` that should be compatible with modern (2015+) hosts. +* **Python:** [Homebrew Python](https://formulae.brew.sh/formula/python) or conda, which provide extension-loading support. +* **C/C++ & Rust:** [Homebrew's sqlite](https://formulae.brew.sh/formula/sqlite), selected by your build. For Rust's rusqlite dependency, set `SQLITE3_LIB_DIR` and `SQLITE3_INCLUDE_DIR` to its `lib` and `include` directories. -You can always `SELECT sqlite_version()` to verify the upgrade in your program. +The JVM binding is unaffected because sqlite-jdbc bundles its own SQLite. + +On other platforms, if SQLite is too old or lacks extension support, upgrade it with your package manager (e.g. apt, yum, conda, brew), or see [How To Compile SQLite](https://www.sqlite.org/howtocompile.html). Make sure your program uses the replacement build. As a last resort for GNU/Linux, our [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases) include a `libsqlite3.so.0` that should be compatible with modern (2015+) hosts. ## Installation It's usually easiest to obtain the extension as a pre-compiled shared library (Linux .so or macOS .dylib), either installed with a package manager, or downloaded from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases). +Whichever binding you use, we also recommend installing the Python package, for its command-line shell; and trying the smoke-test script under [Smoke test](#smoke-test) to verify your installation. + === "Python" ``` bash pip3 install [--user|--system] genomicsqlite # -or- conda install -c mlin genomicsqlite ``` - Python's `sqlite3` module must support [loadable extensions](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.enable_load_extension). Some macOS Python builds omit this support; Homebrew Python enables it. This is an interpreter build option, independent of the Python version. + On Ubuntu 24.04, the system Python (3.12.3) may emit a harmless `DeprecationWarning` about `?NNN` parameter binding. This CPython bug was fixed in 3.12.4. The package loads a bundled shared library by default. To override the bundled file, set environment variable `LIBGENOMICSQLITE` to a filename. @@ -85,8 +85,6 @@ It's usually easiest to obtain the extension as a pre-compiled shared library (L To bundle GenomicSQLite and sqlite-jdbc inside an "Uber JAR" for your application, you may need to enable the [ServicesResourceTransformer for Maven Shade](https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer) or [mergeServiceFiles() for Gradle Shadow](https://stackoverflow.com/questions/32887966/shadow-plugin-gradle-what-does-mergeservicefiles-do), to ensure the JDBC drivers will be registered automatically when the "Uber JAR" loads. - Recommendation: *also* install the Python package, which includes a useful command-line shell and smoke-test script. - === "Rust" Add to your project's `Cargo.toml`: @@ -100,8 +98,6 @@ It's usually easiest to obtain the extension as a pre-compiled shared library (L === "C/C++" Download zip of shared library and `genomicsqlite.h` from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases). Build your program with them, and also ensure the dynamic linker will find the shared library at runtime, by either: (1) installing it in a system or user lib directory & refreshing cache, (2) setting `LD_LIBRARY_PATH` environment variable, (3) building with `-rpath`. - Recommendation: *also* install the Python package, which includes a useful command-line shell and smoke-test script. - GNU/Linux: to link the prebuilt `libgenomicsqlite.so` distributed from our GitHub Releases, you may have to compile your source with `CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=0`. This is because the library is built against an old libstdc++ version to improve runtime compatibility. The From 6ed4ede954d08674b95df35f6da89eb10a439701 Mon Sep 17 00:00:00 2001 From: "Wid L. Hacker" Date: Sat, 31 Oct 2026 16:27:30 -1000 Subject: [PATCH 2/6] Docs: keep the Python-package recommendation in the non-Python tabs --- docs/index.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/index.md b/docs/index.md index fde1df52..962487a4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -37,8 +37,6 @@ On other platforms, if SQLite is too old or lacks extension support, upgrade it It's usually easiest to obtain the extension as a pre-compiled shared library (Linux .so or macOS .dylib), either installed with a package manager, or downloaded from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases). -Whichever binding you use, we also recommend installing the Python package, for its command-line shell; and trying the smoke-test script under [Smoke test](#smoke-test) to verify your installation. - === "Python" ``` bash pip3 install [--user|--system] genomicsqlite @@ -85,6 +83,8 @@ Whichever binding you use, we also recommend installing the Python package, for To bundle GenomicSQLite and sqlite-jdbc inside an "Uber JAR" for your application, you may need to enable the [ServicesResourceTransformer for Maven Shade](https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer) or [mergeServiceFiles() for Gradle Shadow](https://stackoverflow.com/questions/32887966/shadow-plugin-gradle-what-does-mergeservicefiles-do), to ensure the JDBC drivers will be registered automatically when the "Uber JAR" loads. + Recommendation: *also* install the Python package, which includes a useful command-line shell. + === "Rust" Add to your project's `Cargo.toml`: @@ -95,9 +95,13 @@ Whichever binding you use, we also recommend installing the Python package, for Building with the crate will include a platform-appropriate shared library file within your compilation unit, to be extracted & loaded at runtime. To disable this, add `default-features = false` and at runtime, set environment variable `LIBGENOMICSQLITE` to a filename or place the library file somewhere it'll be found by `dlopen("libgenomicsqlite")`. + Recommendation: *also* install the Python package, which includes a useful command-line shell. + === "C/C++" Download zip of shared library and `genomicsqlite.h` from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases). Build your program with them, and also ensure the dynamic linker will find the shared library at runtime, by either: (1) installing it in a system or user lib directory & refreshing cache, (2) setting `LD_LIBRARY_PATH` environment variable, (3) building with `-rpath`. + Recommendation: *also* install the Python package, which includes a useful command-line shell. + GNU/Linux: to link the prebuilt `libgenomicsqlite.so` distributed from our GitHub Releases, you may have to compile your source with `CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=0`. This is because the library is built against an old libstdc++ version to improve runtime compatibility. The From 25c76143a28d6cfbc8f3ec5951dd5c01260103ad Mon Sep 17 00:00:00 2001 From: "Wid L. Hacker" Date: Sat, 31 Oct 2026 17:22:47 -1000 Subject: [PATCH 3/6] Docs: Python package includes a command-line shell and other tools --- docs/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/index.md b/docs/index.md index 962487a4..13fa3194 100644 --- a/docs/index.md +++ b/docs/index.md @@ -83,7 +83,7 @@ It's usually easiest to obtain the extension as a pre-compiled shared library (L To bundle GenomicSQLite and sqlite-jdbc inside an "Uber JAR" for your application, you may need to enable the [ServicesResourceTransformer for Maven Shade](https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer) or [mergeServiceFiles() for Gradle Shadow](https://stackoverflow.com/questions/32887966/shadow-plugin-gradle-what-does-mergeservicefiles-do), to ensure the JDBC drivers will be registered automatically when the "Uber JAR" loads. - Recommendation: *also* install the Python package, which includes a useful command-line shell. + Recommendation: *also* install the Python package, which includes a useful command-line shell and other tools. === "Rust" Add to your project's `Cargo.toml`: @@ -95,12 +95,12 @@ It's usually easiest to obtain the extension as a pre-compiled shared library (L Building with the crate will include a platform-appropriate shared library file within your compilation unit, to be extracted & loaded at runtime. To disable this, add `default-features = false` and at runtime, set environment variable `LIBGENOMICSQLITE` to a filename or place the library file somewhere it'll be found by `dlopen("libgenomicsqlite")`. - Recommendation: *also* install the Python package, which includes a useful command-line shell. + Recommendation: *also* install the Python package, which includes a useful command-line shell and other tools. === "C/C++" Download zip of shared library and `genomicsqlite.h` from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases). Build your program with them, and also ensure the dynamic linker will find the shared library at runtime, by either: (1) installing it in a system or user lib directory & refreshing cache, (2) setting `LD_LIBRARY_PATH` environment variable, (3) building with `-rpath`. - Recommendation: *also* install the Python package, which includes a useful command-line shell. + Recommendation: *also* install the Python package, which includes a useful command-line shell and other tools. GNU/Linux: to link the prebuilt `libgenomicsqlite.so` distributed from our GitHub Releases, you may have to compile your source with `CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=0`. This is because the From 369c460e350424f5c5c756da5b0e3706dc83c1f0 Mon Sep 17 00:00:00 2001 From: Mike Lin Date: Sat, 31 Oct 2026 20:26:25 -1000 Subject: [PATCH 4/6] Revise SQLite requirements and installation instructions Updated requirements and installation instructions for SQLite and GenomicSQLite across various platforms, including clarifications on macOS and package manager options. --- docs/index.md | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/docs/index.md b/docs/index.md index 13fa3194..8759204e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -22,20 +22,15 @@ Our **[Colab notebook](https://colab.research.google.com/drive/1BND1nYHDg9DLsw5b ## SQLite requirements -The host program must support loading extensions and use SQLite 3.31.0 (2020) or newer. Check the version with `SELECT sqlite_version()` in your program. +The SQLite library linked into application must be at least version 3.31.0 (2020 vintage) and support [loadable extensions](https://www.sqlite.org/loadext.html). Check the version with `SELECT sqlite_version()` in your program. -The macOS system SQLite lacks extension-loading entry points, regardless of version, so programs linked to it cannot load the Genomics Extension. On macOS, use: +The **macOS** default SQLite unfortunately disables extension loading. Homebrew and conda provide compatible SQLite packages you can install and configure your build system (if applicable) to use. -* **Python:** [Homebrew Python](https://formulae.brew.sh/formula/python) or conda, which provide extension-loading support. -* **C/C++ & Rust:** [Homebrew's sqlite](https://formulae.brew.sh/formula/sqlite), selected by your build. For Rust's rusqlite dependency, set `SQLITE3_LIB_DIR` and `SQLITE3_INCLUDE_DIR` to its `lib` and `include` directories. - -The JVM binding is unaffected because sqlite-jdbc bundles its own SQLite. - -On other platforms, if SQLite is too old or lacks extension support, upgrade it with your package manager (e.g. apt, yum, conda, brew), or see [How To Compile SQLite](https://www.sqlite.org/howtocompile.html). Make sure your program uses the replacement build. As a last resort for GNU/Linux, our [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases) include a `libsqlite3.so.0` that should be compatible with modern (2015+) hosts. +On other platforms, if SQLite is too old or lacks extension support, upgrade it with your package manager or see [How To Compile SQLite](https://www.sqlite.org/howtocompile.html). As a last resort for GNU/Linux, our [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases) attach a widely-compatible `libsqlite3.so.0`. ## Installation -It's usually easiest to obtain the extension as a pre-compiled shared library (Linux .so or macOS .dylib), either installed with a package manager, or downloaded from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases). +It's usually easiest to obtain the extension as a pre-compiled shared library (GNU/Linux .so or macOS .dylib), either installed with a package manager, or downloaded from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases). === "Python" ``` bash @@ -79,7 +74,7 @@ It's usually easiest to obtain the extension as a pre-compiled shared library (L } ``` - Alternatively, download the JAR from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases) and place it in your classpath, along with [sqlite-jdbc](https://github.com/xerial/sqlite-jdbc)'s JAR which is also required. The package loads a bundled shared library by default. To override the bundled file, set environment variable `LIBGENOMICSQLITE` to a filename. + Alternatively, download the JAR from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases) and place it in your classpath, along with [sqlite-jdbc](https://github.com/xerial/sqlite-jdbc)'s JAR which is also required. The GenomicSQLite JAR loads a bundled shared library by default. To override the bundled file, set environment variable `LIBGENOMICSQLITE` to a filename. sqlite-jdbc similarly bundles a SQLite library. To bundle GenomicSQLite and sqlite-jdbc inside an "Uber JAR" for your application, you may need to enable the [ServicesResourceTransformer for Maven Shade](https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer) or [mergeServiceFiles() for Gradle Shadow](https://stackoverflow.com/questions/32887966/shadow-plugin-gradle-what-does-mergeservicefiles-do), to ensure the JDBC drivers will be registered automatically when the "Uber JAR" loads. @@ -93,15 +88,15 @@ It's usually easiest to obtain the extension as a pre-compiled shared library (L version = "^0" ``` - Building with the crate will include a platform-appropriate shared library file within your compilation unit, to be extracted & loaded at runtime. To disable this, add `default-features = false` and at runtime, set environment variable `LIBGENOMICSQLITE` to a filename or place the library file somewhere it'll be found by `dlopen("libgenomicsqlite")`. + Building with the crate will include a platform-appropriate shared library file within your compilation unit, to be extracted & loaded at runtime. To disable this, add `default-features = false` to the dependencies entry and at runtime, set environment variable `LIBGENOMICSQLITE` to a filename or place the library file somewhere it'll be found by `dlopen("libgenomicsqlite")`. + + To make `rusqlite` use an SQLite library other than the system's default, set `SQLITE3_LIB_DIR` and `SQLITE3_INCLUDE_DIR`. Recommendation: *also* install the Python package, which includes a useful command-line shell and other tools. === "C/C++" Download zip of shared library and `genomicsqlite.h` from [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases). Build your program with them, and also ensure the dynamic linker will find the shared library at runtime, by either: (1) installing it in a system or user lib directory & refreshing cache, (2) setting `LD_LIBRARY_PATH` environment variable, (3) building with `-rpath`. - Recommendation: *also* install the Python package, which includes a useful command-line shell and other tools. - GNU/Linux: to link the prebuilt `libgenomicsqlite.so` distributed from our GitHub Releases, you may have to compile your source with `CXXFLAGS=-D_GLIBCXX_USE_CXX11_ABI=0`. This is because the library is built against an old libstdc++ version to improve runtime compatibility. The @@ -109,6 +104,8 @@ It's usually easiest to obtain the extension as a pre-compiled shared library (L [Dual ABI](https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html). If you build `libgenomicsqlite.so` from source, then the flag will not be needed. + Recommendation: *also* install the Python package, which includes a useful command-line shell and other tools. + See our [GitHub README](https://github.com/mlin/GenomicSQLite) for the source build procedure, if needed. We welcome community contributions to the available language bindings; see the [Language Bindings Guide](bindings.md) if interested. From 5d19f2a35885649c26fc5a635e03a134dea93ce6 Mon Sep 17 00:00:00 2001 From: Mike Lin Date: Sat, 31 Oct 2026 20:36:23 -1000 Subject: [PATCH 5/6] Fix minor wording issues in documentation --- docs/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/index.md b/docs/index.md index 8759204e..dcd7c0f3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -22,9 +22,9 @@ Our **[Colab notebook](https://colab.research.google.com/drive/1BND1nYHDg9DLsw5b ## SQLite requirements -The SQLite library linked into application must be at least version 3.31.0 (2020 vintage) and support [loadable extensions](https://www.sqlite.org/loadext.html). Check the version with `SELECT sqlite_version()` in your program. +The SQLite library linked into your application must be at least version 3.31.0 (2020 vintage) and support [loadable extensions](https://www.sqlite.org/loadext.html). Check the version with `SELECT sqlite_version()` in your program. -The **macOS** default SQLite unfortunately disables extension loading. Homebrew and conda provide compatible SQLite packages you can install and configure your build system (if applicable) to use. +The **macOS** default SQLite unfortunately disables extension loading. Homebrew and conda provide compatible SQLite packages; install one and configure your build system (if applicable) to use it. On other platforms, if SQLite is too old or lacks extension support, upgrade it with your package manager or see [How To Compile SQLite](https://www.sqlite.org/howtocompile.html). As a last resort for GNU/Linux, our [GitHub Releases](https://github.com/mlin/GenomicSQLite/releases) attach a widely-compatible `libsqlite3.so.0`. @@ -90,7 +90,7 @@ It's usually easiest to obtain the extension as a pre-compiled shared library (G Building with the crate will include a platform-appropriate shared library file within your compilation unit, to be extracted & loaded at runtime. To disable this, add `default-features = false` to the dependencies entry and at runtime, set environment variable `LIBGENOMICSQLITE` to a filename or place the library file somewhere it'll be found by `dlopen("libgenomicsqlite")`. - To make `rusqlite` use an SQLite library other than the system's default, set `SQLITE3_LIB_DIR` and `SQLITE3_INCLUDE_DIR`. + To make `rusqlite` use an SQLite library other than the system's default, set `SQLITE3_LIB_DIR` and `SQLITE3_INCLUDE_DIR` to the `bin` and `include` directories where the library is installed. Recommendation: *also* install the Python package, which includes a useful command-line shell and other tools. From 328f43f9b30f4f8fc3bc041d52a85855e3ee1a62 Mon Sep 17 00:00:00 2001 From: Mike Lin Date: Sat, 31 Oct 2026 20:38:49 -1000 Subject: [PATCH 6/6] Correct directory reference in documentation --- docs/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index dcd7c0f3..125e8dd7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -90,7 +90,7 @@ It's usually easiest to obtain the extension as a pre-compiled shared library (G Building with the crate will include a platform-appropriate shared library file within your compilation unit, to be extracted & loaded at runtime. To disable this, add `default-features = false` to the dependencies entry and at runtime, set environment variable `LIBGENOMICSQLITE` to a filename or place the library file somewhere it'll be found by `dlopen("libgenomicsqlite")`. - To make `rusqlite` use an SQLite library other than the system's default, set `SQLITE3_LIB_DIR` and `SQLITE3_INCLUDE_DIR` to the `bin` and `include` directories where the library is installed. + To make `rusqlite` use an SQLite library other than the system's default, set `SQLITE3_LIB_DIR` and `SQLITE3_INCLUDE_DIR` to the `lib` and `include` directories where the library is installed. Recommendation: *also* install the Python package, which includes a useful command-line shell and other tools.