From 6f1ecca956988477732e490c809ca749362a71c1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Xavier=20Escrib=C3=A0=20Montagut?= Date: Thu, 16 Jul 2026 09:34:36 +0200 Subject: [PATCH 1/2] Build on debian:trixie with R 4.5 / BioC 3.22 Rebuild the base image on debian:trixie instead of ubuntu:24.04. The distro swap forces R 4.3->4.5 and BioC 3.18->3.22, which is where the actual work was. Fixes needed to make the chain build: - rspm.R: point PPM at __linux__/trixie, and set HTTPUserAgent. Without the latter R sends "libcurl/x.y.z" and PPM silently serves source packages, so the distro segment has no effect. Also add dev/Rprofile creation to the docs -- rspm.R is only a template; it is never copied into the image on its own, which is why these builds have never used binaries. - functions.R: set repos to BiocManager::repositories(). rspm.R sets repos to CRAN only, which replaces the Bioconductor repos, so every BioC dependency of a Remotes entry resolved as "not available" (PCSF -> org.Hs.eg.db, topGO). - functions.R: install limSolve explicitly. DeconRNASeq was dropped from BioC after 3.18 so it is pinned to a 3.18 tarball, and remotes does not resolve dependencies of url:: remotes. - install_ubuntu.sh: add libmagick++-dev. trixie's imagemagick ships only libMagickCore/libMagickWand, not libMagick++, which the R magick package links against. Missing it cascaded into SpatialExperiment -> GSVA -> plaid. Also drop stale version pins that were already being overridden downstream (rms, rjson, rliger, BiocManager) and document the remaining ones in dev/PINS.md. url:: pins force source builds and bypass the PPM binaries, so this cut the install_playbase.R step from ~143 min to ~38 min. Verified locally through the full chain: app serves, example data loads, plots render, and quarto/typst report export works on Debian 13 / R 4.5.0 / BioC 3.22. Co-Authored-By: Claude Opus 4.8 --- DESCRIPTION | 2 - dev/Dockerfile.os | 7 +-- dev/Dockerfile.rbase | 4 +- dev/PINS.md | 122 +++++++++++++++++++++++++++++++++++++++++ dev/functions.R | 24 +++++--- dev/install_playbase.R | 4 +- dev/install_ubuntu.sh | 26 ++++----- dev/rspm.R | 16 +++++- 8 files changed, 169 insertions(+), 36 deletions(-) create mode 100644 dev/PINS.md diff --git a/DESCRIPTION b/DESCRIPTION index 60d9dc43..0a847089 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -202,8 +202,6 @@ Remotes: azimuth=url::https://github.com/satijalab/azimuth/archive/refs/heads/master.zip, RefMet=url::https://github.com/metabolomicsworkbench/RefMet/archive/refs/tags/v2.zip, grimon=github::https://github.com/mkanai/grimon, - rliger=url::https://github.com/welch-lab/liger/archive/refs/tags/v2.1.0.zip, - rms=url::https://cran.r-project.org/src/contrib/Archive/rms/rms_6.8-0.tar.gz, RaMP=url::https://github.com/ncats/RaMP-DB/archive/refs/tags/v3.0.2.zip, DeconRNASeq=url::https://bioconductor.org/packages/3.18/bioc/src/contrib/DeconRNASeq_1.44.0.tar.gz, plaid=url::https://github.com/bigomics/plaid/archive/HEAD.zip, diff --git a/dev/Dockerfile.os b/dev/Dockerfile.os index 4234e744..5212fc9b 100644 --- a/dev/Dockerfile.os +++ b/dev/Dockerfile.os @@ -3,7 +3,7 @@ ## Copyright (c) 2018-2026 BigOmics Analytics Sagl. All rights reserved. ## -FROM ubuntu:24.04 AS playbase-os +FROM debian:trixie AS playbase-os MAINTAINER BigOmics "support@bigomics.ch" @@ -26,11 +26,6 @@ WORKDIR / COPY dev/install_ubuntu.sh install_ubuntu.sh RUN sh install_ubuntu.sh -# disable c++ version warning (msa package) -RUN mv /usr/include/c++/13/bits/c++0x_warning.h \ - /usr/include/c++/13/bits/c++0x_warning.h.DISABLED && \ - touch /usr/include/c++/13/bits/c++0x_warning.h - #------------------------------------------------------------ # Clean up when done. #------------------------------------------------------------ diff --git a/dev/Dockerfile.rbase b/dev/Dockerfile.rbase index 56880976..de585051 100644 --- a/dev/Dockerfile.rbase +++ b/dev/Dockerfile.rbase @@ -22,8 +22,8 @@ COPY dev/Rprofil[e] /root/.Rprofile COPY dev/Renviro[n] /root/.Renviron RUN R -e "install.packages(c('remotes','renv','reticulate'))" -RUN R -e "remotes::install_version('BiocManager', version='1.30.23')" -RUN R -e "BiocManager::install(version='3.18')" +RUN R -e "install.packages('BiocManager')" +RUN R -e "BiocManager::install(version='3.22')" RUN echo ********* INSTALLING MINI-CONDA *************** COPY dev/install_conda.R dev/ diff --git a/dev/PINS.md b/dev/PINS.md new file mode 100644 index 00000000..8a120c9f --- /dev/null +++ b/dev/PINS.md @@ -0,0 +1,122 @@ +# Pinned versions in the Omics Playground image chain + +Every deliberate version pin across `playbase` and `omicsplayground`, why it exists, +and what has to happen before it can be dropped. + +**Rule of thumb: do not add a pin unless the package is unobtainable otherwise.** +A pin that merely says "the version we happened to test" rots silently and gets +overridden downstream (see *Removed pins* below — most of the old ones did nothing). + +## How dependency resolution actually works + +Two different code paths, easy to confuse: + +| Context | What drives installation | +|---|---| +| **Docker build** (`R/` not copied, `use.remotes=TRUE`) | `DESCRIPTION` `Imports:` + `Remotes:` via `remotes::install_deps('.')` | +| **Local dev** (`R/` present, `use.remotes=FALSE`) | `scan_packages()` in `dev/functions.R` (`remotes.url`, `add_github()`) | + +So **`DESCRIPTION: Remotes:` is the operative pin list for the images.** +The lists in `dev/functions.R` only affect local developer installs. + +`Remotes:` overrides *where* a package comes from — it applies to direct and +transitive dependencies alike, which is why a package can be pinned there +without appearing in `Imports:`. + +## Load-bearing pins — DO NOT REMOVE + +> **A `Remotes:` entry is not only a version pin — for a package that is not in +> `Imports:` it is the *only thing that installs the package at all*.** Delete it +> and the package silently disappears from the image; nothing errors at build +> time, the feature just breaks at runtime. This was confirmed the hard way on +> 2026-07-15 (see below). Always check `Imports:` **and** `grep -r 'pkg::' R/` +> before removing an entry. + +These packages no longer exist upstream. Unpinning breaks the build. + +| Package | Pin | Why | Droppable when | +|---|---|---|---| +| `KEGG.db` | `3.2.4` (BioC **3.11** tarball) | removed from Bioconductor after 3.11 | code stops using KEGG.db | +| `DeconRNASeq` | `1.44.0` (BioC **3.18** tarball) | removed from Bioconductor after 3.18 | code stops using it, or it returns to BioC | +| `grr` | `0.9.1` (CRAN Archive) | archived from CRAN | it returns to CRAN, or is vendored | + +These two *look* like stale pins but are load-bearing **dependency declarations** — +neither is in `Imports:`, so the entry is what installs them: + +| Package | Pin | Used by | Why still pinned | +|---|---|---|---| +| `infercnv` | `@infercnv-v1.3.3` | `R/pgx-cna.R` | not in `Imports:`; removing the pin removed the package entirely. Current BioC is **1.26.0** — a ~7-year API jump, so moving to it needs the CNA features actually tested, not just a green build. | +| `org.Pf.plasmo.db` | `3.14.0` (BioC 3.14) | `R/pgx-annot.R` | not in `Imports:`; same story. Present in current BioC annotation as 3.22.0. | + +The clean fix for both is to declare them in `Imports:` and drop the pin, so they +resolve from the current repos — but that is a behaviour change that needs testing +of CNA analysis and Plasmodium annotation. Do it as its own task. + +`DeconRNASeq` has a trap: `remotes` does **not** resolve dependencies of `url::` +remotes, so its dependency `limSolve` is installed explicitly in +`install_dependencies()`. If you ever repin DeconRNASeq, keep that line. + +## Deliberate forks / tags — intentional, revisit occasionally + +| Package | Pin | Why | +|---|---|---| +| `Seurat` | `satijalab/seurat@fix/v.5.3.1` | fork branch (`Dockerfile.update`) | +| `inspectdf` | `alastairrushworth/inspectdf@bugfix/cran-dplyr-failing` | upstream CRAN build broken (`Dockerfile.update`) | +| `visNetwork` | `bigomics/visNetwork` | bigomics fork (`Dockerfile.update`) | +| `RaMP` | `ncats/RaMP-DB@v3.0.2` | tag pin (`DESCRIPTION`) | +| `RefMet` | `metabolomicsworkbench/RefMet@v2` | tag pin (`DESCRIPTION`) | + +Each should be re-checked when upstream releases: if the fix landed upstream, drop the fork. + +## Platform / toolchain pins + +| Thing | Pin | Where | Notes | +|---|---|---|---| +| Bioconductor | **3.22** | `Dockerfile.rbase`, `install_playbase.R`, `functions.R` | pinned in **3 places** — change all three together. Tied to R 4.5 (trixie); BiocManager refuses 3.18 on R 4.5. | +| CRAN snapshot | `__linux__/trixie/latest` | `dev/rspm.R`, `omicsplayground/dev/Rprofile` | the `` segment **must** match the base image, or PPM serves binaries built against the wrong distro. Needs `options(HTTPUserAgent=...)` in the same file or PPM silently serves source. | +| quarto CLI | **1.8.27** | `omicsplayground/docker/Dockerfile` **and** `docker/Dockerfile.update` | keep both in sync. Need >=1.8 for typst >=0.12. tinytex installs on top of the base one, so the base block cannot simply be deleted. | +| R | 4.5.0 | implied by `debian:trixie` | not pinned explicitly; comes from the distro. | + +## Unpinned and floating — the real reproducibility gap + +These track `HEAD`/`master` with no pin, so a rebuild silently picks up whatever +upstream became. There is **no `renv.lock`** in either repo. + +`PCSF` · `playdata` · `EPIC` · `SuperCell` · `NNLM` · `azimuth` · `plaid` · +`metaLINCS` · `grimon` + +Deciding whether to pin these (or add a lockfile) is an open question — it is a +bigger risk than any stale version pin. + +## Removed pins (2026-07-15) — kept here so nobody "restores" them + +All of these were **already being overridden** later in the build; the shipped +image never contained the pinned version, so removing them changed nothing except +removing wasted work. + +Each was verified by rebuild: the package is still installed, at the version the +image was *already* shipping. All are transitive dependencies of something else in +`Imports:`, which is why removing their entry does not remove them. + +| Package | Was pinned to | Still installed as | Why removal is safe | +|---|---|---|---| +| `rms` | `6.8-0` (Archive) | **8.1-1**, as a PPM **binary** | build compiled 6.8-0 from source, then threw it away and installed 8.1-1 — pure waste | +| `rjson` | `0.2.21` (Archive) | **0.2.23** binary | pin never applied | +| `rliger` | `@v2.1.0` | **2.2.1** | in `Imports:`, resolves from CRAN | +| `BiocManager` | `1.30.23` | **1.30.27** | `install_playbase.R` runs `update.packages()`, which bumped it anyway. The pin also made the `BiocManager::valid()` build gate permanently unsatisfiable. | + +`force.remotes.url` in `dev/functions.R` held `rjson`/`rms` and force-installed them +on every build regardless of need; it is gone. + +### Why removing dead pins is worth doing + +A `url::` pin always installs from a tarball, which means a **source build** — it +bypasses the Posit binary repo entirely. Dropping the `rms` pin alone let it arrive +as a prebuilt binary. Measured on the `playbase-pkg` layer: + +| | before | after | +|---|---|---| +| `install_playbase.R` step | **8567 s** (143 min) | **2243 s** (37 min) | + +So stale pins are not merely cosmetic — they cost build time by forcing source +compiles of packages that PPM already ships as binaries. diff --git a/dev/functions.R b/dev/functions.R index 725c478e..aeb49a65 100644 --- a/dev/functions.R +++ b/dev/functions.R @@ -51,17 +51,15 @@ scan_packages <- function(path='R') { pkg.name <- gsub(".*[/]|@.*","",repo) remotes.url[pkg.name] <<- github_url(repo) } + ## See dev/PINS.md before touching these. They are not just version pins: + ## none of these packages is in DESCRIPTION Imports, so the entry here is the + ## only thing that installs them at all. Removing one removes the package. remotes.url <- c( "KEGG.db" = "url::https://bioconductor.org/packages/3.11/data/annotation/src/contrib/KEGG.db_3.2.4.tar.gz", "org.Pf.plasmo.db" = "url::https://bioconductor.org/packages/3.14/data/annotation/src/contrib/org.Pf.plasmo.db_3.14.0.tar.gz", "Azimuth" = "url::https://github.com/satijalab/azimuth/archive/refs/heads/master.zip" ) - force.remotes.url <- c( - "rjson" = "url::https://cran.r-project.org/src/contrib/Archive/rjson/rjson_0.2.21.tar.gz", - "rms" = "url::https://cran.r-project.org/src/contrib/Archive/rms/rms_6.8-0.tar.gz" - ) - ## commented out entries are now in standard CRAN/cBio repo add_github("bigomics/PCSF") add_github("bigomics/playdata") @@ -90,7 +88,6 @@ scan_packages <- function(path='R') { pkg.missing <- setdiff( c(pkg.imports,names(pkg.remotes)), pkg.installed) missing.imports <- setdiff(pkg.imports, pkg.installed) missing.remotes <- pkg.remotes[!(names(pkg.remotes) %in% pkg.installed)] - missing.remotes <- c(missing.remotes, force.remotes.url) list( used = pkg.used, @@ -120,10 +117,21 @@ install_dependencies <- function(use.remotes=FALSE) { if(!require("remotes")) install.packages('remotes') if(!require("BiocManager")) { - remotes::install_version('BiocManager', version='1.30.23') - if(!BiocManager::version()=="3.18") BiocManager::install(version='3.18') + install.packages('BiocManager') + if(!BiocManager::version()=="3.22") BiocManager::install(version='3.22') } + ## remotes resolves dependencies from getOption("repos"), and rspm.R sets + ## that to CRAN only -- so every Bioconductor dependency of a Remotes entry + ## is "not available" (PCSF -> org.Hs.eg.db, topGO). Add the BioC repos; + ## repositories() keeps the CRAN entry rspm.R configured. + options(repos = BiocManager::repositories()) + + ## DeconRNASeq was dropped from Bioconductor after 3.18, so DESCRIPTION pins + ## it to a 3.18 tarball. remotes does not resolve dependencies for url:: + ## remotes, so limSolve would never be installed and DeconRNASeq fails. + if(!require("limSolve")) install.packages("limSolve") + if(use.remotes) { # install dependencies using remotes remotes::install_deps('.', dependencies = c("Imports","Remotes")) diff --git a/dev/install_playbase.R b/dev/install_playbase.R index 36b1f1bf..f61b5c64 100644 --- a/dev/install_playbase.R +++ b/dev/install_playbase.R @@ -31,8 +31,8 @@ require <- function(pkg) (pkg %in% installed.packages()[,'Package']) if(!require("remotes")) install.packages('remotes') if(!require("BiocManager")) { - remotes::install_version('BiocManager', version='1.30.23') - if(!BiocManager::version()=="3.18") BiocManager::install(version='3.18') + install.packages('BiocManager') + if(!BiocManager::version()=="3.22") BiocManager::install(version='3.22') } missing.imports <- NULL diff --git a/dev/install_ubuntu.sh b/dev/install_ubuntu.sh index abd76bf9..7894707e 100644 --- a/dev/install_ubuntu.sh +++ b/dev/install_ubuntu.sh @@ -4,7 +4,7 @@ export LANGUAGE=en_US:en export LC_ALL=en_US.UTF-8 apt update && apt install -y \ - locales apt-utils software-properties-common \ + locales apt-utils \ libcurl4-gnutls-dev libnode-dev libv8-dev \ libssl-dev libxml2-dev libjpeg-dev \ libgl-dev libglu-dev tk-dev libhdf5-dev \ @@ -13,26 +13,26 @@ apt update && apt install -y \ jags cmake git procps htop \ python3 python3-pip python-is-python3 \ pdftk vim-tiny less wget gdebi-core \ - pandoc imagemagick libfftw3-dev libglpk-dev \ + pandoc imagemagick libmagick++-dev libfftw3-dev libglpk-dev \ libgsl-dev librsvg2-dev libgsl-dev curl \ libsodium-dev libnlopt-dev \ libharfbuzz-dev libfribidi-dev \ fonts-lato -# remove ugly snaps -#snap remove --purge -y firefox -#apt purge snapd - -# Install Chrome +# Install Chrome (apt-key no longer exists on trixie, so use a keyring) apt-get update && apt-get install -y wget gnupg2 -wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - -echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list +install -d -m 0755 /etc/apt/keyrings +wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub \ + | gpg --dearmor -o /etc/apt/keyrings/google-chrome.gpg +echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/google-chrome.gpg] http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list apt-get update && apt-get install -y google-chrome-stable -# disable c++ version warning (msa package) -mv /usr/include/c++/13/bits/c++0x_warning.h \ - /usr/include/c++/13/bits/c++0x_warning.h.DISABLED && \ - touch /usr/include/c++/13/bits/c++0x_warning.h +# disable c++ version warning (msa package). Glob the gcc version rather +# than hardcoding it: noble had gcc-13, trixie has gcc-14. +for h in /usr/include/c++/*/bits/c++0x_warning.h; do + [ -f "$h" ] || continue + mv "$h" "$h.DISABLED" && : > "$h" +done ## basic R apt install -y r-base r-base-dev diff --git a/dev/rspm.R b/dev/rspm.R index 85b11838..95d63330 100644 --- a/dev/rspm.R +++ b/dev/rspm.R @@ -6,6 +6,16 @@ options(timeout = 99999) ## download time.out options(BioC_mirror = "https://packagemanager.posit.co/bioconductor") options(BIOCONDUCTOR_CONFIG_FILE = "https://packagemanager.posit.co/bioconductor/config.yaml") -# Configure a CRAN snapshot compatible with Bioconductor 3.18: -##options(repos = c(CRAN = "https://packagemanager.posit.co/cran/__linux__/noble/2024-05-01")) -options(repos = c(CRAN = "https://packagemanager.posit.co/cran/__linux__/noble/latest")) +# Configure a CRAN snapshot compatible with Bioconductor 3.22. +# NOTE: the __linux__/ segment must match the base image, otherwise +# PPM silently serves source packages instead of binaries. +options(repos = c(CRAN = "https://packagemanager.posit.co/cran/__linux__/trixie/latest")) + +# PPM picks the binary build from the User-Agent. R does not send its own +# identity unless HTTPUserAgent is set explicitly -- it sends "libcurl/x.y.z", +# which PPM answers with source packages. Without this line the __linux__/ +# trixie URL above has no effect. +options(HTTPUserAgent = sprintf( + "R/%s R (%s)", getRversion(), + paste(getRversion(), R.version["platform"], R.version["arch"], R.version["os"]) +)) From a2e430449591d9c76313daeb7c5585530ff4ab06 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Xavier=20Escrib=C3=A0=20Montagut?= Date: Fri, 14 Aug 2026 10:12:14 +0200 Subject: [PATCH 2/2] Pin the PPM snapshot instead of tracking `latest` PPM's binary channel served lattice 0.23-1, which dropped the `parallel` export that methylumi 2.56.0 still imports. That cascades methylumi -> lumi -> wateRmelon, and wateRmelon is a playbase Import, so `install_playbase.R` then fails outright: playbase is removed and scan_description() dies reading the DESCRIPTION of the package that was just deleted. Note the source and binary channels disagreed -- src/contrib/PACKAGES advertised 0.22-9 while the binary channel shipped 0.23-1 -- so checking the source index does not reveal this. PPM later withdrew 0.23-1, meaning `latest` also flip-flops; a dated snapshot is the only stable choice for a migration we are trying to validate. 2026-07-15 is the validated date and still serves binaries. Must stay in step with omicsplayground's dev/Rprofile. Pinning only one of the two is worse than pinning neither: the base builds correctly against 0.22-9 and the opg stages then upgrade lattice over it, leaving methylumi, lumi and wateRmelon installed but unloadable, with the build exiting 0. Co-Authored-By: Claude Opus 5 --- dev/rspm.R | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/dev/rspm.R b/dev/rspm.R index 95d63330..1f5b75eb 100644 --- a/dev/rspm.R +++ b/dev/rspm.R @@ -9,7 +9,15 @@ options(BIOCONDUCTOR_CONFIG_FILE = "https://packagemanager.posit.co/bioconductor # Configure a CRAN snapshot compatible with Bioconductor 3.22. # NOTE: the __linux__/ segment must match the base image, otherwise # PPM silently serves source packages instead of binaries. -options(repos = c(CRAN = "https://packagemanager.posit.co/cran/__linux__/trixie/latest")) +# +# Pinned to a date, NOT `latest`. On `latest` the binary channel served lattice +# 0.23-1, which dropped the `parallel` export that methylumi 2.56.0 still +# imports; that cascades methylumi -> lumi -> wateRmelon, and wateRmelon is a +# playbase Import, so playbase itself then fails to install. This snapshot +# still serves binaries, so the PPM speedup is retained. +# Keep this date in step with omicsplayground's dev/Rprofile -- if the two +# disagree, the opg build stages upgrade packages this base was compiled against. +options(repos = c(CRAN = "https://packagemanager.posit.co/cran/__linux__/trixie/2026-07-15")) # PPM picks the binary build from the User-Agent. R does not send its own # identity unless HTTPUserAgent is set explicitly -- it sends "libcurl/x.y.z",