From a3335601c112afa18b84bacd5ce9112fa4c79107 Mon Sep 17 00:00:00 2001 From: Sophia Wen Date: Thu, 27 Aug 2026 17:26:44 -0400 Subject: [PATCH] setup: build CPU + Thrust + OMP-offload from one install when NVHPC is present Previously auto built only CPU + Thrust, and the OMP-offload backend had to be installed alone in a separate venv on the assumption it conflicted on the OpenMP runtime. On a GPU box all three extensions compile with nvc++ and link the same runtime (libnvomp), so they co-load in one process (available_backends() -> ['cpu', 'gpu', 'gpu-omp'], all agree bit-for-bit). setup.py: - auto: build CPU + Thrust + OMP-offload when NVHPC is present; CPU only (g++/clang) otherwise. - add SBD_BUILD_BACKEND=all (builds all three; errors without NVHPC). - when CPU is built alongside any GPU backend, route the whole build through nvc++ up front so the CPU extension also links libnvomp. - macOS (clang) and Linux-without-NVHPC (g++) paths unchanged. README: rewrite the install section for the single-venv unified build and correct the backend co-load description. --- README.md | 120 +++++++++++++++++++++--------------------------------- setup.py | 56 +++++++++++++++++++------ 2 files changed, 91 insertions(+), 85 deletions(-) diff --git a/README.md b/README.md index afaabeb..d01e3f6 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # SBD Python Bindings -Python bindings for the Selected Basis Diagonalization (SBD) library with dual CPU/GPU backend support. +Python bindings for the Selected Basis Diagonalization (SBD) library with CPU and GPU backend support. ## Overview @@ -8,7 +8,7 @@ SBD (Selected Basis Diagonalization) is a high-performance library for quantum c **Key Features:** - **TPB diagonalization** for quantum chemistry Hamiltonians -- Dual backend: CPU (OpenMP) and GPU (CUDA), switchable at runtime +- Three backends, switchable at runtime: CPU (OpenMP), GPU Thrust (CUDA), and GPU OpenMP target offload - MPI parallelization - Integration with [qiskit-addon-sqd](https://github.com/Qiskit/qiskit-addon-sqd) for SQD workflows @@ -95,33 +95,30 @@ export SBD_GPU_ARCH=cc100 You do **not** need to set `CC=nvc` / `CXX=nvc++` or clear `CFLAGS` / `CXXFLAGS` manually for the GPU builds — setup.py auto-routes those extensions through nvc++ and filters the RHEL 9 sysconfig flags that -nvc++ rejects. (Earlier versions required this; the v1.6 refactor -moved it into `setup.py:_route_build_through_nvhpc()`.) If you DO -set them, your values win — distutils' `os.environ.setdefault` -semantics. - -The OpenMP target-offload backend cannot share a Python process with -CPU or Thrust. They link incompatible OpenMP runtimes (NVHPC's -`libnvomp` for OMP-offload, `libgomp`/`libomp` for CPU and Thrust), -and `import sbd` eagerly loads every `_core_*.so` it finds in -`python/`. Co-resident `.so` files therefore pull both runtimes into -the same address space, producing "Another OpenMP runtime library has -been detected" and potentially deadlocking at the first `#pragma omp` -region. **The cleanest setup is two venvs with two checkouts** — one -for CPU + Thrust, one for OMP-offload. Within a single venv you can -also switch profiles by removing the other profile's `_core_*.so` -before rebuilding (only files present in `python/` get loaded), but a -second venv avoids the bookkeeping. +nvc++ rejects. If you DO set them, your values win — distutils' +`os.environ.setdefault` semantics. + +All three GPU-box backends co-exist in **one venv, one process**. On a +GPU box the CPU, Thrust, and OMP-offload extensions all compile with +NVHPC `nvc++` and link the same OpenMP runtime (`libnvomp`): CPU via +`-fopenmp` (nvc++ maps it to `-mp`), Thrust via `-mp -cuda`, OMP-offload +via `-mp=gpu`. `import sbd` eagerly loads every `_core_*.so` in +`python/`; a single process reports `['cpu', 'gpu', 'gpu-omp']` and you +select one per call with `device=`. ### Build -Pick **one** of two installation profiles. They produce mutually -incompatible Python processes (different OpenMP runtimes — see the -paragraph above the "Build" section) so put them in **separate venvs -backed by separate checkouts** if you need both. The `source …` -lines below are not optional: forgetting to activate the right venv -before `pip install -e .` either puts the build into the wrong venv -or fails outright. +One venv, one `pip install`, and the default (`SBD_BUILD_BACKEND` +unset, i.e. `auto`) builds **every backend the toolchain supports**: + +- **Linux with a GPU + NVHPC `nvc++`** → CPU + Thrust + OMP-offload, all + through nvc++ so they share `libnvomp` and co-load. Devices: `'cpu'`, + `'gpu'` (Thrust), `'gpu-omp'` (target offload). +- **macOS**, or **Linux without NVHPC** → CPU only, built with the + native `clang`/`g++`. Device: `'cpu'`. + +No separate venvs or checkouts are needed — a single install exposes +all three and you switch per call (`device=...`) or per run. > **Note — `setuptools` in the setup line:** because the build uses > `--no-build-isolation`, it relies on the build tools already present @@ -130,61 +127,40 @@ or fails outright. > the `pip install … setuptools wheel` line below). Without it the > build fails with `Cannot import 'setuptools.build_meta'`. -**Profile 1 — CPU + Thrust GPU** (the common case) - ```bash # one-time setup -git clone --recurse-submodules https://github.com/Qiskit/sbd-eigensolver-python.git sbd-thrust -python -m venv ~/venvs/sbd-thrust -source ~/venvs/sbd-thrust/bin/activate -MPICC=$(which mpicc) pip install --no-binary=mpi4py mpi4py pybind11 numpy setuptools wheel - -# build (re-run after pulling main) -source ~/venvs/sbd-thrust/bin/activate # always activate first -cd sbd-thrust -SBD_GPU_ARCH=cc100 pip install -e . --no-build-isolation -``` - -Builds the CPU backend always. Adds the Thrust GPU backend when NVHPC -`nvc++` is on PATH; otherwise CPU-only. Devices: `'cpu'` and `'gpu'`. - -**Profile 2 — OpenMP target-offload GPU** (use a **separate** venv + -checkout from Profile 1) - -```bash -# one-time setup -git clone --recurse-submodules https://github.com/Qiskit/sbd-eigensolver-python.git sbd-omp-offload -python -m venv ~/venvs/sbd-omp-offload -source ~/venvs/sbd-omp-offload/bin/activate +git clone --recurse-submodules https://github.com/Qiskit/sbd-eigensolver-python.git +python -m venv ~/venvs/sbd +source ~/venvs/sbd/bin/activate MPICC=$(which mpicc) pip install --no-binary=mpi4py mpi4py pybind11 numpy setuptools wheel # build (re-run after pulling main) -source ~/venvs/sbd-omp-offload/bin/activate # always activate first -cd sbd-omp-offload -SBD_BUILD_BACKEND=gpu_omp_offload SBD_GPU_ARCH=cc100 \ - pip install -e . --no-build-isolation +source ~/venvs/sbd/bin/activate # always activate first +cd sbd-eigensolver-python +SBD_GPU_ARCH=cc90 pip install -e . --no-build-isolation ``` -Builds the OMP-offload GPU backend only. Device: `'gpu-omp'`. - -After both profiles are installed, switching is a one-liner -(`source ~/venvs//bin/activate`) — no rebuild needed. +On a GPU box this produces `_core_cpu`, `_core_gpu_thrust`, and +`_core_gpu_omp_offload` side by side in `python/`; on a CPU-only host it +produces just `_core_cpu`. **Multi-arch fat binary**: comma-separate the arches: `SBD_GPU_ARCH=cc80,cc90,cc100`. nvc++ embeds one SASS cubin per arch and picks the matching one at runtime. -#### Advanced `SBD_BUILD_BACKEND` overrides +#### `SBD_BUILD_BACKEND` overrides -Only needed when you want to deviate from the two profiles above. +Set this only to build a **subset** of the default. Any value that +requests a GPU backend routes the whole build through nvc++. | Value | Builds | |---|---| -| *unset* (default) | CPU always; Thrust GPU if `nvc++` found. The "Profile 1" default. | -| `cpu` | CPU only — skip GPU even if `nvc++` is present. | -| `gpu` | Thrust GPU only — skip CPU. Errors if `nvc++` missing. | +| *unset* / `auto` (default) | All backends `nvc++` can build (CPU + Thrust + OMP-offload) when NVHPC is present; CPU only (gcc/clang) otherwise. | +| `all` | CPU + Thrust + OMP-offload. Errors if `nvc++` is missing (no CPU-only fallback). | +| `cpu` | CPU only — skip GPU even if `nvc++` is present. Uses the native compiler. | +| `gpu` (alias `gpu_thrust`) | Thrust GPU only — skip CPU. Errors if `nvc++` missing. | | `both` | CPU + Thrust GPU. Errors instead of falling back if `nvc++` missing. | -| `gpu_omp_offload` | OMP-offload GPU only. The "Profile 2" install. | +| `gpu_omp_offload` | OMP-offload GPU only. | **Reverting to the LLVM/clang offload path:** prior versions of this repo supported a separate `_core_gpu_omp_nvidia` backend built with @@ -198,9 +174,9 @@ recipe there if you need the clang path back. ```bash python -c "import sbd; print(sbd.available_backends())" -# CPU only: ['cpu'] -# CPU + NVHPC Thrust: ['cpu', 'gpu'] -# OMP-offload-only install: ['gpu-omp'] +# CPU-only host (no NVHPC): ['cpu'] +# GPU box, default build: ['cpu', 'gpu', 'gpu-omp'] +# subset builds report only what they built, e.g. ['cpu', 'gpu'] or ['gpu-omp'] ``` ## Usage @@ -229,18 +205,16 @@ sbd.finalize() ### Runtime backend switching -Compatible backends coexist as separate `_core_*.so` modules and load -at `import sbd` into independent pybind11 namespaces. CPU + Thrust GPU -can co-load; the OMP-offload backend cannot (different OpenMP runtime — -see the build section). Pick one per call with the `device` parameter: +All compiled backends coexist as separate `_core_*.so` modules and load +at `import sbd` into independent pybind11 namespaces. Pick one per call +with the `device` parameter: ```python import sbd # All compiled backends are auto-loaded sbd.available_backends() -# CPU + Thrust install: ['cpu', 'gpu'] -# OMP-offload-only install: ['gpu-omp'] +# GPU box: ['cpu', 'gpu', 'gpu-omp'] # Per-call override — auto-initializes on first use result_cpu = sbd.tpb_diag(..., device='cpu') diff --git a/setup.py b/setup.py index 52840ff..be1e1fd 100644 --- a/setup.py +++ b/setup.py @@ -259,18 +259,29 @@ def find_nvidia_hpc_sdk(): gpu_compiler, has_nvhpc = find_nvidia_hpc_sdk() # Determine which backends to build. -# auto : cpu + thrust GPU (if nvc++ present) +# auto : all backends nvc++ can build (cpu + thrust + +# omp-offload) when NVHPC is present; cpu only otherwise # cpu : cpu only # gpu | gpu_thrust : thrust GPU only # both : cpu + thrust +# all : cpu + thrust + omp-offload (requires NVHPC) # gpu_omp_offload : OpenMP target offload only (nvc++ -mp=gpu) # -# gpu_omp_offload is built ALONE — it uses a different OpenMP runtime -# (libnvomp) than cpu (libgomp/libomp) and Thrust GPU (CPU OMP via -mp), -# and loading two backends with different OMP runtimes in one Python -# process produces "Another OpenMP runtime library has been detected" -# warnings and can deadlock at first OMP region. Build it into its own -# venv / install dir. +# Co-loading all three backends in ONE Python process is SAFE, because since +# v1.6 all three compile with nvc++ and therefore link the SAME OpenMP runtime +# (NVHPC's libnvomp): cpu via -fopenmp (nvc++ treats it as -mp), Thrust via +# -mp -cuda, and offload via -mp=gpu. `import sbd` eagerly loads every +# _core_*.so beside it and they coexist without conflict (verified: cpu, omp5, +# thrust all agree bit-for-bit in a single process on NVHPC 26.3). +# +# The historical "Another OpenMP runtime library has been detected" deadlock +# came from the pre-v1.6 LLVM/clang offload backend (libomp), or from a CPU +# backend compiled with a NON-nvc++ toolchain (gcc -> libgomp) being mixed with +# an nvc++ offload backend (libnvomp) in separate installs. To keep the +# co-loadable guarantee, whenever the CPU backend is built ALONGSIDE a GPU +# backend it is routed through nvc++ (see the _route_build_through_nvhpc call +# below), so it links libnvomp too. A standalone `cpu` build may use gcc +# (libgomp); that is fine because nothing else is loaded beside it. build_backend = os.environ.get('SBD_BUILD_BACKEND', 'auto').lower() build_cpu = False @@ -280,8 +291,10 @@ def find_nvidia_hpc_sdk(): if build_backend == 'auto': build_cpu = True build_gpu_thrust = has_nvhpc - if build_gpu_thrust: - print("\nAuto-detected nvc++ - will build both CPU and Thrust GPU backends") + build_gpu_omp_offload = has_nvhpc + if has_nvhpc: + print("\nAuto-detected nvc++ - will build ALL backends " + "(CPU + Thrust + OMP-offload)") else: print("\nnvc++ not found - will build CPU backend only") elif build_backend == 'cpu': @@ -298,9 +311,19 @@ def find_nvidia_hpc_sdk(): print("\nBuilding both CPU and Thrust GPU backends (SBD_BUILD_BACKEND=both)") if not has_nvhpc: print("Warning: nvc++ not found, GPU build may fail") +elif build_backend == 'all': + build_cpu = True + build_gpu_thrust = True + build_gpu_omp_offload = True + print("\nBuilding ALL backends: CPU + Thrust + OMP-offload " + "(SBD_BUILD_BACKEND=all)") + if not has_nvhpc: + print("Error: SBD_BUILD_BACKEND=all requires NVHPC_HOME / nvc++.") + sys.exit(1) elif build_backend == 'gpu_omp_offload': - # Stand-alone build: this mode only emits _core_gpu_omp_offload.so. - # See note above on the OpenMP-runtime exclusivity constraint. + # Standalone offload build. Since v1.6 this links libnvomp (nvc++), the + # same runtime as cpu/thrust, so co-installing all three is supported via + # SBD_BUILD_BACKEND=all; this mode remains for building offload alone. build_gpu_omp_offload = True print("\nBuilding GPU OpenMP target-offload backend only " "(SBD_BUILD_BACKEND=gpu_omp_offload)") @@ -309,9 +332,18 @@ def find_nvidia_hpc_sdk(): sys.exit(1) else: print(f"Error: Invalid SBD_BUILD_BACKEND='{build_backend}'") - print("Valid values: auto, cpu, gpu (alias gpu_thrust), both, gpu_omp_offload") + print("Valid values: auto, cpu, gpu (alias gpu_thrust), both, all, " + "gpu_omp_offload") sys.exit(1) +# Co-load safety: if the CPU backend is built together with any GPU backend, +# route the whole build through nvc++ up front so the CPU extension links +# libnvomp (matching the GPU backends) instead of libgomp. This is +# idempotent with the per-extension calls in the GPU blocks below, and being +# early it is independent of extension ordering. +if build_cpu and (build_gpu_thrust or build_gpu_omp_offload) and gpu_compiler: + _route_build_through_nvhpc(gpu_compiler) + ext_modules = [] if build_cpu: