Skip to content

Build CPU + Thrust + OMP-offload from a single install when NVHPC is present #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hfwen0502 wants to merge 1 commit into from
Closed

Build CPU + Thrust + OMP-offload from a single install when NVHPC is present #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 47 additions & 73 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# SBD Python Bindings

Python bindings for the Selected Basis Diagonalization (SBD) library with dual CPU/GPU backend support.
Python bindings for the Selected Basis Diagonalization (SBD) library with CPU and GPU backend support.

## Overview

SBD (Selected Basis Diagonalization) is a high-performance library for quantum chemistry calculations. The Python bindings provide access to SBD's **Tensor-Product Basis (TPB)** diagonalization method with support for both CPU and GPU backends.

**Key Features:**
- **TPB diagonalization** for quantum chemistry Hamiltonians
- Dual backend: CPU (OpenMP) and GPU (CUDA), switchable at runtime
- Three backends, switchable at runtime: CPU (OpenMP), GPU Thrust (CUDA), and GPU OpenMP target offload
- MPI parallelization
- Integration with [qiskit-addon-sqd](https://github.com/Qiskit/qiskit-addon-sqd) for SQD workflows

Expand Down Expand Up @@ -95,33 +95,30 @@ export SBD_GPU_ARCH=cc100
You do **not** need to set `CC=nvc` / `CXX=nvc++` or clear `CFLAGS` /
`CXXFLAGS` manually for the GPU builds — setup.py auto-routes those
extensions through nvc++ and filters the RHEL 9 sysconfig flags that
nvc++ rejects. (Earlier versions required this; the v1.6 refactor
moved it into `setup.py:_route_build_through_nvhpc()`.) If you DO
set them, your values win — distutils' `os.environ.setdefault`
semantics.

The OpenMP target-offload backend cannot share a Python process with
CPU or Thrust. They link incompatible OpenMP runtimes (NVHPC's
`libnvomp` for OMP-offload, `libgomp`/`libomp` for CPU and Thrust),
and `import sbd` eagerly loads every `_core_*.so` it finds in
`python/`. Co-resident `.so` files therefore pull both runtimes into
the same address space, producing "Another OpenMP runtime library has
been detected" and potentially deadlocking at the first `#pragma omp`
region. **The cleanest setup is two venvs with two checkouts** — one
for CPU + Thrust, one for OMP-offload. Within a single venv you can
also switch profiles by removing the other profile's `_core_*.so`
before rebuilding (only files present in `python/` get loaded), but a
second venv avoids the bookkeeping.
nvc++ rejects. If you DO set them, your values win — distutils'
`os.environ.setdefault` semantics.

All three GPU-box backends co-exist in **one venv, one process**. On a
GPU box the CPU, Thrust, and OMP-offload extensions all compile with
NVHPC `nvc++` and link the same OpenMP runtime (`libnvomp`): CPU via
`-fopenmp` (nvc++ maps it to `-mp`), Thrust via `-mp -cuda`, OMP-offload
via `-mp=gpu`. `import sbd` eagerly loads every `_core_*.so` in
`python/`; a single process reports `['cpu', 'gpu', 'gpu-omp']` and you
select one per call with `device=`.

### Build

Pick **one** of two installation profiles. They produce mutually
incompatible Python processes (different OpenMP runtimes — see the
paragraph above the "Build" section) so put them in **separate venvs
backed by separate checkouts** if you need both. The `source …`
lines below are not optional: forgetting to activate the right venv
before `pip install -e .` either puts the build into the wrong venv
or fails outright.
One venv, one `pip install`, and the default (`SBD_BUILD_BACKEND`
unset, i.e. `auto`) builds **every backend the toolchain supports**:

- **Linux with a GPU + NVHPC `nvc++`** → CPU + Thrust + OMP-offload, all
through nvc++ so they share `libnvomp` and co-load. Devices: `'cpu'`,
`'gpu'` (Thrust), `'gpu-omp'` (target offload).
- **macOS**, or **Linux without NVHPC** → CPU only, built with the
native `clang`/`g++`. Device: `'cpu'`.

No separate venvs or checkouts are needed — a single install exposes
all three and you switch per call (`device=...`) or per run.

> **Note — `setuptools` in the setup line:** because the build uses
> `--no-build-isolation`, it relies on the build tools already present
Expand All @@ -130,61 +127,40 @@ or fails outright.
> the `pip install … setuptools wheel` line below). Without it the
> build fails with `Cannot import 'setuptools.build_meta'`.

**Profile 1 — CPU + Thrust GPU** (the common case)

```bash
# one-time setup
git clone --recurse-submodules https://github.com/Qiskit/sbd-eigensolver-python.git sbd-thrust
python -m venv ~/venvs/sbd-thrust
source ~/venvs/sbd-thrust/bin/activate
MPICC=$(which mpicc) pip install --no-binary=mpi4py mpi4py pybind11 numpy setuptools wheel

# build (re-run after pulling main)
source ~/venvs/sbd-thrust/bin/activate # always activate first
cd sbd-thrust
SBD_GPU_ARCH=cc100 pip install -e . --no-build-isolation
```

Builds the CPU backend always. Adds the Thrust GPU backend when NVHPC
`nvc++` is on PATH; otherwise CPU-only. Devices: `'cpu'` and `'gpu'`.

**Profile 2 — OpenMP target-offload GPU** (use a **separate** venv +
checkout from Profile 1)

```bash
# one-time setup
git clone --recurse-submodules https://github.com/Qiskit/sbd-eigensolver-python.git sbd-omp-offload
python -m venv ~/venvs/sbd-omp-offload
source ~/venvs/sbd-omp-offload/bin/activate
git clone --recurse-submodules https://github.com/Qiskit/sbd-eigensolver-python.git
python -m venv ~/venvs/sbd
source ~/venvs/sbd/bin/activate
MPICC=$(which mpicc) pip install --no-binary=mpi4py mpi4py pybind11 numpy setuptools wheel

# build (re-run after pulling main)
source ~/venvs/sbd-omp-offload/bin/activate # always activate first
cd sbd-omp-offload
SBD_BUILD_BACKEND=gpu_omp_offload SBD_GPU_ARCH=cc100 \
pip install -e . --no-build-isolation
source ~/venvs/sbd/bin/activate # always activate first
cd sbd-eigensolver-python
SBD_GPU_ARCH=cc90 pip install -e . --no-build-isolation
```

Builds the OMP-offload GPU backend only. Device: `'gpu-omp'`.

After both profiles are installed, switching is a one-liner
(`source ~/venvs/<which>/bin/activate`) — no rebuild needed.
On a GPU box this produces `_core_cpu`, `_core_gpu_thrust`, and
`_core_gpu_omp_offload` side by side in `python/`; on a CPU-only host it
produces just `_core_cpu`.

**Multi-arch fat binary**: comma-separate the arches:
`SBD_GPU_ARCH=cc80,cc90,cc100`. nvc++ embeds one SASS cubin per arch
and picks the matching one at runtime.

#### Advanced `SBD_BUILD_BACKEND` overrides
#### `SBD_BUILD_BACKEND` overrides

Only needed when you want to deviate from the two profiles above.
Set this only to build a **subset** of the default. Any value that
requests a GPU backend routes the whole build through nvc++.

| Value | Builds |
|---|---|
| *unset* (default) | CPU always; Thrust GPU if `nvc++` found. The "Profile 1" default. |
| `cpu` | CPU only — skip GPU even if `nvc++` is present. |
| `gpu` | Thrust GPU only — skip CPU. Errors if `nvc++` missing. |
| *unset* / `auto` (default) | All backends `nvc++` can build (CPU + Thrust + OMP-offload) when NVHPC is present; CPU only (gcc/clang) otherwise. |
| `all` | CPU + Thrust + OMP-offload. Errors if `nvc++` is missing (no CPU-only fallback). |
| `cpu` | CPU only — skip GPU even if `nvc++` is present. Uses the native compiler. |
| `gpu` (alias `gpu_thrust`) | Thrust GPU only — skip CPU. Errors if `nvc++` missing. |
| `both` | CPU + Thrust GPU. Errors instead of falling back if `nvc++` missing. |
| `gpu_omp_offload` | OMP-offload GPU only. The "Profile 2" install. |
| `gpu_omp_offload` | OMP-offload GPU only. |

**Reverting to the LLVM/clang offload path:** prior versions of this
repo supported a separate `_core_gpu_omp_nvidia` backend built with
Expand All @@ -198,9 +174,9 @@ recipe there if you need the clang path back.

```bash
python -c "import sbd; print(sbd.available_backends())"
# CPU only: ['cpu']
# CPU + NVHPC Thrust: ['cpu', 'gpu']
# OMP-offload-only install: ['gpu-omp']
# CPU-only host (no NVHPC): ['cpu']
# GPU box, default build: ['cpu', 'gpu', 'gpu-omp']
# subset builds report only what they built, e.g. ['cpu', 'gpu'] or ['gpu-omp']
```

## Usage
Expand Down Expand Up @@ -229,18 +205,16 @@ sbd.finalize()

### Runtime backend switching

Compatible backends coexist as separate `_core_*.so` modules and load
at `import sbd` into independent pybind11 namespaces. CPU + Thrust GPU
can co-load; the OMP-offload backend cannot (different OpenMP runtime —
see the build section). Pick one per call with the `device` parameter:
All compiled backends coexist as separate `_core_*.so` modules and load
at `import sbd` into independent pybind11 namespaces. Pick one per call
with the `device` parameter:

```python
import sbd

# All compiled backends are auto-loaded
sbd.available_backends()
# CPU + Thrust install: ['cpu', 'gpu']
# OMP-offload-only install: ['gpu-omp']
# GPU box: ['cpu', 'gpu', 'gpu-omp']

# Per-call override — auto-initializes on first use
result_cpu = sbd.tpb_diag(..., device='cpu')
Expand Down
56 changes: 44 additions & 12 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,18 +259,29 @@ def find_nvidia_hpc_sdk():
gpu_compiler, has_nvhpc = find_nvidia_hpc_sdk()

# Determine which backends to build.
# auto : cpu + thrust GPU (if nvc++ present)
# auto : all backends nvc++ can build (cpu + thrust +
# omp-offload) when NVHPC is present; cpu only otherwise
# cpu : cpu only
# gpu | gpu_thrust : thrust GPU only
# both : cpu + thrust
# all : cpu + thrust + omp-offload (requires NVHPC)
# gpu_omp_offload : OpenMP target offload only (nvc++ -mp=gpu)
#
# gpu_omp_offload is built ALONE — it uses a different OpenMP runtime
# (libnvomp) than cpu (libgomp/libomp) and Thrust GPU (CPU OMP via -mp),
# and loading two backends with different OMP runtimes in one Python
# process produces "Another OpenMP runtime library has been detected"
# warnings and can deadlock at first OMP region. Build it into its own
# venv / install dir.
# Co-loading all three backends in ONE Python process is SAFE, because since
# v1.6 all three compile with nvc++ and therefore link the SAME OpenMP runtime
# (NVHPC's libnvomp): cpu via -fopenmp (nvc++ treats it as -mp), Thrust via
# -mp -cuda, and offload via -mp=gpu. `import sbd` eagerly loads every
# _core_*.so beside it and they coexist without conflict (verified: cpu, omp5,
# thrust all agree bit-for-bit in a single process on NVHPC 26.3).
#
# The historical "Another OpenMP runtime library has been detected" deadlock
# came from the pre-v1.6 LLVM/clang offload backend (libomp), or from a CPU
# backend compiled with a NON-nvc++ toolchain (gcc -> libgomp) being mixed with
# an nvc++ offload backend (libnvomp) in separate installs. To keep the
# co-loadable guarantee, whenever the CPU backend is built ALONGSIDE a GPU
# backend it is routed through nvc++ (see the _route_build_through_nvhpc call
# below), so it links libnvomp too. A standalone `cpu` build may use gcc
# (libgomp); that is fine because nothing else is loaded beside it.
build_backend = os.environ.get('SBD_BUILD_BACKEND', 'auto').lower()

build_cpu = False
Expand All @@ -280,8 +291,10 @@ def find_nvidia_hpc_sdk():
if build_backend == 'auto':
build_cpu = True
build_gpu_thrust = has_nvhpc
if build_gpu_thrust:
print("\nAuto-detected nvc++ - will build both CPU and Thrust GPU backends")
build_gpu_omp_offload = has_nvhpc
if has_nvhpc:
print("\nAuto-detected nvc++ - will build ALL backends "
"(CPU + Thrust + OMP-offload)")
else:
print("\nnvc++ not found - will build CPU backend only")
elif build_backend == 'cpu':
Expand All @@ -298,9 +311,19 @@ def find_nvidia_hpc_sdk():
print("\nBuilding both CPU and Thrust GPU backends (SBD_BUILD_BACKEND=both)")
if not has_nvhpc:
print("Warning: nvc++ not found, GPU build may fail")
elif build_backend == 'all':
build_cpu = True
build_gpu_thrust = True
build_gpu_omp_offload = True
print("\nBuilding ALL backends: CPU + Thrust + OMP-offload "
"(SBD_BUILD_BACKEND=all)")
if not has_nvhpc:
print("Error: SBD_BUILD_BACKEND=all requires NVHPC_HOME / nvc++.")
sys.exit(1)
elif build_backend == 'gpu_omp_offload':
# Stand-alone build: this mode only emits _core_gpu_omp_offload.so.
# See note above on the OpenMP-runtime exclusivity constraint.
# Standalone offload build. Since v1.6 this links libnvomp (nvc++), the
# same runtime as cpu/thrust, so co-installing all three is supported via
# SBD_BUILD_BACKEND=all; this mode remains for building offload alone.
build_gpu_omp_offload = True
print("\nBuilding GPU OpenMP target-offload backend only "
"(SBD_BUILD_BACKEND=gpu_omp_offload)")
Expand All @@ -309,9 +332,18 @@ def find_nvidia_hpc_sdk():
sys.exit(1)
else:
print(f"Error: Invalid SBD_BUILD_BACKEND='{build_backend}'")
print("Valid values: auto, cpu, gpu (alias gpu_thrust), both, gpu_omp_offload")
print("Valid values: auto, cpu, gpu (alias gpu_thrust), both, all, "
"gpu_omp_offload")
sys.exit(1)

# Co-load safety: if the CPU backend is built together with any GPU backend,
# route the whole build through nvc++ up front so the CPU extension links
# libnvomp (matching the GPU backends) instead of libgomp. This is
# idempotent with the per-extension calls in the GPU blocks below, and being
# early it is independent of extension ordering.
if build_cpu and (build_gpu_thrust or build_gpu_omp_offload) and gpu_compiler:
_route_build_through_nvhpc(gpu_compiler)

ext_modules = []

if build_cpu:
Expand Down