From d3e91070c233f1fd023c665879871538f03c6b13 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Sun, 9 Aug 2026 15:43:40 -0700 Subject: [PATCH 1/4] container family: multi-architecture container deployment (port from rift_O4d) Point SINGULARITY_RIFT_IMAGE at a .yaml/.yml *manifest* describing several images that target different GPU compute capabilities, and let HTCondor pick the right one per matched machine. A plain .sif path or single osdf:// URL keeps the exact legacy single-image behavior -- a manifest is recognized purely by its file extension, and pyyaml is only needed when one is used. Ported from origin/rift_O4d, where the wiring lives in the (master-absent) dag_utils_generic.py rewrite; here it is applied to dag_utils.py directly. * RIFT/misc/container_manifest.py -- manifest parse/validate plus the expression builders: capability ifThenElse image selection, comma-free $$() selective-transfer token, require_gpus capability floor, container-universe container_image selector, capability-defined Requirements clause, CPU-safe single-fallback resolution, and the runtime-selection wrapper script. * dag_utils.write_ILE_sub_simple / write_CIP_sub -- emit the above. GPU ILE gets the per-machine selection, the floor composed (not replaced) with RIFT_REQUIRE_GPUS, and TARGET. =!= undefined so it never matches a slot where the selection cannot expand. CPU-only CIP collapses to the single CPU-safe fallback image (a $$() would hold it). * Two opt-in OSG delivery modes, since OSPool pilots read SingularityImage as a literal string: RIFT_CONTAINER_UNIVERSE=1 (universe=container + container_image = $$([...]), schedd-side match-time substitution -- recommended) and RIFT_CONTAINER_RUNTIME_SELECT=1 (ILE-only wrapper that detects the real GPU and fetches just that image). * create_event_parameter_pipeline_BasicIteration -- enable the osdf transfer credential by inspecting the manifest's image URLs; the single-image checks only see a .yaml path and would leave every job held. * util_RIFT_pseudo_pipe.py -- accept singularity_rift_image / singularity_base_exe_dir from the ini (env still wins). * containers/ build kit (multi-target build_family.sh + template + example manifest) and docs/source/containers.rst. * test/test_container_manifest.py -- 25 tests; the integration cases inspect the generated condor commands, no pool needed. Not ported (separate features on rift_O4d): containers/survey_scan/ and the GitHub Actions dependency canaries (this branch has no .github/). Co-Authored-By: Claude Opus 5 --- .../Code/RIFT/misc/container_manifest.py | 516 ++++++++++++++++++ .../Code/RIFT/misc/dag_utils.py | 183 ++++++- ...te_event_parameter_pipeline_BasicIteration | 17 + .../Code/bin/util_RIFT_pseudo_pipe.py | 12 +- .../Code/test/test_container_manifest.py | 465 ++++++++++++++++ containers/README.md | 246 +++++++++ containers/build_family.sh | 129 +++++ containers/requirements-container.txt | 27 + containers/rift_container.def.in | 74 +++ containers/rift_container_family.yaml | 52 ++ docs/source/containers.rst | 295 ++++++++++ docs/source/index.rst | 1 + 12 files changed, 2003 insertions(+), 14 deletions(-) create mode 100644 MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py create mode 100644 MonteCarloMarginalizeCode/Code/test/test_container_manifest.py create mode 100644 containers/README.md create mode 100755 containers/build_family.sh create mode 100644 containers/requirements-container.txt create mode 100644 containers/rift_container.def.in create mode 100644 containers/rift_container_family.yaml create mode 100644 docs/source/containers.rst diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py new file mode 100644 index 000000000..49076edd6 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py @@ -0,0 +1,516 @@ +""" +container_manifest +================== + +Support for "container family" manifests used by the RIFT pipeline. + +Historically ``SINGULARITY_RIFT_IMAGE`` names a single ``.sif`` image (a local +path or an ``osdf://`` URL), and the ILE/CIP Condor jobs hard-code + + MY.SingularityImage = "" + +A *manifest* lets us instead advertise a *family* of images, each targeting a +different GPU compute capability, and let HTCondor pick the right one per matched +machine. When ``SINGULARITY_RIFT_IMAGE`` points at a ``.yaml``/``.yml`` file, +the job-submission code turns it into: + + * an expression-valued ``MY.SingularityImage`` -- a nested ``ifThenElse`` over + the matched machine's GPU capability attribute (default ``GPUs_Capability``) + that selects the highest-capability image the machine can run; and + * a ``require_gpus`` capability floor (the lowest capability any image in the + family supports), composed (``&&``) with any user-supplied + ``RIFT_REQUIRE_GPUS``; and + * for ``osdf://`` images, a *selective* ``transfer_input_files`` entry using + HTCondor ``$$()`` match-time substitution, so only the *matched* image is + transferred (CVMFS/local images are referenced in place and never + transferred). + +Single-``.sif`` behavior is completely unchanged: only ``.yaml``/``.yml`` values +exercise any of this. + +YAML schema +----------- + + version: 1 + capability_attr: GPUs_Capability # machine ClassAd attr the ifThenElse tests + fallback: ancient # label used as the innermost else-branch + containers: + - label: ancient + image: /cvmfs/.../rift_ancient_cuda11.sif # in-place (CVMFS/local) + cuda_capability_min: 3.0 # inclusive + cuda_capability_max: 7.0 # exclusive; null/omitted => open-ended + note: "cupy-cuda11x, ancient base" + - label: modern + image: osdf:///igwn/.../rift_modern_cuda12.sif # selectively transferred + cuda_capability_min: 7.0 + cuda_capability_max: null + note: "cupy-cuda12x, newer base" +""" + +import os + +__all__ = [ + "ContainerManifestError", + "is_container_manifest", + "load_container_manifest", + "build_singularity_image_expr", + "build_transfer_input_expr", + "build_require_gpus_floor", + "build_container_image_select", + "build_capability_defined_requirement", + "build_fallback_single_image", + "build_runtime_selection_wrapper", +] + +# Default machine ClassAd attribute advertising GPU compute capability. The +# user's pools advertise this via e.g. +# condor_status -constraint 'TotalGPUs > 0' -autoformat GPUs_DeviceName GPUs_Capability +DEFAULT_CAPABILITY_ATTR = "GPUs_Capability" + + +class ContainerManifestError(Exception): + """Raised for a missing/malformed container family manifest.""" + + +def is_container_manifest(value): + """Return True iff ``value`` (the ``SINGULARITY_RIFT_IMAGE`` string) names a + multi-container manifest rather than a single ``.sif``/``osdf://`` image. + + Pure string check (no filesystem access) so single-image callers pay zero + cost and their behavior is unchanged. + """ + if not value or not isinstance(value, str): + return False + return value.lower().endswith((".yaml", ".yml")) + + +def _image_needs_transfer(image): + """True iff ``image`` is a URL that must be fetched via Condor file transfer + (e.g. ``osdf://``). CVMFS/local paths (``/cvmfs/...``, ``./foo.sif``) are + resolved in place and return False. + """ + return "://" in image + + +def _image_runtime_path(image): + """The string used *inside* ``MY.SingularityImage`` for this image. + + Transferred (URL) images land in the job scratch dir under their basename, + so the pilot must reference ``./`` -- matching the existing + single-image osdf rewrite convention. In-place (CVMFS/local) images are + referenced verbatim. + """ + if _image_needs_transfer(image): + return "./{}".format(image.rstrip("/").split("/")[-1]) + return image + + +def _fmt_cap(value): + """Format a capability number for a ClassAd expression (e.g. 7.0 -> '7.0').""" + return repr(float(value)) + + +def load_container_manifest(path): + """Parse and validate a YAML container family manifest. + + Returns a dict ``{capability_attr, fallback, containers}`` where + ``containers`` is sorted by ``cuda_capability_min`` *descending* (containers + with no min sort last). + + Raises ``ContainerManifestError`` on a missing pyyaml, an unreadable or + malformed file, an empty container list, or an unknown ``fallback`` label. + """ + try: + import yaml + except ImportError as exc: # pragma: no cover - environment dependent + raise ContainerManifestError( + "PyYAML is required to read a container family manifest ({}); " + "install pyyaml or point SINGULARITY_RIFT_IMAGE at a single .sif".format(path) + ) from exc + + try: + with open(path, "r") as f: + data = yaml.safe_load(f) + except (IOError, OSError) as exc: + raise ContainerManifestError("Cannot read container manifest {}: {}".format(path, exc)) + except yaml.YAMLError as exc: + raise ContainerManifestError("Malformed container manifest {}: {}".format(path, exc)) + + if not isinstance(data, dict): + raise ContainerManifestError("Container manifest {} is not a mapping".format(path)) + + raw_containers = data.get("containers") + if not raw_containers or not isinstance(raw_containers, list): + raise ContainerManifestError( + "Container manifest {} must define a non-empty 'containers' list".format(path) + ) + + containers = [] + for idx, entry in enumerate(raw_containers): + if not isinstance(entry, dict): + raise ContainerManifestError( + "Container manifest {} entry #{} is not a mapping".format(path, idx) + ) + image = entry.get("image") + label = entry.get("label") + if not image: + raise ContainerManifestError( + "Container manifest {} entry #{} is missing 'image'".format(path, idx) + ) + if not label: + raise ContainerManifestError( + "Container manifest {} entry #{} is missing 'label'".format(path, idx) + ) + cap_min = entry.get("cuda_capability_min") + cap_max = entry.get("cuda_capability_max") + try: + cap_min = None if cap_min is None else float(cap_min) + cap_max = None if cap_max is None else float(cap_max) + except (TypeError, ValueError): + raise ContainerManifestError( + "Container manifest {} entry '{}' has non-numeric capability bounds".format( + path, label + ) + ) + containers.append( + { + "label": label, + "image": image, + "cuda_capability_min": cap_min, + "cuda_capability_max": cap_max, + "note": entry.get("note"), + } + ) + + # Sort by min capability descending; None mins (open-ended-low catch-alls) + # sort last. float('-inf') keeps them at the bottom. + containers.sort( + key=lambda c: (c["cuda_capability_min"] if c["cuda_capability_min"] is not None else float("-inf")), + reverse=True, + ) + + labels = {c["label"] for c in containers} + fallback = data.get("fallback") + if fallback is None: + # Default fallback = the most-compatible (lowest-min) container, i.e. the + # last one after the descending sort. This is the CPU-safe catch-all. + fallback = containers[-1]["label"] + elif fallback not in labels: + raise ContainerManifestError( + "Container manifest {} fallback '{}' is not one of {}".format( + path, fallback, sorted(labels) + ) + ) + + capability_attr = data.get("capability_attr") or DEFAULT_CAPABILITY_ATTR + + return { + "capability_attr": capability_attr, + "fallback": fallback, + "containers": containers, + } + + +def _capability_attr(manifest): + """Resolve the machine attribute used by the selection ifThenElse. + + Precedence: ``RIFT_GPU_CAPABILITY_ATTR`` env override > manifest + ``capability_attr`` > module default. + """ + return os.environ.get("RIFT_GPU_CAPABILITY_ATTR") or manifest["capability_attr"] + + +def _build_selector(manifest, value_fn, ternary=False): + """Build a nested capability selector over the family. + + ``value_fn(container)`` returns the ClassAd literal for a container branch + (already quoted as appropriate). The highest-min container is the outermost + test; the ``fallback`` container is the innermost else (catch-all, used when + the capability is below every threshold). + + With ``ternary=False`` the selector uses ``ifThenElse(cond, a, b)`` (commas). + With ``ternary=True`` it uses the comma-free ClassAd ternary ``cond ? a : b`` + -- required when the result is embedded as one element of a comma-separated + ``transfer_input_files`` list, where internal commas would be mis-split. + + NOTE: the selector is intentionally NOT undefined-guarded. A GPU job must add + a ``Requirements`` clause excluding slots that do not advertise the capability + attribute (:func:`build_capability_defined_requirement`); guessing an image + for an undefined slot is unsafe (it could be a Blackwell that hard-fails on the + older fallback image), so the correct action is to NOT match such a slot. A + non-GPU job must not use this selector at all -- it has no capability to read. + """ + attr = _capability_attr(manifest) + containers = manifest["containers"] # sorted desc by min + by_label = {c["label"]: c for c in containers} + fb = by_label[manifest["fallback"]] + + # Containers that contribute a capability threshold test (exclude the + # fallback so it is not duplicated as both a branch and the else). + thresholds = [ + c + for c in containers + if c["cuda_capability_min"] is not None and c["label"] != fb["label"] + ] + # Fold ascending so the highest min ends up outermost. + thresholds.sort(key=lambda c: c["cuda_capability_min"]) + + expr = value_fn(fb) + for c in thresholds: + cond = "TARGET.{attr} >= {mn}".format(attr=attr, mn=_fmt_cap(c["cuda_capability_min"])) + if ternary: + expr = "({cond} ? {val} : {inner})".format(cond=cond, val=value_fn(c), inner=expr) + else: + expr = "ifThenElse({cond}, {val}, {inner})".format( + cond=cond, val=value_fn(c), inner=expr + ) + return expr + + +def build_singularity_image_expr(manifest): + """Return the unquoted ClassAd expression for ``MY.SingularityImage``. + + Each branch literal is the container's *runtime* path (CVMFS/local verbatim, + ``./`` for transferred images). + + GPU jobs that emit this MUST also add + :func:`build_capability_defined_requirement` so they never match a slot that + does not advertise the capability attribute (where this expression would be + ``undefined``). + """ + return _build_selector( + manifest, lambda c: '"{}"'.format(_image_runtime_path(c["image"])) + ) + + +def build_transfer_input_expr(manifest): + """Return a single ``$$([ ... ])`` token for ``transfer_input_files`` that + fetches *only the matched* image, or ``None`` if no container in the family + needs transfer. + + Transfer branches yield the URL verbatim; in-place (CVMFS/local) branches + yield ``""`` (no transfer on those machines). Uses the comma-free ternary + form so the token survives comma-splitting of ``transfer_input_files``. + """ + if not any(_image_needs_transfer(c["image"]) for c in manifest["containers"]): + return None + + def value_fn(c): + return '"{}"'.format(c["image"]) if _image_needs_transfer(c["image"]) else '""' + + # GPU jobs that emit this MUST also add build_capability_defined_requirement so + # they never match a slot where TARGET. is undefined (this $$ token would + # then "cannot expand" and HOLD the job). + return "$$([ {} ])".format(_build_selector(manifest, value_fn, ternary=True)) + + +def build_container_image_select(manifest, request_gpu=True): + """Return the value for the HTCondor *container universe* ``container_image`` + submit command for this family. + + GPU jobs (``request_gpu=True``, the default) get a per-machine selection: an + unquoted ``$$([ ... ])`` token. ``$$()`` is HTCondor's *match-time machine-ad + substitution* -- the schedd evaluates the bracketed expression against the + matched machine ad and substitutes a literal image string into + ``container_image`` before the job reaches the execution point. Unlike + :func:`build_singularity_image_expr` (an execute-side ClassAd expression that + OSPool glidein pilots read as a literal string and hold on), the pilot only + ever sees a literal URL. ``$$`` in ``container_image`` is HTCondor's + documented mechanism for per-GPU-capability image selection, and it works on + both the CIT-local pool and OSPool glideins. The branch value is the manifest + image *verbatim* (an ``osdf://`` URL the container-universe file-transfer + plugin fetches, or a CVMFS/local path used in place) -- NOT a ``./basename`` + rewrite. ``container_image`` is a single submit command (not a comma list), + so the comma-bearing ``ifThenElse`` form is fine. + + **Non-GPU jobs (``request_gpu=False``) collapse to a SINGLE fixed container**: + the plain ``fallback`` image (a literal ``container_image``, no ``$$()``). + A CPU-only job (e.g. CIP) matches a slot that advertises **no** GPU capability + attribute, so a ``$$()`` capability expression has nothing to resolve against + -- it fails to expand and HTCondor *holds the job*. There is also nothing to + select between, so the CPU-safe fallback image is the right (and only) choice. + + The GPU-path ``$$()`` is NOT undefined-guarded: the GPU job that emits it MUST + also add :func:`build_capability_defined_requirement` so it never matches a + slot where the capability attr is undefined (guessing an image for such a slot + is unsafe -- it could be a Blackwell that hard-fails on the older fallback). + The non-GPU case never reaches the ``$$()`` (it returns the literal fallback). + """ + by_label = {c["label"]: c for c in manifest["containers"]} + fb_image = by_label[manifest["fallback"]]["image"] + if not request_gpu: + # Single fixed container: no capability, no $$() -- a plain literal. + return fb_image + selector = _build_selector(manifest, lambda c: '"{}"'.format(c["image"])) + return "$$([ {} ])".format(selector) + + +def build_capability_defined_requirement(manifest): + """Return a ``Requirements`` clause that excludes machines which do not + advertise the capability attribute the family selection reads. + + A GPU family job MUST add this. Measured on the CIT pool (2026-06-12), ~45% + of GPU slots satisfy the per-GPU ``require_gpus`` floor (which matches the + per-GPU ``Capability`` inside ``AvailableGPUs``) yet do NOT advertise the + machine-level rollup attribute (default ``GPUs_Capability``) that the + ``$$()``/``ifThenElse`` selection reads. On such a slot the selection cannot + expand and the job HOLDS ("Cannot expand $$ expression"). Excluding these + slots is the safe fix: an undefined-capability slot could be a Blackwell that + hard-fails on the older fallback image, so we must NOT match it (rather than + guess its image). The defined set still includes the high-capability nodes, + so the family's purpose is preserved. + + Generic on ``capability_attr``; a no-op on pools where every GPU slot + advertises it, hence merge-safe. + """ + return "TARGET.{attr} =!= undefined".format(attr=_capability_attr(manifest)) + + +def build_fallback_single_image(manifest): + """For jobs that must use a SINGLE fixed container (no capability selection) -- + e.g. CPU-only CIP, which requests no GPU and so cannot resolve a + ``$$()``/``ifThenElse`` capability selection (its matched slot advertises no + capability attribute -> the selection holds the job). + + Returns ``(runtime_path, transfer_url)`` for the manifest ``fallback`` (the + CPU-safe image): + + * ``runtime_path`` -- what ``MY.SingularityImage`` / ``container_image`` + references: ``./`` for a transferred ``osdf://`` image, the path + verbatim for a CVMFS/local image. (``MY.SingularityImage`` callers must + quote it; ``container_image`` takes it unquoted.) + * ``transfer_url`` -- the ``osdf://`` URL to add to ``transfer_input_files``, + or ``None`` if the image is referenced in place (CVMFS/local). + """ + fb_image = {c["label"]: c for c in manifest["containers"]}[manifest["fallback"]]["image"] + runtime_path = _image_runtime_path(fb_image) + transfer_url = fb_image if _image_needs_transfer(fb_image) else None + return runtime_path, transfer_url + + +# Body of the OSG-safe runtime-selection wrapper. @@TOKENS@@ are substituted by +# build_runtime_selection_wrapper (str.replace, not .format; the script is full +# of ${...} bash expansions that would collide with format()). +_RUNTIME_WRAPPER_BODY = r'''#!/bin/bash +# AUTO-GENERATED by RIFT.misc.container_manifest.build_runtime_selection_wrapper. +# OSG-safe runtime container selection. Runs as the Condor executable on the +# bare execute node, with no +SingularityImage. At job start it detects the real +# GPU compute capability, selects the matching family image, acquires only that +# image, and execs the real command inside it via nested apptainer. +set -euo pipefail +LABELS=( @@LABELS@@ ) +CAP_MIN=( @@MINS@@ ) +CAP_MAX=( @@MAXS@@ ) +RTPATH=( @@RTPATHS@@ ) +FETCH=( @@FETCHES@@ ) +FALLBACK_LABEL="@@FALLBACK@@" +INNER_COMMAND="@@INNER@@" + +log() { echo "[rift_container_select] $*" >&2; } + +cap="${RIFT_CONTAINER_FORCE_CAP:-}" +if [ -z "$cap" ] && command -v nvidia-smi >/dev/null 2>&1; then + cap="$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader,nounits 2>/dev/null | head -1 | tr -d '[:space:]')" || cap="" +fi +log "detected compute capability: ${cap:-}" + +sel=-1 +if [ -n "$cap" ]; then + best_min=-1 + for i in "${!LABELS[@]}"; do + lo="${CAP_MIN[$i]}"; hi="${CAP_MAX[$i]}" + if awk -v c="$cap" -v lo="$lo" -v hi="$hi" 'BEGIN{exit !(c+0>=lo+0 && c+0<=hi+0)}'; then + if awk -v lo="$lo" -v b="$best_min" 'BEGIN{exit !(lo+0>b+0)}'; then + best_min="$lo"; sel="$i" + fi + fi + done +fi +if [ "$sel" -lt 0 ]; then + log "no GPU match for cap='${cap:-}' -> fallback ${FALLBACK_LABEL}" + for i in "${!LABELS[@]}"; do + if [ "${LABELS[$i]}" = "$FALLBACK_LABEL" ]; then sel="$i"; fi + done +fi +[ "$sel" -lt 0 ] && { log "FATAL: fallback '${FALLBACK_LABEL}' not in table"; exit 3; } +log "selected: ${LABELS[$sel]} (${RTPATH[$sel]}) [cap band ${CAP_MIN[$sel]}-${CAP_MAX[$sel]}]" + +rt="${RTPATH[$sel]}"; fetch="${FETCH[$sel]}"; SIF="" +if [ -e "$rt" ]; then + SIF="$rt"; log "using in-place/local image: $SIF" +elif [ -n "$fetch" ]; then + log "fetching single image: $fetch -> $rt" + if command -v stashcp >/dev/null 2>&1; then stashcp "$fetch" "$rt" + elif command -v pelican >/dev/null 2>&1; then pelican object get "$fetch" "$rt" + else log "FATAL: no local image and no stashcp/pelican to fetch $fetch"; exit 4; fi + SIF="$rt" +else + log "FATAL: image '$rt' absent and no fetch URL"; exit 4 +fi + +if [ -n "$INNER_COMMAND" ]; then + log "exec: apptainer exec --nv ${RIFT_CONTAINER_APPTAINER_FLAGS:-} $SIF $INNER_COMMAND $*" + exec apptainer exec --nv ${RIFT_CONTAINER_APPTAINER_FLAGS:-} "$SIF" $INNER_COMMAND "$@" +else + log "exec: apptainer exec --nv ${RIFT_CONTAINER_APPTAINER_FLAGS:-} $SIF $*" + exec apptainer exec --nv ${RIFT_CONTAINER_APPTAINER_FLAGS:-} "$SIF" "$@" +fi +''' + + +def _runtime_image_fields(container): + """Return (runtime_path, fetch_url, cap_min, cap_max) for one container.""" + image = container["image"] + runtime_path = _image_runtime_path(image) + fetch_url = image if _image_needs_transfer(image) else "" + return runtime_path, fetch_url, container["cuda_capability_min"], container["cuda_capability_max"] + + +def build_runtime_selection_wrapper(manifest, inner_command=None): + """Return an OSG-safe runtime container-selection wrapper script. + + The wrapper is intended to run as the Condor executable on the bare execute + node. It chooses a container at job start from the same manifest used by the + ClassAd/container-universe selectors, then runs ``inner_command`` or the + wrapper arguments inside the selected image with apptainer. + """ + labels, mins, maxs, rtpaths, fetches = [], [], [], [], [] + for c in manifest["containers"]: + runtime_path, fetch_url, cap_min, cap_max = _runtime_image_fields(c) + labels.append(c["label"]) + mins.append("-1" if cap_min is None else repr(float(cap_min))) + maxs.append("9999" if cap_max is None else repr(float(cap_max))) + rtpaths.append(runtime_path) + fetches.append(fetch_url) + + def _arr(values): + return " ".join('"{}"'.format(v) for v in values) + + return ( + _RUNTIME_WRAPPER_BODY + .replace("@@LABELS@@", _arr(labels)) + .replace("@@MINS@@", _arr(mins)) + .replace("@@MAXS@@", _arr(maxs)) + .replace("@@RTPATHS@@", _arr(rtpaths)) + .replace("@@FETCHES@@", _arr(fetches)) + .replace("@@FALLBACK@@", manifest["fallback"]) + .replace("@@INNER@@", "" if not inner_command else str(inner_command)) + ) + + +def build_require_gpus_floor(manifest): + """Return a ``require_gpus`` capability floor expression for the family, or + ``None``. + + The floor is the lowest ``cuda_capability_min`` across the family -- i.e. do + not match a GPU less capable than anything we ship. Uses the require_gpus + sub-ad attribute ``Capability`` (unprefixed -- *not* ``TARGET.`` and *not* + ``GPUs_Capability``). + + If any container has no min (open-ended-low catch-all), there is effectively + no lower bound and ``None`` is returned. + """ + mins = [c["cuda_capability_min"] for c in manifest["containers"]] + if any(m is None for m in mins) or not mins: + return None + return "Capability >= {}".format(_fmt_cap(min(mins))) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py index ab2b63ec1..070a85d36 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py @@ -26,6 +26,30 @@ from glue import pipeline import configparser +# Container family manifest support (multi-architecture container deployment). +# Import-guarded so that plain single-.sif runs never require pyyaml; only an +# actual .yaml/.yml manifest exercises this path. +try: + from RIFT.misc.container_manifest import ( + is_container_manifest, + load_container_manifest, + build_singularity_image_expr, + build_transfer_input_expr, + build_require_gpus_floor, + build_container_image_select, + build_capability_defined_requirement, + build_fallback_single_image, + build_runtime_selection_wrapper, + ContainerManifestError, + ) + _HAVE_CONTAINER_MANIFEST = True +except ImportError: + _HAVE_CONTAINER_MANIFEST = False + + def is_container_manifest(value): + # Without the helper module available, never treat a value as a manifest. + return False + __author__ = "Evan Ochsner , Chris Pankow " # getenv=True deprecated, will need workaround to explicitly pull extra environment variables @@ -517,7 +541,30 @@ def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output- singularity_image_used = "{}".format(singularity_image) # make copy extra_files = [] - if singularity_image: + # Container family manifest support. CIP is CPU-only: it requests NO GPU, so a + # per-capability $$()/ifThenElse selection cannot resolve on its matched slot + # (no capability attribute -> the $$ "cannot expand" -> HOLD). CIP needs no GPU + # and no arch-specific image, so it uses a SINGLE fixed container = the manifest + # fallback (CPU-safe) image on BOTH the legacy and container-universe paths. + singularity_is_family = False + singularity_container_universe = False + singularity_container_image_select = None + singularity_fallback_runtime = None + if singularity_image and is_container_manifest(singularity_image): + singularity_is_family = True + _manifest = load_container_manifest(singularity_image) + singularity_container_universe = bool(use_singularity and os.environ.get('RIFT_CONTAINER_UNIVERSE')) + if singularity_container_universe: + # container_image = the fallback image URL verbatim (container universe + # fetches it); request_gpu=False -> single image, no $$() selection. + singularity_container_image_select = build_container_image_select(_manifest, request_gpu=False) + else: + # legacy: MY.SingularityImage = the single fallback (quoted in the image + # block below); transfer just that one image if it is an osdf URL. + singularity_fallback_runtime, _fb_transfer = build_fallback_single_image(_manifest) + if _fb_transfer: + extra_files += [_fb_transfer] + elif singularity_image: if 'osdf:' in singularity_image: singularity_image_used = "./{}".format(singularity_image.split('/')[-1]) extra_files += [singularity_image] @@ -534,7 +581,8 @@ def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output- exe=singularity_base_exe_path + path_split[-1] if path_split[-1] == 'true': # special universal path for /bin/true, don't override it! exe = "/usr/bin/true" - ile_job = pipeline.CondorDAGJob(universe=universe, executable=exe) + # Container universe (opt-in) runs the job inside container_image directly. + ile_job = pipeline.CondorDAGJob(universe=("container" if singularity_container_universe else universe), executable=exe) # This is a hack since CondorDAGJob hides the queue property ile_job._CondorJob__queue = ncopies @@ -635,8 +683,18 @@ def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output- # Compare to https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py ile_job.add_condor_cmd('request_CPUs', str(1)) ile_job.add_condor_cmd('transfer_executable', 'False') - ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') - ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') + if singularity_container_universe: + # Container universe: the image is delivered via container_image (a plain + # literal here -- CIP requests no GPU, so no $$() selection), emitted + # unquoted, with NO MY.SingularityImage / MY.SingularityBindCVMFS. + ile_job.add_condor_cmd("container_image", singularity_container_image_select) + else: + ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') + if singularity_is_family: + # Single fixed (CPU-safe fallback) image from the family manifest. + ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_fallback_runtime + '"') + else: + ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') requirements.append("HAS_SINGULARITY=?=TRUE") if use_oauth_files: @@ -833,12 +891,60 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, singularity_image_used = "{}".format(singularity_image) # make copy extra_files = [] - if singularity_image: + # Container family manifest support: if singularity_image points at a + # .yaml/.yml manifest, build an expression-valued MY.SingularityImage plus a + # selective ($$()) transfer entry and a require_gpus capability floor. A + # plain .sif / osdf:// value keeps the legacy single-image behavior below. + singularity_is_family = False + singularity_container_universe = False + singularity_container_image_select = None + singularity_runtime_select = False + singularity_inner_exe = None + singularity_image_expr = None + singularity_transfer_expr = None + singularity_require_gpus_floor = None + if singularity_image and is_container_manifest(singularity_image): + singularity_is_family = True + _manifest = load_container_manifest(singularity_image) + singularity_image_expr = build_singularity_image_expr(_manifest) + singularity_transfer_expr = build_transfer_input_expr(_manifest) + singularity_require_gpus_floor = build_require_gpus_floor(_manifest) + # OSG-safe alternative (opt-in: RIFT_CONTAINER_UNIVERSE). The execute-side + # ifThenElse MY.SingularityImage below works on the CIT-local pool but + # OSPool glidein pilots read SingularityImage as a LITERAL string and hold + # the job. Container universe with container_image = $$([...]) instead + # uses HTCondor match-time machine-ad substitution: the schedd resolves it + # to a literal image before the job reaches the EP. $$ in container_image + # is HTCondor's documented per-GPU-capability selection, and works on both + # CIT-local and OSPool. Requires use_singularity. + singularity_container_universe = bool(use_singularity and os.environ.get('RIFT_CONTAINER_UNIVERSE')) + if singularity_container_universe: + # request_gpu gates per-capability selection vs a single fixed + # container: a job that requests no GPU matches a slot with no GPU + # capability, so a $$() capability expression cannot resolve (it + # holds the job) -- collapse to the single fallback image instead. + singularity_container_image_select = build_container_image_select(_manifest, request_gpu=request_gpu) + else: + # Older OSG-safe fallback (opt-in): run a wrapper on the bare execute + # node that detects the runtime GPU, fetches just that image, and + # execs the real command under apptainer. Keep this independent of + # container universe, which is the preferred OSG path when enabled. + singularity_runtime_select = bool(use_singularity and os.environ.get('RIFT_CONTAINER_RUNTIME_SELECT')) + # Selective transfer: only the matched osdf image is fetched (via the + # $$() token, which is comma-free so it survives transfer_input_files + # comma-splitting). CVMFS/local images are referenced in place and + # never transferred, so the whole family is never pulled. In container- + # universe mode the image is delivered via container_image itself; in + # runtime-select mode the wrapper self-fetches. In both cases do NOT add + # the match-time transfer token. + if singularity_transfer_expr and not singularity_container_universe and not singularity_runtime_select: + extra_files += [singularity_transfer_expr] + elif singularity_image: if 'osdf:' in singularity_image: singularity_image_used = "./{}".format(singularity_image.split('/')[-1]) extra_files += [singularity_image] - + exe = exe or which("integrate_likelihood_extrinsic") frames_local = None if use_singularity: @@ -851,6 +957,7 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, # singularity_base_exe_path = "/opt/lscsoft/rift/MonteCarloMarginalizeCode/Code/" # should not hardcode this ...! singularity_base_exe_path = "/usr/bin/" # should not hardcode this ...! exe=singularity_base_exe_path + path_split[-1] + singularity_inner_exe = exe if not(frames_dir is None): frames_local = frames_dir.split("/")[-1] elif use_osg: # NOT using singularity! @@ -888,7 +995,9 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, exe = exe_here # update executable - ile_job = pipeline.CondorDAGJob(universe="vanilla", executable=exe) + # Container universe (opt-in) runs the job inside container_image directly; + # otherwise stay vanilla + (optional) condor singularity. + ile_job = pipeline.CondorDAGJob(universe=("container" if singularity_container_universe else "vanilla"), executable=exe) # This is a hack since CondorDAGJob hides the queue property ile_job._CondorJob__queue = ncopies @@ -994,9 +1103,30 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, if use_singularity: # Compare to https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py ile_job.add_condor_cmd('request_CPUs', str(1)) - ile_job.add_condor_cmd('transfer_executable', 'False') - ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') - ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') + if singularity_runtime_select: + # Runtime-select wrapper: the wrapper is the transferred executable + # and it invokes apptainer itself, so do not ask HTCondor to enter + # singularity or suppress executable transfer. + pass + else: + ile_job.add_condor_cmd('transfer_executable', 'False') + if singularity_container_universe: + # Container universe: the per-machine image is delivered via + # container_image, a $$()-substituted (match-time) literal -- emit it + # raw/unquoted (a $$() value must not be wrapped in quotes), with NO + # MY.SingularityImage / MY.SingularityBindCVMFS. GPU access is + # automatic under request_gpus; the executable runs inside the image. + ile_job.add_condor_cmd("container_image", singularity_container_image_select) + elif singularity_runtime_select: + pass + else: + ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') + if singularity_is_family: + # Expression-valued: emit the ifThenElse raw, with NO surrounding + # double quotes (a classad expression must not be quoted). + ile_job.add_condor_cmd("MY.SingularityImage", singularity_image_expr) + else: + ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') ile_job.add_condor_cmd("MY.flock_local",'true') # jobs can match to local pool ! requirements.append("HAS_SINGULARITY=?=TRUE") # if not(use_simple_osg_requirements): @@ -1095,6 +1225,15 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, for name in name_list: requirements.append('TARGET.Machine =!= "{}" '.format(name)) + # Container-family GPU jobs: exclude slots that don't advertise the machine-level + # capability attribute the per-machine image selection reads. ~45% of CIT GPU + # slots satisfy the per-GPU require_gpus floor but don't advertise the rollup + # attr, so the $$()/ifThenElse "cannot expand" and the job HOLDS. Excluding + # them is safe; guessing an image is not (an undefined slot could be a Blackwell + # that hard-fails on the older fallback). Only when a GPU is actually requested. + if singularity_is_family and request_gpu: + requirements.append(build_capability_defined_requirement(_manifest)) + # Write requirements # From https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) @@ -1128,9 +1267,19 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, remove_str = 'JobStatus =?= 2 && (CurrentTime - JobStartDate) > ( {})'.format(60*max_runtime_minutes) ile_job.add_condor_cmd('periodic_remove', remove_str) - if 'RIFT_REQUIRE_GPUS' in os.environ: # new convention 'require_gpus = ' to specify conditions on GPU properties - ile_job.add_condor_cmd('require_gpus',os.environ['RIFT_REQUIRE_GPUS']) - + # require_gpus: compose the user's RIFT_REQUIRE_GPUS (used today to block + # incompatible hosts by DeviceName) with the container family's capability + # floor (lowest capability any image in the family supports). Both apply; + # neither is silently dropped. (new convention 'require_gpus = ' specifies + # conditions on GPU properties) + require_gpus_terms = [] + if 'RIFT_REQUIRE_GPUS' in os.environ: + require_gpus_terms.append('({})'.format(os.environ['RIFT_REQUIRE_GPUS'])) + if singularity_is_family and singularity_require_gpus_floor: + require_gpus_terms.append('({})'.format(singularity_require_gpus_floor)) + if require_gpus_terms: + ile_job.add_condor_cmd('require_gpus', ' && '.join(require_gpus_terms)) + ### ### SUGGESTION FROM STUART (for later) @@ -1141,6 +1290,14 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, for cmd, value in condor_commands.items(): ile_job.add_condor_cmd(cmd, value) + if singularity_runtime_select: + wrapper_text = build_runtime_selection_wrapper(_manifest, inner_command=singularity_inner_exe) + wrapper_name = 'rift_container_select.sh' + with open(wrapper_name, 'w') as f: + f.write(wrapper_text) + os.system('chmod a+x ' + wrapper_name) + ile_job.set_executable(wrapper_name) + return ile_job, ile_sub_name diff --git a/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration b/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration index 47f8043f6..8a1e51967 100755 --- a/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration +++ b/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration @@ -356,6 +356,23 @@ if opts.use_singularity: opts.use_oauth_files = 'igwn' # force-set this variable, to save time for end user. This will always be the case elif 'osdf:' in singularity_image and not opts.use_oauth_files: opts.use_oauth_files = 'scitokens' # force-set this variable, to save time for end user. This will always be the case + elif not opts.use_oauth_files: + # Container FAMILY manifest (.yaml/.yml): the per-machine image osdf:// URLs live + # INSIDE the manifest, so the 'osdf:' substring checks above miss them and the + # transfer credential is never enabled -> the execute point cannot fetch the + # selected container ("credential is required for osdf://... but was not + # discovered" -> all ILE/CIP jobs held). Inspect the manifest's image URLs and + # pick the same credential the single-image path would have. + try: + from RIFT.misc.container_manifest import is_container_manifest, load_container_manifest + if is_container_manifest(singularity_image): + _imgs = ' '.join(c.get('image', '') for c in load_container_manifest(singularity_image).get('containers', [])) + if 'igwn+osdf:' in _imgs: + opts.use_oauth_files = 'igwn' + elif 'osdf:' in _imgs: + opts.use_oauth_files = 'scitokens' + except Exception: + pass if (opts.cip_args is None) and (opts.cip_args_list is None): print(" No arguments provided for low-level job") diff --git a/MonteCarloMarginalizeCode/Code/bin/util_RIFT_pseudo_pipe.py b/MonteCarloMarginalizeCode/Code/bin/util_RIFT_pseudo_pipe.py index c196de4f3..8eb7ece30 100755 --- a/MonteCarloMarginalizeCode/Code/bin/util_RIFT_pseudo_pipe.py +++ b/MonteCarloMarginalizeCode/Code/bin/util_RIFT_pseudo_pipe.py @@ -344,7 +344,17 @@ def unsafe_parse_arg_string_dict(my_argstr): if not('RIFT_REQUIRE_GPUS' in os.environ) and 'ile_require_gpus' in rift_items: os.environ['RIFT_REQUIRE_GPUS'] = rift_items['ile_require_gpus'] - + + # Container family (multi-container per-machine image selection): let the ini + # file provide the image/exe-dir, as the environment normally would. The value + # may be a single .sif/osdf URL (legacy) or a .yaml/.yml family manifest. + # Environment still dominates, matching the accounting/require-GPUs behavior above. + if not('SINGULARITY_RIFT_IMAGE' in os.environ) and 'singularity_rift_image' in rift_items: + os.environ['SINGULARITY_RIFT_IMAGE'] = rift_items['singularity_rift_image'] + if not('SINGULARITY_BASE_EXE_DIR' in os.environ) and 'singularity_base_exe_dir' in rift_items: + os.environ['SINGULARITY_BASE_EXE_DIR'] = rift_items['singularity_base_exe_dir'] + + # attempt to lazy-select the command-line that are present in the ini file section for item in rift_items: item_renamed = item.replace('-','_') diff --git a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py new file mode 100644 index 000000000..436fb5663 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py @@ -0,0 +1,465 @@ +""" +Tests for container family manifest parsing and the expression-valued +SingularityImage / selective-transfer / require_gpus wiring. + +These run without a real HTCondor pool: the parser + expression builders are +pure, and the integration test inspects the generated ``condor_cmds`` on the +job object returned by ``write_ILE_sub_simple`` (no .sub file or condor needed). + +Run directly: python test/test_container_manifest.py +Or via pytest: pytest test/test_container_manifest.py +""" + +import os +import shutil +import stat +import subprocess +import sys +import textwrap + +import pytest + +yaml = pytest.importorskip("yaml") # manifest parsing requires PyYAML + +import RIFT.misc.container_manifest as cm + + +# --------------------------------------------------------------------------- +# helpers +# --------------------------------------------------------------------------- + +MIXED_MANIFEST = textwrap.dedent( + """ + version: 1 + fallback: ancient + containers: + - label: ancient + image: /cvmfs/sw/rift_ancient_cuda11.sif + cuda_capability_min: 3.0 + cuda_capability_max: 7.0 + - label: modern + image: osdf:///igwn/rift_modern_cuda12.sif + cuda_capability_min: 7.0 + """ +) + +ALL_CVMFS_MANIFEST = textwrap.dedent( + """ + version: 1 + fallback: ancient + containers: + - label: ancient + image: /cvmfs/sw/rift_ancient.sif + cuda_capability_min: 3.0 + - label: modern + image: /cvmfs/sw/rift_modern.sif + cuda_capability_min: 7.0 + """ +) + + +def _write(tmp_path, text, name="fam.yaml"): + p = tmp_path / name + p.write_text(text) + return str(p) + + +# ``dag_utils`` builds glue.pipeline CondorDAGJob objects, whose submit commands +# and universe are only reachable through accessors (the attributes themselves +# are name-mangled). Keep the assertions below written against plain values. +def _cmds(job): + return dict(job.get_condor_cmds()) + + +def _universe(job): + return job.get_universe() + + +# --------------------------------------------------------------------------- +# 1. parser +# --------------------------------------------------------------------------- + +def test_parser_sorts_and_resolves_fallback(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + # sorted by capability descending + assert [c["label"] for c in m["containers"]] == ["modern", "ancient"] + assert m["fallback"] == "ancient" + assert m["capability_attr"] == cm.DEFAULT_CAPABILITY_ATTR + + +def test_parser_default_fallback_is_lowest(tmp_path): + # no explicit fallback -> most-compatible (lowest-min) container + text = MIXED_MANIFEST.replace("fallback: ancient\n", "") + m = cm.load_container_manifest(_write(tmp_path, text)) + assert m["fallback"] == "ancient" + + +def test_parser_rejects_unknown_fallback(tmp_path): + text = MIXED_MANIFEST.replace("fallback: ancient", "fallback: nope") + with pytest.raises(cm.ContainerManifestError): + cm.load_container_manifest(_write(tmp_path, text)) + + +def test_parser_rejects_empty(tmp_path): + with pytest.raises(cm.ContainerManifestError): + cm.load_container_manifest(_write(tmp_path, "version: 1\ncontainers: []\n")) + + +def test_parser_rejects_missing_image(tmp_path): + text = "containers:\n - label: x\n cuda_capability_min: 5.0\n" + with pytest.raises(cm.ContainerManifestError): + cm.load_container_manifest(_write(tmp_path, text)) + + +# --------------------------------------------------------------------------- +# 2. expressions +# --------------------------------------------------------------------------- + +def test_image_expression(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + expr = cm.build_singularity_image_expr(m) + # Bare capability selector (NOT undefined-guarded): a GPU job must instead add + # the build_capability_defined_requirement Requirements clause so it never + # matches a slot where this would be undefined. + assert expr == ( + 'ifThenElse(TARGET.GPUs_Capability >= 7.0, ' + '"./rift_modern_cuda12.sif", "/cvmfs/sw/rift_ancient_cuda11.sif")' + ) + # an expression must NOT be a quoted string literal + assert not expr.startswith('"') + + +def test_transfer_expression_is_comma_free_ternary(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + expr = cm.build_transfer_input_expr(m) + assert expr == ( + '$$([ (TARGET.GPUs_Capability >= 7.0 ? ' + '"osdf:///igwn/rift_modern_cuda12.sif" : "") ])' + ) + # the token sits inside a comma-separated transfer_input_files list, so it + # must contain no commas of its own + assert "," not in expr + + +def test_transfer_expression_none_when_all_in_place(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, ALL_CVMFS_MANIFEST)) + assert cm.build_transfer_input_expr(m) is None + + +def test_require_gpus_floor(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + assert cm.build_require_gpus_floor(m) == "Capability >= 3.0" + + +def test_selectors_are_not_undefined_guarded(tmp_path): + # The capability selectors must NOT default an undefined-capability slot to the + # fallback image: that slot could be a Blackwell that hard-fails on the older + # fallback. The safe fix is the Requirements exclusion below, not a guess. + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + assert "=?= undefined" not in cm.build_singularity_image_expr(m) + assert "=?= undefined" not in cm.build_transfer_input_expr(m) + assert "=?= undefined" not in cm.build_container_image_select(m) + + +def test_capability_defined_requirement(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + # excludes slots that don't advertise the (machine-level) capability attr + assert cm.build_capability_defined_requirement(m) == "TARGET.GPUs_Capability =!= undefined" + + +def test_capability_defined_requirement_respects_attr_override(tmp_path, monkeypatch): + monkeypatch.setenv("RIFT_GPU_CAPABILITY_ATTR", "CUDACapability") + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + assert cm.build_capability_defined_requirement(m) == "TARGET.CUDACapability =!= undefined" + + +def test_fallback_single_image(tmp_path): + # MIXED fallback (ancient) is a CVMFS image -> referenced in place, no transfer + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + runtime, transfer = cm.build_fallback_single_image(m) + assert runtime == "/cvmfs/sw/rift_ancient_cuda11.sif" + assert transfer is None + # an osdf fallback -> runtime is ./basename and it IS transferred + osdf_text = MIXED_MANIFEST.replace("/cvmfs/sw/rift_ancient_cuda11.sif", + "osdf:///igwn/rift_ancient_cuda11.sif") + m2 = cm.load_container_manifest(_write(tmp_path, osdf_text, name="fam2.yaml")) + runtime2, transfer2 = cm.build_fallback_single_image(m2) + assert runtime2 == "./rift_ancient_cuda11.sif" + assert transfer2 == "osdf:///igwn/rift_ancient_cuda11.sif" + + +def test_capability_attr_env_override(tmp_path, monkeypatch): + monkeypatch.setenv("RIFT_GPU_CAPABILITY_ATTR", "CUDACapability") + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + assert "TARGET.CUDACapability >=" in cm.build_singularity_image_expr(m) + + +# --------------------------------------------------------------------------- +# 3-5. integration with write_ILE_sub_simple (inspect generated condor_cmds) +# --------------------------------------------------------------------------- + +def _make_ile_job(tmp_path, monkeypatch, singularity_image): + """Call write_ILE_sub_simple in an isolated cwd; return its condor_cmds dict. + + Skips if the dag_utils backend cannot be imported in this env. + """ + dag = pytest.importorskip("RIFT.misc.dag_utils") + monkeypatch.chdir(tmp_path) + job, _ = dag.write_ILE_sub_simple( + tag="ILE", + log_dir=str(tmp_path) + "/", + exe="/usr/bin/true", + arg_str="--foo bar", + transfer_files=["../all.net"], + use_singularity=True, + singularity_image=singularity_image, + request_gpu=True, + cache_file="local.cache", + ) + return _cmds(job) + + +def test_integration_family_mixed(tmp_path, monkeypatch): + monkeypatch.setenv( + "RIFT_REQUIRE_GPUS", '(DeviceName=!="Tesla K10.G1.8GB")' + ) + cmds = _make_ile_job(tmp_path, monkeypatch, _write(tmp_path, MIXED_MANIFEST)) + + img = cmds["MY.SingularityImage"] + assert img.startswith("ifThenElse(") # expression, not a literal + assert not img.startswith('"') + + # selective transfer: exactly one $$() token, whole family NOT dumped + tif = cmds["transfer_input_files"] + assert tif.count("$$([") == 1 + assert "/cvmfs/sw/rift_ancient_cuda11.sif" not in tif # cvmfs image not transferred + assert tif.count("osdf:///igwn/rift_modern_cuda12.sif") == 1 + + # floor composed with (not replacing) the user's RIFT_REQUIRE_GPUS + rg = cmds["require_gpus"] + assert "Capability >= 3.0" in rg + assert 'DeviceName=!="Tesla K10.G1.8GB"' in rg + assert "&&" in rg + + # GPU family job: Requirements exclude slots that don't advertise the + # machine-level capability attr (else the selection $$/ifThenElse holds). + assert "TARGET.GPUs_Capability =!= undefined" in cmds["requirements"] + + +def test_integration_all_cvmfs_no_transfer_token(tmp_path, monkeypatch): + cmds = _make_ile_job(tmp_path, monkeypatch, _write(tmp_path, ALL_CVMFS_MANIFEST)) + assert "$$([" not in cmds.get("transfer_input_files", "") + # still an expression-valued image + a capability floor + assert cmds["MY.SingularityImage"].startswith("ifThenElse(") + assert "Capability >= 3.0" in cmds["require_gpus"] + + +def test_backward_compat_single_sif(tmp_path, monkeypatch): + monkeypatch.delenv("RIFT_REQUIRE_GPUS", raising=False) + cmds = _make_ile_job(tmp_path, monkeypatch, "./foo.sif") + # byte-identical legacy behavior: quoted literal, no $$() token, no floor + assert cmds["MY.SingularityImage"] == '"./foo.sif"' + assert "$$([" not in cmds.get("transfer_input_files", "") + assert "require_gpus" not in cmds + + +# --------------------------------------------------------------------------- +# 6. container universe: $$()-substituted container_image selection (OSG-safe) +# --------------------------------------------------------------------------- + +def test_container_image_select_expression(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + expr = cm.build_container_image_select(m) + # a $$() match-time substitution token with VERBATIM image values (osdf URL + # fetched by container universe; cvmfs path used in place) -- NOT a ./basename + # rewrite, and NOT undefined-guarded (Requirements exclusion is used instead) + assert expr.startswith("$$([ ") and expr.endswith(" ])") + assert "=?= undefined" not in expr # not a guess-guard + assert "ifThenElse(TARGET.GPUs_Capability >= 7.0," in expr + assert '"osdf:///igwn/rift_modern_cuda12.sif"' in expr # raw osdf URL + assert '"/cvmfs/sw/rift_ancient_cuda11.sif"' in expr # fallback verbatim + assert "./rift_modern_cuda12.sif" not in expr # no basename rewrite + + +def test_integration_container_universe(tmp_path, monkeypatch): + # Opt-in container-universe mode: per-machine image via $$()-substituted + # container_image; no MY.SingularityImage / BindCVMFS / $$() transfer token; + # universe=container; require_gpus floor still applied. + monkeypatch.setenv("RIFT_CONTAINER_UNIVERSE", "1") + monkeypatch.delenv("RIFT_REQUIRE_GPUS", raising=False) + monkeypatch.chdir(tmp_path) + dag = pytest.importorskip("RIFT.misc.dag_utils") + job, _ = dag.write_ILE_sub_simple( + tag="ILE", + log_dir=str(tmp_path) + "/", + exe="/usr/bin/true", + arg_str="--foo bar", + transfer_files=["../all.net"], + use_singularity=True, + singularity_image=_write(tmp_path, MIXED_MANIFEST), + request_gpu=True, + cache_file="local.cache", + ) + cmds = _cmds(job) + + ci = cmds["container_image"] + assert ci.startswith("$$([") # match-time substitution, unquoted + assert not ci.startswith('"') + assert "MY.SingularityImage" not in cmds # the OSG-breaking attr is gone + assert "MY.SingularityBindCVMFS" not in cmds + assert "$$([" not in cmds.get("transfer_input_files", "") # image via container_image, not transfer + assert "Capability >= 3.0" in cmds["require_gpus"] # floor still steers GPUs + # GPU family job: still excludes slots that don't advertise the capability attr + assert "TARGET.GPUs_Capability =!= undefined" in cmds["requirements"] + + assert _universe(job) == "container" # HTCondor container universe + + +def test_container_image_select_no_gpu_collapses_to_single(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + # A job that requests no GPU has no capability to key on -- a $$() expression + # would not resolve on a CPU-only slot and would HOLD the job. Collapse to + # the single (CPU-safe) fallback image: a plain literal, no $$(), no ifThenElse. + val = cm.build_container_image_select(m, request_gpu=False) + assert val == "/cvmfs/sw/rift_ancient_cuda11.sif" # the fallback image, verbatim + assert "$$(" not in val + assert "ifThenElse" not in val + + +def test_integration_cip_container_universe_single_image(tmp_path, monkeypatch): + # CIP is CPU-only: under container universe it must use a SINGLE fixed + # container (the fallback image), never the $$() capability selection. + monkeypatch.setenv("RIFT_CONTAINER_UNIVERSE", "1") + monkeypatch.chdir(tmp_path) + dag = pytest.importorskip("RIFT.misc.dag_utils") + job, _ = dag.write_CIP_sub( + tag="CIP", + out_dir=str(tmp_path), + log_dir=str(tmp_path) + "/", + exe="/usr/bin/true", + arg_str="--foo bar", + transfer_files=["../all.net"], + use_singularity=True, + singularity_image=_write(tmp_path, MIXED_MANIFEST), + ) + cmds = _cmds(job) + ci = cmds["container_image"] + assert ci == "/cvmfs/sw/rift_ancient_cuda11.sif" # single fixed fallback image + assert "$$(" not in ci # NOT a capability $$() selection + assert "MY.SingularityImage" not in cmds + assert "$$([" not in cmds.get("transfer_input_files", "") + assert "require_gpus" not in cmds # CPU job: no GPU floor + assert _universe(job) == "container" + + +def test_integration_cip_legacy_single_image(tmp_path, monkeypatch): + # CIP (CPU-only) on the LEGACY path: a single QUOTED fallback MY.SingularityImage + # (a bare path is a ClassAd parse error), NOT the family $$()/ifThenElse selection + # (a CPU slot can't resolve it), no $$() transfer token, and NO capability + # Requirements exclusion (CIP requests no GPU, so it must not be GPU-constrained). + monkeypatch.delenv("RIFT_CONTAINER_UNIVERSE", raising=False) + monkeypatch.chdir(tmp_path) + dag = pytest.importorskip("RIFT.misc.dag_utils") + job, _ = dag.write_CIP_sub( + tag="CIP", + out_dir=str(tmp_path), + log_dir=str(tmp_path) + "/", + exe="/usr/bin/true", + arg_str="--foo bar", + transfer_files=["../all.net"], + use_singularity=True, + singularity_image=_write(tmp_path, MIXED_MANIFEST), + ) + cmds = _cmds(job) + assert cmds["MY.SingularityImage"] == '"/cvmfs/sw/rift_ancient_cuda11.sif"' # single, quoted + assert "ifThenElse" not in cmds["MY.SingularityImage"] + assert "container_image" not in cmds + assert "$$([" not in cmds.get("transfer_input_files", "") + assert "=!= undefined" not in cmds.get("requirements", "") # CPU job: no GPU exclusion + assert "require_gpus" not in cmds + + +# --------------------------------------------------------------------------- +# 7. runtime-selection wrapper fallback +# --------------------------------------------------------------------------- + +def test_runtime_wrapper_text_contents(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + text = cm.build_runtime_selection_wrapper(m, inner_command="./ile_pre.sh") + assert text.startswith("#!/bin/bash") + assert '"./rift_modern_cuda12.sif"' in text + assert '"/cvmfs/sw/rift_ancient_cuda11.sif"' in text + assert '"osdf:///igwn/rift_modern_cuda12.sif"' in text + assert 'FALLBACK_LABEL="ancient"' in text + assert 'INNER_COMMAND="./ile_pre.sh"' in text + bash = shutil.which("bash") + if bash: + r = subprocess.run([bash, "-n", "-c", text], capture_output=True, text=True) + assert r.returncode == 0, r.stderr + + +@pytest.mark.skipif(not shutil.which("bash") or not shutil.which("awk"), + reason="needs bash + awk to exercise the wrapper") +def test_runtime_wrapper_selects_by_capability(tmp_path): + anc = tmp_path / "anc.sif"; anc.write_text("x") + mod = tmp_path / "mod.sif"; mod.write_text("x") + manifest_text = textwrap.dedent( + """ + version: 1 + fallback: ancient + containers: + - label: ancient + image: {anc} + cuda_capability_min: 3.0 + cuda_capability_max: 7.0 + - label: modern + image: {mod} + cuda_capability_min: 7.0 + """ + ).format(anc=anc, mod=mod) + m = cm.load_container_manifest(_write(tmp_path, manifest_text)) + wrapper = tmp_path / "select.sh" + wrapper.write_text(cm.build_runtime_selection_wrapper(m, inner_command="/bin/true")) + wrapper.chmod(wrapper.stat().st_mode | stat.S_IEXEC) + + fakebin = tmp_path / "bin"; fakebin.mkdir() + fake = fakebin / "apptainer" + fake.write_text('#!/bin/bash\necho "APPTAINER $*"\n') + fake.chmod(fake.stat().st_mode | stat.S_IEXEC) + env = dict(os.environ, PATH="{}:{}".format(fakebin, os.environ["PATH"])) + + def run(cap): + env2 = dict(env, RIFT_CONTAINER_FORCE_CAP=cap) + return subprocess.run([str(wrapper)], capture_output=True, text=True, env=env2) + + r = run("12.0") + assert r.returncode == 0, r.stderr + assert "selected: modern" in r.stderr + r = run("5.0") + assert "selected: ancient" in r.stderr + r = run("2.0") + assert "fallback" in r.stderr and "selected: ancient" in r.stderr + + +def test_integration_runtime_select(tmp_path, monkeypatch): + monkeypatch.delenv("RIFT_CONTAINER_UNIVERSE", raising=False) + monkeypatch.setenv("RIFT_CONTAINER_RUNTIME_SELECT", "1") + monkeypatch.delenv("RIFT_REQUIRE_GPUS", raising=False) + monkeypatch.setenv("SINGULARITY_BASE_EXE_DIR", "/opt/rift/bin/") + cmds = _make_ile_job(tmp_path, monkeypatch, _write(tmp_path, MIXED_MANIFEST)) + + assert "MY.SingularityImage" not in cmds + assert "MY.SingularityBindCVMFS" not in cmds + assert "transfer_executable" not in cmds + assert "$$([" not in cmds.get("transfer_input_files", "") + assert "Capability >= 3.0" in cmds["require_gpus"] + + wrapper = tmp_path / "rift_container_select.sh" + assert wrapper.exists() + body = wrapper.read_text() + assert body.startswith("#!/bin/bash") + assert 'INNER_COMMAND="/opt/rift/bin/true"' in body + + +if __name__ == "__main__": + sys.exit(pytest.main([os.path.abspath(__file__), "-v"])) diff --git a/containers/README.md b/containers/README.md new file mode 100644 index 000000000..bcfeadc81 --- /dev/null +++ b/containers/README.md @@ -0,0 +1,246 @@ +# RIFT containers + +This directory holds the multi-architecture container build and the "container +family" deployment mechanism. It has three related pieces: + +1. **Multi-target build** — build a *family* of RIFT containers (different base + image + cupy/CUDA variant, targeting different GPU compute capabilities) from + one template. +2. **Family deployment** — let `SINGULARITY_RIFT_IMAGE` point at a YAML + *manifest* describing that family, so each Condor job picks the right image + for the machine it lands on. +3. **Survey + warmup scans** — survey a target Condor GPU pool and emit + representative CuPy/JAX warmup jobs for the image bands that pool actually + uses. + +The top-level [`rift_container.def`](../rift_container.def) is unchanged and +remains the default single-image build. + +--- + +## 1. Building a family + +``` +containers/build_family.sh [--render-only] [OUTPUT_DIR] +``` + +- [`rift_container.def.in`](rift_container.def.in) is a template with + `@@BASE_IMAGE@@` / `@@CUPY_PKG@@` placeholders (apptainer `.def` files take no + build args, so we render then build). +- [`build_family.sh`](build_family.sh) holds the build `MATRIX`. The **first** + entry is the default and uses the current production base image, so the family + always includes a broadly-compatible image for older machines. Add rows to + target more architectures. +- `--render-only` writes the per-entry `.def` files without invoking apptainer + (useful in CI or on a machine without apptainer). +- Each build also emits a `rift_container_family.generated.yaml` stub — fill in + each `image:` with where you published the `.sif` (a CVMFS path or `osdf://` + URL), and you have a deployable manifest. + +All matrix entries share the pip set in +[`requirements-container.txt`](requirements-container.txt) (the cupy wheel is the +only per-entry difference). That file is the **single source of truth** also +consumed by the CI dependency canary (below). `build_family.sh` stages it into +each image via the `.def`'s `%files` section, so the build does **not** depend on +the cloned RIFT branch shipping the file. + +### Build troubleshooting + +**`proot error: ptrace(TRACEME): Operation not permitted` / +`mksquashfs command failed`** (seen on shared clusters such as CIT). Apptainer +has no usable user namespaces or setuid install, so it falls back to its +unprivileged `proot` build engine — which cannot run the `mksquashfs` helper. +**Setting `PROOT_NO_SECCOMP=1` is not sufficient** (it silences the seccomp +message but proot still fails to exec mksquashfs). Avoid the proot path instead: + +1. **Build with `--fakeroot`** (recommended; the IGWN/CIT path): + + ```console + containers/build_family.sh --fakeroot ./container_family + ``` + + Requires `/etc/subuid` + `/etc/subgid` entries for your user and unprivileged + user namespaces enabled (check: `grep $USER /etc/subuid` and + `apptainer build --fakeroot` on a tiny def). This produces a real `.sif` + without proot. + +2. **If even `--fakeroot` is unavailable, build a `--sandbox`** (a directory). + This skips `mksquashfs` entirely, so it sidesteps the failing step: + + ```console + containers/build_family.sh --sandbox ./container_family + # later, on a host where apptainer can make a SIF: + apptainer build rift_container_default.sif ./container_family/rift_container_default/ + ``` + +3. **Or build elsewhere** — on a node/registry with proper apptainer (or build + the OCI image with Docker/podman, push to a registry, then + `apptainer pull`/`build` the `.sif` on a capable host). + +`build_family.sh` still exports `PROOT_NO_SECCOMP=1` as a harmless best-effort, +and passes any extra `--flag` you give it straight through to `apptainer build`. +If a build runs out of space mid-way, point `APPTAINER_TMPDIR` at a large local +disk. + +--- + +## 2. Deploying a family via a manifest + +Set `SINGULARITY_RIFT_IMAGE` to a `.yaml`/`.yml` manifest instead of a single +`.sif`. Everything else (pseudo_pipe, `--use-singularity`, etc.) is unchanged — +the manifest is detected by file extension. A plain `.sif` path or single +`osdf://` URL keeps the **exact** legacy single-image behavior; the manifest path +is never consulted in that case. + +See [`rift_container_family.yaml`](rift_container_family.yaml) for a worked +example. Schema: + +| field | meaning | +|-------------------|---------| +| `version` | manifest schema version (currently `1`) | +| `capability_attr` | machine ClassAd attribute the selection expression tests (default `GPUs_Capability`) | +| `fallback` | label of the catch-all image (innermost `else`); **must be CPU-safe** | +| `containers[]` | the family | +| ↳ `label` | human id; also referenced by `fallback` | +| ↳ `image` | a CVMFS/local path (referenced in place, lazy-fetched) **or** an `osdf://` URL (selectively transferred) | +| ↳ `cuda_capability_min` | inclusive lower capability bound for this image | +| ↳ `cuda_capability_max` | informational upper bound (`null` = open-ended) | +| ↳ `note` | free-text | + +> **Keep the family consistent.** A *single* `SINGULARITY_BASE_EXE_DIR` is +> applied to **every** image in the family — the ILE/CIP jobs locate the +> executable as `SINGULARITY_BASE_EXE_DIR + `, with no per-image +> override. So all images in a manifest **must install RIFT's executables at the +> same in-container path** (and share a common layout/Python/entrypoints). Build +> them from the same `rift_container.def.in` template (`build_family.sh` does +> this) and do **not** hand-mix images with different internal layouts. The same +> applies to `SINGULARITY_BASE_EXE_DIR_HYPERPIPE` if you use hyperpipe. + +### What the pipeline generates + +For the ILE (and CIP) Condor submit, a manifest produces: + +- **`MY.SingularityImage`** — an *unquoted* `ifThenElse(...)` expression that + selects the highest-capability image the matched machine can run, with the + `fallback` image as the innermost `else` (used when the machine's capability is + below every threshold): + + ``` + ifThenElse(TARGET.GPUs_Capability >= 8.0, "./rift_container_modern.sif", "/cvmfs/.../rift_container_default.sif") + ``` + +- **Selective transfer** — only `osdf://` images get fetched, and only on the + machine that selected them, via one HTCondor `$$()` match-time token appended + to `transfer_input_files` (CVMFS/local images are referenced in place and never + transferred, so the *whole family is never pulled*): + + ``` + $$([ (TARGET.GPUs_Capability >= 8.0 ? "osdf:///.../rift_container_modern.sif" : "") ]) + ``` + + `request_disk` is **not** auto-sized (image sizes are unknown at submit time) — + size it to your largest single transferred image. + +- **`require_gpus` floor** — `Capability >= `, + composed (`&&`) with any user-supplied `RIFT_REQUIRE_GPUS` (which today you use + to block incompatible hosts by `DeviceName`). Both apply; neither is dropped. + +- **A capability-defined `Requirements` clause** — `TARGET.GPUs_Capability =!= + undefined`. The selection is deliberately *not* undefined-guarded: an + undefined-capability slot could be anything (including a Blackwell that + hard-fails on the older fallback), so the safe action is to not match it rather + than guess. Measured on CIT, a large fraction of GPU slots satisfy the per-GPU + `require_gpus` floor yet do not advertise the machine-level rollup attribute; + without this clause those jobs hold with "Cannot expand $$ expression". + +**CIP is different.** It requests no GPU, so its matched slot advertises no +capability and a capability-keyed selection cannot resolve (it would hold the +job). CIP therefore collapses to a **single fixed container** — the manifest +`fallback` image as a quoted literal, with no `$$()` token and no capability +`Requirements` clause. This is why the fallback must be CPU-safe. + +### OSG: pick a delivery mode + +The expression-valued `MY.SingularityImage` is evaluated *execute-side*. OSPool +glidein pilots read `SingularityImage` as a **literal string**, so an +`ifThenElse` lands verbatim and the job holds. Two opt-in modes fix this, +selected by an environment variable at DAG-build time: + +| env var | behaviour | +|---|---| +| *(unset)* | legacy `universe = vanilla` + expression-valued `MY.SingularityImage`. Correct on a local/CIT pool; **not OSG-safe**. | +| `RIFT_CONTAINER_UNIVERSE=1` | **recommended for OSG.** `universe = container` + `container_image = $$([ ifThenElse(...) ])`. `$$()` is HTCondor's match-time (schedd-side) machine-ad substitution, so the pilot only ever sees a literal URL. No `MY.SingularityImage`, no `MY.SingularityBindCVMFS`, no `$$()` transfer token — the image arrives via `container_image`. GPU access is automatic under `request_gpus`. Works on CIT-local too. | +| `RIFT_CONTAINER_RUNTIME_SELECT=1` | older ILE-only fallback: Condor runs a generated `rift_container_select.sh` on the bare node, which reads the real capability from `nvidia-smi`, fetches only the matching image (`stashcp`/`pelican`) and re-execs under `apptainer exec --nv`. | + +Under asimov set it from the blueprint, not the shell: + +```yaml +scheduler: + singularity image: /path/to/rift_container_family.yaml + singularity base exe directory: /usr/local/bin/ + environment variables: + RIFT_CONTAINER_UNIVERSE: 1 +``` + +With `osdf://` images inside a manifest, the pipeline also enables the matching +transfer credential automatically (`use_oauth_services = scitokens`, or `igwn` +for `igwn+osdf:`) by inspecting the manifest's image URLs — the single-image path +keys off the `SINGULARITY_RIFT_IMAGE` string, which for a family is only a +`.yaml` path. + +### HTCondor GPU attribute names — important + +Two different namespaces are in play and are kept separate: + +- The **image-selection `ifThenElse`** reads the *machine* ClassAd. Default + `GPUs_Capability` (advertised on the OSG; some pools differ). Override per-run + with `RIFT_GPU_CAPABILITY_ATTR`, or per-manifest with `capability_attr`. Verify + on your pool: + + ``` + condor_status -constraint 'TotalGPUs > 0' -autoformat GPUs_DeviceName GPUs_Capability GPUs_GlobalMemoryMb + ``` + + Not every GPU host advertises this; on such hosts the expression collapses to + the fallback image and the `require_gpus` floor does the steering. + +- The **`require_gpus` floor** uses the require_gpus sub-ad attribute + `Capability` (unprefixed — *not* `TARGET.`, *not* `GPUs_`). + +### Requirements + +- `PyYAML` must be importable wherever the pipeline is built (only when a + manifest is actually used). Single-`.sif` runs never require it. + +### Validation status + +Validated on a real HTCondor pool + GPU (a cap-3.0 machine): + +- The advertised attributes are `GPUs_Capability` (machine ad) and `Capability` + (require_gpus sub-ad) — matching the defaults above. +- The `require_gpus` capability floor matches a compatible GPU and correctly + *excludes* an incompatible one (`Capability >= 7.0` did not match a cap-3.0 + GPU), so the floor steers GPU selection as intended. +- The `$$([ ifThenElse(TARGET.GPUs_Capability >= …, …) ])` transfer token is + honored at match time: only the matched image's URL is selected/transferred. +- The empty-result case — when a manifest *mixes* CVMFS and osdf entries and a + CVMFS branch is selected, the `$$()` token expands to `""` — is **tolerated**: + the empty entry is skipped and the job runs clean. (So mixed manifests are + safe; you do *not* need uniform all-osdf / all-cvmfs retrieval.) + +The expression-valued `MY.SingularityImage` is *not* OSPool-safe — a GWMS pilot +reads it as a literal string. Use `RIFT_CONTAINER_UNIVERSE=1` on the OSG (see +"OSG: pick a delivery mode" above). + +--- + +## 3. Not included on this branch + +Two adjacent pieces live on ``rift_O4d`` and were deliberately left out of this +port, which is scoped to the container *family* deployment path: + +- `containers/survey_scan/` — an operator workflow that inventories a target + GPU pool and emits per-image container-cache warmup jobs. +- the `container-dep-canary` GitHub Actions job, which tracks unpinned + dependency drift in `requirements-container.txt`. This branch has no + `.github/` workflows (CI is GitLab-based here). diff --git a/containers/build_family.sh b/containers/build_family.sh new file mode 100755 index 000000000..3b9348c8b --- /dev/null +++ b/containers/build_family.sh @@ -0,0 +1,129 @@ +#!/bin/bash +# Build a *family* of RIFT containers from containers/rift_container.def.in, one +# per build-matrix entry (different base image + cupy/CUDA variant, targeting +# different GPU compute capabilities). +# +# Usage: +# containers/build_family.sh [--render-only] [--fakeroot] [--sandbox] \ +# [other apptainer build flags] [OUTPUT_DIR] +# +# --render-only render the per-entry .def files but do NOT run apptainer +# (useful on machines without apptainer, or in CI) +# --fakeroot pass --fakeroot to `apptainer build` (RECOMMENDED on shared +# clusters such as CIT: avoids the unprivileged `proot` engine, +# whose mksquashfs step fails. Needs /etc/subuid + /etc/subgid +# entries for your user and unprivileged user namespaces.) +# --sandbox build a writable directory instead of a .sif. This SKIPS the +# mksquashfs step entirely, so it sidesteps the proot squashfs +# failure when even --fakeroot is unavailable. Convert to .sif +# later on a capable host: apptainer build out.sif sandbox_dir/ +# any other --flag is passed straight through to `apptainer build`. +# OUTPUT_DIR where rendered .def and built images land (default: ./container_family) +# +# The DEFAULT (first) matrix entry keeps the current production base image, so +# the family always includes a broadly-compatible image for older machines. +# Add rows to MATRIX to target more architectures. +# +# After building, publish the .sif files to CVMFS or osdf and edit +# containers/rift_container_family.yaml so SINGULARITY_RIFT_IMAGE can point at it. +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +TEMPLATE="${HERE}/rift_container.def.in" + +RENDER_ONLY=0 +SANDBOX=0 +BUILD_OPTS=() +OUTPUT_DIR="./container_family" +for arg in "$@"; do + case "$arg" in + --render-only) RENDER_ONLY=1 ;; + --sandbox) SANDBOX=1 ;; + --fakeroot) BUILD_OPTS+=(--fakeroot) ;; + --*) BUILD_OPTS+=("$arg") ;; # passthrough to apptainer build + *) OUTPUT_DIR="$arg" ;; + esac +done + +# Build matrix: "label|base_image|cupy_pkg|cuda_capability_min|cuda_capability_max" +# - The first entry is the DEFAULT and uses the current production base image. +# - cuda_capability_max may be empty (open-ended); it is informational and is +# echoed into the manifest stub for convenience. +MATRIX=( + "default|nvidia/cuda:11.8.0-runtime-ubuntu22.04|cupy-cuda11x|3.5|8.0" + "modern|nvidia/cuda:12.4.1-runtime-ubuntu22.04|cupy-cuda12x|8.0|" +) + +# Workaround for unprivileged apptainer builds that fall back to the `proot` +# engine (no usable user namespaces / setuid apptainer, common on shared +# clusters like CIT): proot's mksquashfs step is blocked by seccomp on ptrace +# ("proot error: ptrace(TRACEME): Operation not permitted"). PROOT_NO_SECCOMP=1 +# disables proot's seccomp filtering and lets the build finish. Harmless when +# proot is not used; override by exporting it yourself before running. +# A fully privileged / fakeroot / userns-capable build does not need this. +export PROOT_NO_SECCOMP="${PROOT_NO_SECCOMP:-1}" + +mkdir -p "${OUTPUT_DIR}" +MANIFEST_STUB="${OUTPUT_DIR}/rift_container_family.generated.yaml" +{ + echo "# Auto-generated manifest stub from containers/build_family.sh." + echo "# Edit 'image:' to the published CVMFS path or osdf:// URL of each .sif." + echo "# IMPORTANT: a single SINGULARITY_BASE_EXE_DIR is applied to the whole" + echo "# family -- all images must install RIFT executables at the SAME" + echo "# in-container path. These are built from one template, so they are" + echo "# consistent; do not hand-swap in images with a different layout." + echo "version: 1" + echo "capability_attr: GPUs_Capability" + echo "fallback: default" + echo "containers:" +} > "${MANIFEST_STUB}" + +for row in "${MATRIX[@]}"; do + IFS='|' read -r label base cupy cap_min cap_max <<< "$row" + rendered="${OUTPUT_DIR}/rift_container_${label}.def" + sif="${OUTPUT_DIR}/rift_container_${label}.sif" + + echo ">>> Rendering ${label}: base=${base} cupy=${cupy}" + sed -e "s#@@BASE_IMAGE@@#${base}#g" \ + -e "s#@@CUPY_PKG@@#${cupy}#g" \ + -e "s#@@REQFILE@@#${HERE}/requirements-container.txt#g" \ + "${TEMPLATE}" > "${rendered}" + + { + echo " - label: ${label}" + echo " image: REPLACE_ME/rift_container_${label}.sif # publish to CVMFS or osdf" + echo " cuda_capability_min: ${cap_min}" + if [ -n "${cap_max}" ]; then + echo " cuda_capability_max: ${cap_max}" + else + echo " cuda_capability_max: null" + fi + echo " note: \"base=${base}, ${cupy}\"" + } >> "${MANIFEST_STUB}" + + if [ "${RENDER_ONLY}" -eq 1 ]; then + echo " (render-only) wrote ${rendered}" + continue + fi + if ! command -v apptainer >/dev/null 2>&1; then + echo " apptainer not found; wrote ${rendered} (build skipped)" >&2 + continue + fi + + if [ "${SANDBOX}" -eq 1 ]; then + target="${OUTPUT_DIR}/rift_container_${label}" # writable directory (no mksquashfs) + sandbox_opt=(--sandbox) + else + target="${sif}" + sandbox_opt=() + fi + echo ">>> Building ${target}${BUILD_OPTS[*]:+ (opts: ${BUILD_OPTS[*]})}" + # ${arr[@]+"${arr[@]}"} expands safely even when the array is empty under set -u + apptainer build ${sandbox_opt[@]+"${sandbox_opt[@]}"} ${BUILD_OPTS[@]+"${BUILD_OPTS[@]}"} "${target}" "${rendered}" +done + +echo +echo "Done. Rendered defs (and any built .sif) are in ${OUTPUT_DIR}/" +echo "Manifest stub: ${MANIFEST_STUB}" +echo "Next: publish the .sif images, fill in their 'image:' locations, and point" +echo "SINGULARITY_RIFT_IMAGE at the resulting .yaml manifest." diff --git a/containers/requirements-container.txt b/containers/requirements-container.txt new file mode 100644 index 000000000..e90cf5e6b --- /dev/null +++ b/containers/requirements-container.txt @@ -0,0 +1,27 @@ +# Shared pip dependency set for the RIFT container builds. +# +# SINGLE SOURCE OF TRUTH: the multi-target build (containers/rift_container.def.in, +# which stages this file into the image via its %files section) and the CI +# "dependency-resolution canary" (.github/workflows/ci.yml :: container-dep-canary) +# both install from this file, so the canary exercises the same unpinned set the +# family containers ship. (The top-level rift_container.def keeps an equivalent +# inline list for the default single build.) +# +# NOTE: the GPU-specific cupy wheel (cupy-cuda11x vs cupy-cuda12x) is NOT listed +# here -- it varies per build-matrix entry and is installed by the .def itself. +# The canary has no GPU, so it skips cupy entirely. +# +# Intentionally UNPINNED (mirrors rift_container.def). The canary's whole job is +# to catch when a fresh upstream release of one of these (e.g. swig>=4.4.0 via a +# transitive build, lalsuite, numpy) breaks RIFT -- see issue #136 -- before it +# surprises a container rebuild. +asimov>=0.5.6 +asimov-gwdata>=0.4.0 +gwdatafind==1.2.0 +gwosc>=0.7.1 +lalsuite>=7.26 +numpy>=1.24.4 +natsort +pybind11>=2.12 +scipy>=1.9.3 +pyseobnr diff --git a/containers/rift_container.def.in b/containers/rift_container.def.in new file mode 100644 index 000000000..3d881c426 --- /dev/null +++ b/containers/rift_container.def.in @@ -0,0 +1,74 @@ +# Parameterized apptainer definition for the RIFT container *family*. +# +# This is a TEMPLATE. Apptainer .def files take no build args, so +# containers/build_family.sh renders a concrete .def per build-matrix entry by +# substituting the @@PLACEHOLDERS@@ below, then runs `apptainer build`. +# +# Placeholders: +# @@BASE_IMAGE@@ - docker base image (e.g. nvidia/cuda:11.8.0-runtime-ubuntu22.04) +# @@CUPY_PKG@@ - cupy wheel matched to the base CUDA version (cupy-cuda11x / cupy-cuda12x) +# +# The top-level rift_container.def is left in place as the default single build; +# this template + build_family.sh is the multi-target path. +Bootstrap: docker +From: @@BASE_IMAGE@@ + +%files + # Stage the shared dependency list from the HOST build tree into the image, + # so the build does NOT depend on the cloned RIFT branch carrying this file + # (the clone below may be a branch/release that predates it). build_family.sh + # fills in the absolute host path of containers/requirements-container.txt. + @@REQFILE@@ /opt/requirements-container.txt + +%post + # Update the system and install essential libraries + apt-get update -y + apt-get install -y \ + build-essential \ + cmake \ + g++ \ + wget \ + python3.10 \ + python3.10-venv \ + python3-pip \ + curl \ + bc \ + locales \ + git \ + libkrb5-dev \ + libgsl-dev + + # Configure locale + locale-gen en_US.UTF-8 + + # Ensure Python symlink is in place + ln -s /usr/bin/python3.10 /usr/local/bin/python3 + ln -s /usr/bin/python3 /usr/local/bin/python + + # Set up RIFT installation, using MAIN SOURCE. Modify if you want a release version, or a different branch! + cd /opt + mkdir installed_RIFT + cd installed_RIFT + git clone https://github.com/oshaughn/research-projects-RIT.git + cd research-projects-RIT + #git checkout rift_O4c + pip3 install --upgrade pip + pip3 install --upgrade setuptools --break-system-packages + pip3 install -e . + + # GPU-specific cupy variant, matched to the base image CUDA version. + pip3 install @@CUPY_PKG@@ + + # Shared dependency set -- single source of truth, also exercised by the CI + # dependency-resolution canary (containers/requirements-container.txt). + # Staged into the image via the %files section above (independent of the + # cloned branch). + pip3 install -r /opt/requirements-container.txt + +%environment + # Set environment variables + alias python=python3 + +%labels + org.rift.base @@BASE_IMAGE@@ + org.rift.cupy @@CUPY_PKG@@ diff --git a/containers/rift_container_family.yaml b/containers/rift_container_family.yaml new file mode 100644 index 000000000..1a87b1a5e --- /dev/null +++ b/containers/rift_container_family.yaml @@ -0,0 +1,52 @@ +# Example RIFT container *family* manifest. +# +# Point SINGULARITY_RIFT_IMAGE at a copy of this file (a .yaml / .yml path) to +# deploy a family of containers instead of a single .sif. The pipeline turns it +# into an expression-valued MY.SingularityImage that picks the right image per +# matched machine's GPU capability, a selective ($$()) transfer for osdf images, +# and a require_gpus capability floor. +# +# A plain .sif path or single osdf:// URL keeps the legacy single-image behavior +# (this file is NOT consulted in that case). +# +# See containers/README.md for the full schema and the HTCondor GPU-attribute +# caveat. +# +# IMPORTANT -- keep the family CONSISTENT. A single SINGULARITY_BASE_EXE_DIR is +# applied to *every* image in the family (the ILE/CIP jobs locate the executable +# as SINGULARITY_BASE_EXE_DIR + , with no per-image override). So all +# images listed below MUST install RIFT's executables at the SAME in-container +# path (and otherwise share a common layout/Python/entrypoints). Build them from +# the same containers/rift_container.def.in template (build_family.sh does this) +# -- do NOT mix images with different internal layouts. Same goes for +# SINGULARITY_BASE_EXE_DIR_HYPERPIPE if you use hyperpipe. + +version: 1 + +# Machine ClassAd attribute the image-selection ifThenElse tests. Default +# GPUs_Capability (advertised on the OSG; verify on your pool with e.g. +# condor_status -constraint 'TotalGPUs > 0' -af GPUs_DeviceName GPUs_Capability +# ). Overridable per-run via the RIFT_GPU_CAPABILITY_ATTR env var. +capability_attr: GPUs_Capability + +# Innermost else-branch of the selection expression: used when the machine +# advertises no/low capability (and on CPU-only CIP slots). MUST be the +# CPU-safe / most broadly compatible image. +fallback: default + +containers: + # Default, broadly-compatible image for older machines. Referenced in place + # on CVMFS -- never transferred; CVMFS lazy-fetches it only when selected. + - label: default + image: /cvmfs/singularity.opensciencegrid.org/oshaughn/rift_container_default.sif + cuda_capability_min: 3.5 + cuda_capability_max: 8.0 + note: "base=nvidia/cuda:11.8.0-runtime-ubuntu22.04, cupy-cuda11x" + + # Newer image for higher-capability GPUs. Delivered via osdf: only the + # matched machine fetches it (selective $$() transfer). + - label: modern + image: osdf:///igwn/staging/oshaughn/rift_containers/rift_container_modern.sif + cuda_capability_min: 8.0 + cuda_capability_max: null + note: "base=nvidia/cuda:12.4.1-runtime-ubuntu22.04, cupy-cuda12x" diff --git a/docs/source/containers.rst b/docs/source/containers.rst new file mode 100644 index 000000000..05dab2d1a --- /dev/null +++ b/docs/source/containers.rst @@ -0,0 +1,295 @@ +Containers and multi-architecture deployment +============================================= + +RIFT runs its compute jobs (ILE, CIP) inside a Singularity/Apptainer container +on HTCondor pools such as the OSG. Historically the environment variable +``SINGULARITY_RIFT_IMAGE`` names a **single** image, and every job is pinned to +it:: + + export SINGULARITY_RIFT_IMAGE=/cvmfs/singularity.opensciencegrid.org/.../rift:production + +That still works exactly as before. This page documents two additions: + +* a **container *family*** — point ``SINGULARITY_RIFT_IMAGE`` at a YAML + *manifest* describing several images that target different GPU compute + capabilities, and let HTCondor pick the right one per matched machine; and +* a **multi-target build** that produces such a family from one template. + +.. note:: + + If ``SINGULARITY_RIFT_IMAGE`` is a plain ``.sif`` path or a single + ``osdf://`` URL, behavior is **unchanged** — the manifest machinery is never + engaged. A manifest is recognized purely by its ``.yaml`` / ``.yml`` suffix. + + +Deploying a container family +---------------------------- + +Set ``SINGULARITY_RIFT_IMAGE`` to a manifest file instead of a single image:: + + export SINGULARITY_RIFT_IMAGE=`pwd`/rift_container_family.yaml + +Everything else — ``util_RIFT_pseudo_pipe.py``, ``--use-singularity``, +``--use-osg`` — is identical. When the pipeline builds the ILE/CIP submit +files it reads the manifest and emits an *expression-valued* container +selection (see `What the pipeline generates`_ below). + +Manifest format +~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + version: 1 + + # Machine ClassAd attribute the selection expression tests. + # Default GPUs_Capability (see "GPU attribute names" below). + capability_attr: GPUs_Capability + + # Catch-all image (innermost else of the selection); MUST be CPU-safe, + # since it is also used when no GPU capability is advertised. + fallback: default + + containers: + # Broadly-compatible image for older machines. On CVMFS: referenced in + # place and lazy-fetched (only the selected image is ever pulled), never + # transferred. + - label: default + image: /cvmfs/singularity.opensciencegrid.org/oshaughn/rift_container_default.sif + cuda_capability_min: 3.5 # inclusive lower bound for this image + cuda_capability_max: 8.0 # informational; null = open-ended + note: "cupy-cuda11x, ubuntu22.04/cuda11.8" + + # Newer image for higher-capability GPUs. Delivered via osdf: only the + # matched machine fetches it (selective transfer). + - label: modern + image: osdf:///igwn/staging/oshaughn/rift_containers/rift_container_modern.sif + cuda_capability_min: 8.0 + cuda_capability_max: null + note: "cupy-cuda12x, ubuntu22.04/cuda12.4" + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Field + - Meaning + * - ``version`` + - Manifest schema version (currently ``1``). + * - ``capability_attr`` + - Machine ClassAd attribute the selection expression tests (default + ``GPUs_Capability``). + * - ``fallback`` + - ``label`` of the catch-all image (innermost ``else``); **must be + CPU-safe**. + * - ``containers[].label`` + - Human id; also referenced by ``fallback``. + * - ``containers[].image`` + - A CVMFS/local path (referenced in place, lazy-fetched) **or** an + ``osdf://`` URL (selectively transferred). + * - ``containers[].cuda_capability_min`` + - Inclusive lower capability bound for this image. + * - ``containers[].cuda_capability_max`` + - Informational upper bound (``null`` = open-ended). + * - ``containers[].note`` + - Free text. + +A starting manifest lives at :code:`containers/rift_container_family.yaml` in the +source tree. + +.. warning:: + + **Keep the family consistent.** A *single* ``SINGULARITY_BASE_EXE_DIR`` is + applied to **every** image in the family — the ILE/CIP jobs locate the + executable as ``SINGULARITY_BASE_EXE_DIR + ``, with no per-image + override. Every image in a manifest **must install RIFT's executables at the + same in-container path** (and share a common layout / Python / entrypoints). + Build them from the same ``rift_container.def.in`` template + (``build_family.sh`` does this); do **not** hand-mix images with different + internal layouts. The same applies to ``SINGULARITY_BASE_EXE_DIR_HYPERPIPE`` + if you use hyperpipe. + + +What the pipeline generates +--------------------------- + +For a manifest, the GPU ILE Condor submit files get: + +* **``MY.SingularityImage``** — an *unquoted* ``ifThenElse`` expression that + selects the highest-capability image the matched machine can run, with the + ``fallback`` image as the innermost ``else`` (used when the machine's + capability is below every threshold):: + + ifThenElse(TARGET.GPUs_Capability >= 8.0, "./rift_container_modern.sif", "/cvmfs/.../rift_container_default.sif") + +* **Selective transfer** — only ``osdf://`` images are fetched, and only on the + machine that selected them, via a single HTCondor ``$$()`` match-time token + appended to ``transfer_input_files``. CVMFS/local images are referenced in + place and never transferred, so the **whole family is never pulled**:: + + $$([ (TARGET.GPUs_Capability >= 8.0 ? "osdf:///.../rift_container_modern.sif" : "") ]) + + ``request_disk`` is **not** auto-sized — set it to your largest single + transferred image. + +* **``require_gpus`` floor** — ``Capability >= ``, + combined (``&&``) with any ``RIFT_REQUIRE_GPUS`` you set. Both apply; neither + is dropped. This stops jobs matching a GPU that *no* image in the family + supports. + +* **A capability-defined ``Requirements`` clause** — + ``TARGET.GPUs_Capability =!= undefined``. The selection above is deliberately + *not* undefined-guarded: a slot that does not advertise the machine-level + capability rollup could be anything (including a Blackwell that hard-fails on + the older fallback image), so the safe action is to **not match it** rather + than guess. Measured on the CIT pool, a large fraction of GPU slots satisfy + the per-GPU ``require_gpus`` floor yet do not advertise the rollup attribute; + without this clause those jobs go on hold with "Cannot expand $$ expression". + +CPU-only jobs are handled differently. **CIP requests no GPU**, so its matched +slot advertises no capability at all and a capability-keyed selection cannot +resolve — it would hold the job. CIP therefore collapses to a **single fixed +container**: the manifest ``fallback`` image (hence the requirement that the +fallback be CPU-safe), quoted as a plain literal, with no ``$$()`` token and no +capability ``Requirements`` clause. + + +.. _osg-container-modes: + +Choosing a delivery mode (legacy vs container universe) +------------------------------------------------------- + +The expression-valued ``MY.SingularityImage`` above is evaluated on the +*execute* side. That works on a local HTCondor pool, but **OSPool glidein +pilots read** ``SingularityImage`` **as a literal string**, so an ``ifThenElse`` +lands verbatim and the job holds. Two opt-in modes solve this; pick one with an +environment variable at DAG-build time. + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - Mode + - Behaviour + * - *(default)* + - Legacy: ``universe = vanilla`` + expression-valued + ``MY.SingularityImage``. Correct on a local/CIT pool. **Not OSG-safe.** + * - ``RIFT_CONTAINER_UNIVERSE=1`` + - **Recommended for OSG.** ``universe = container`` and + ``container_image = $$([ ifThenElse(...) ])``. ``$$()`` is HTCondor's + documented *match-time* (schedd-side) machine-ad substitution, so the + pilot only ever sees a literal image URL. No ``MY.SingularityImage``, + no ``MY.SingularityBindCVMFS``, no ``$$()`` transfer token — the image is + delivered by ``container_image`` itself. GPU access is automatic under + ``request_gpus``. Works on CIT-local too. + * - ``RIFT_CONTAINER_RUNTIME_SELECT=1`` + - Older fallback, ILE only. Condor runs a generated + ``rift_container_select.sh`` on the bare execute node; the wrapper reads + the *real* GPU capability from ``nvidia-smi``, fetches only the matching + image (``stashcp``/``pelican``), and re-execs the command under + ``apptainer exec --nv``. Detects the actual device rather than trusting + an advertised attribute, at the cost of running outside a container. + +Under asimov, set the variable from the blueprint rather than the shell: + +.. code-block:: yaml + + scheduler: + singularity image: /path/to/rift_container_family.yaml + singularity base exe directory: /usr/local/bin/ + environment variables: + RIFT_CONTAINER_UNIVERSE: 1 + +.. note:: + + With a family manifest whose images are ``osdf://`` URLs, the pipeline also + enables the matching transfer credential automatically + (``use_oauth_services = scitokens``, or ``igwn`` for ``igwn+osdf:``). The + single-image code path keys that off the ``SINGULARITY_RIFT_IMAGE`` string + itself, which for a manifest is just a ``.yaml`` path — so the manifest's + image URLs are inspected instead. + + +GPU attribute names +------------------- + +Two different ClassAd namespaces are involved, and they are kept separate: + +* The **image selection** ``ifThenElse`` reads the *machine* ad. The default + attribute is ``GPUs_Capability``. Override it per run with the environment + variable ``RIFT_GPU_CAPABILITY_ATTR``, or per manifest with ``capability_attr``. + Verify what your pool advertises:: + + condor_status -constraint 'TotalGPUs > 0' -autoformat GPUs_DeviceName GPUs_Capability GPUs_GlobalMemoryMb + + Not every GPU host advertises this; on such hosts the expression collapses to + the fallback image and the ``require_gpus`` floor does the steering. + +* The **``require_gpus`` floor** uses the require_gpus sub-ad attribute + ``Capability`` (unprefixed — *not* ``TARGET.``, *not* ``GPUs_``). + +.. note:: + + These mechanisms have been validated on a real HTCondor pool + GPU: the + attribute names, the ``require_gpus`` floor (matching a compatible GPU and + excluding an incompatible one), the ``$$()`` match-time image selection, and + tolerance of the empty-result case for a manifest that mixes CVMFS and + ``osdf`` entries. The expression-valued ``MY.SingularityImage`` is *not* + OSPool-safe (see :ref:`osg-container-modes`); use + ``RIFT_CONTAINER_UNIVERSE=1`` there. + + +Building a container family +--------------------------- + +The build lives under :code:`containers/`: + +* :code:`rift_container.def.in` — an Apptainer definition template with + ``@@BASE_IMAGE@@`` / ``@@CUPY_PKG@@`` placeholders. +* :code:`build_family.sh` — renders one ``.def`` per build-matrix entry and runs + ``apptainer build``. The **first** matrix entry keeps the current production + base image, so the family always includes a broadly-compatible image for older + machines. +* :code:`requirements-container.txt` — the shared, unpinned pip dependency set + (the cupy wheel is the only per-entry difference). + +.. code-block:: console + + # render the per-entry .def files only (no apptainer needed) + containers/build_family.sh --render-only ./container_family + + # render and build each .sif (requires apptainer) + containers/build_family.sh ./container_family + + # on shared clusters (e.g. CIT), build with --fakeroot to avoid the + # unprivileged proot engine (whose mksquashfs step fails): + containers/build_family.sh --fakeroot ./container_family + +Each run also writes a ``rift_container_family.generated.yaml`` stub: fill in +each ``image:`` with where you published the ``.sif`` (a CVMFS path or +``osdf://`` URL) and you have a deployable manifest. + +.. note:: + + On clusters without setuid apptainer or unprivileged user namespaces, a plain + build falls back to the ``proot`` engine and fails at ``mksquashfs`` + (``ptrace(TRACEME): Operation not permitted``). Use ``--fakeroot`` (needs + ``/etc/subuid`` + ``/etc/subgid`` entries), or ``--sandbox`` to build a + directory that skips ``mksquashfs`` and convert it to a ``.sif`` later on a + capable host. See the build-troubleshooting section of + :code:`containers/README.md`. + +The top-level :code:`rift_container.def` is unchanged and remains the default +single-image build. + + +Catching dependency breakage early +---------------------------------- + +The container ships an *unpinned* dependency set and clones RIFT at build time, +so a fresh upstream release (for example ``swig>=4.4.0``) can silently break +RIFT and only surface when a container rebuild fails. ``rift_O4d`` carries two +non-blocking GitHub Actions canaries for this (``container-dep-canary`` and +``container-swig-canary``); they are not part of this branch, whose CI is +GitLab-based. Until an equivalent exists here, re-run the install of +``containers/requirements-container.txt`` plus a RIFT import check by hand before +a family rebuild. diff --git a/docs/source/index.rst b/docs/source/index.rst index d46d50776..c59eda07e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -17,6 +17,7 @@ Rapid inference via Iterative FiTting: this algorithm provides a framework for e examples-noini getting-data osg + containers injections plotting hyperpipe From 38dc643b04e021419548f0571eedd378852bffbc Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Sun, 9 Aug 2026 15:58:16 -0700 Subject: [PATCH 2/4] containers: LALAPPS_PATH2CACHE inside the image under container universe; record condor_submit finding - create_event_parameter_pipeline_BasicIteration: container universe deliberately drops MY.SingularityBindCVMFS, so the hardcoded /cvmfs lalapps_path2cache that ile_pre.sh calls may not exist inside the image. Under RIFT_CONTAINER_UNIVERSE, use the container's own SINGULARITY_BASE_EXE_DIR/lal_path2cache instead. The legacy single-image path is untouched. - docs + containers/README: record what condor_submit 25.11 actually does with container_image = $$([...]). It derives ContainerImage as the text after the last '/' BEFORE any $$ expansion, so the selection is truncated ("...default.sif\") ])"). ContainerImageFullPath and transfer_input_files keep the intact $$ token and expand at match time, but ContainerImage no longer contains a $$ and nothing repairs it unless the schedd re-derives it -- which is unverified against a live GPU match. Flagged as an open item to check on one real ILE job before a production OSPool campaign; the runtime-select mode is unaffected. Evidence and a reproducer live in ~/LVK/IR1/demo_multi_container. Co-Authored-By: Claude Opus 5 --- ...te_event_parameter_pipeline_BasicIteration | 5 +++++ containers/README.md | 16 ++++++++++++++ docs/source/containers.rst | 21 +++++++++++++++++++ 3 files changed, 42 insertions(+) diff --git a/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration b/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration index 8a1e51967..2e28aca69 100755 --- a/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration +++ b/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration @@ -349,6 +349,11 @@ if opts.use_singularity: # SINGULARITY IMAGES ARE ON CVMFS, SO WE CAN AVOID THE SINGULARITY EXEC CALL # hardcoding a fiducial copy of lalapps_path2cache; beware about the executable name change os.environ['LALAPPS_PATH2CACHE'] = "/cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn-py39/bin/lalapps_path2cache" #"singularity exec {singularity_image} lalapps_path2cache".format(singularity_image=singularity_image) + if os.environ.get('RIFT_CONTAINER_UNIVERSE') and ('SINGULARITY_BASE_EXE_DIR' in os.environ): + # Container universe deliberately does NOT set MY.SingularityBindCVMFS, so the + # hardcoded /cvmfs path above may not exist inside the image. Use the copy the + # container itself ships (RIFT's own executables live in SINGULARITY_BASE_EXE_DIR). + os.environ['LALAPPS_PATH2CACHE'] = os.environ['SINGULARITY_BASE_EXE_DIR'].rstrip('/') + "/lal_path2cache" print(singularity_image) # see https://computing.docs.ligo.org/guide/htcondor/credentials/ diff --git a/containers/README.md b/containers/README.md index bcfeadc81..fce584f00 100644 --- a/containers/README.md +++ b/containers/README.md @@ -188,6 +188,22 @@ for `igwn+osdf:`) by inspecting the manifest's image URLs — the single-image p keys off the `SINGULARITY_RIFT_IMAGE` string, which for a family is only a `.yaml` path. +> **Open item on the container-universe mode.** `condor_submit` parses +> `container_image` *before* any `$$` expansion and derives the job ad's +> `ContainerImage` (the name the image gets in the job scratch dir) as the text +> after the **last** `/`. A `$$([ ... ])` selection contains slashes, so that +> derivation cuts it in half. On `$CondorVersion: 25.11.1`, `condor_submit +> -dry-run` of a generated `ILE.sub` gives +> `ContainerImage="rift_container_default.sif\") ])"` while +> `ContainerImageFullPath` keeps the intact `$$` token. The full path and +> `transfer_input_files` expand correctly at match time; `ContainerImage` has no +> `$$` left, so nothing repairs it unless the schedd re-derives it after +> expansion -- unverified against a live GPU match. Check this on one real ILE +> job (`condor_q -l`) before a production OSPool campaign. +> `RIFT_CONTAINER_RUNTIME_SELECT=1` emits no container-universe attributes and is +> unaffected. + + ### HTCondor GPU attribute names — important Two different namespaces are in play and are kept separate: diff --git a/docs/source/containers.rst b/docs/source/containers.rst index 05dab2d1a..5fbb19909 100644 --- a/docs/source/containers.rst +++ b/docs/source/containers.rst @@ -208,6 +208,27 @@ Under asimov, set the variable from the blueprint rather than the shell: itself, which for a manifest is just a ``.yaml`` path — so the manifest's image URLs are inspected instead. +.. warning:: + + **Open item on the container-universe mode.** ``condor_submit`` parses + ``container_image`` *before* any ``$$`` expansion, and derives the job ad's + ``ContainerImage`` (the name the image will have in the job scratch dir) as + the text after the **last** ``/``. A ``$$([ ... ])`` selection contains + slashes, so that derivation cuts the expression in half. On + ``$CondorVersion: 25.11.1``, ``condor_submit -dry-run`` of a generated + ``ILE.sub`` yields:: + + ContainerImage="rift_container_default.sif\") ])" + ContainerImageFullPath="$$([ ifThenElse(TARGET.GPUs_Capability >= 8.0, \"osdf:///...\", \"osdf:///...\") ])" + + ``ContainerImageFullPath`` and ``transfer_input_files`` keep the ``$$`` token + and expand correctly at match time; ``ContainerImage`` no longer contains a + ``$$``, so nothing repairs it unless the schedd re-derives it after expansion + — which has not been verified against a live GPU match. Confirm this on one + real ILE job (``condor_q -l`` the matched job) before running a production + OSPool campaign on this mode. ``RIFT_CONTAINER_RUNTIME_SELECT=1`` emits no + container-universe attributes at all and is not affected. + GPU attribute names ------------------- From 115960735767554c93aafe0a95f4dc8cd5822284 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Mon, 10 Aug 2026 01:59:42 -0700 Subject: [PATCH 3/4] container universe: fix container_image selector truncation (verified on a live OSPool match) The container-universe mode inherited from rift_O4d does not work on OSPool. condor_submit parses container_image BEFORE any $$ expansion and derives the job ad's ContainerImage -- the name the image gets in the job scratch dir -- as the text after the LAST '/'. rift_O4d put full osdf:// URLs in the $$([...]) selector, so it was cut in half and the surviving fragment is not a valid image name: ContainerImage="rift_o4d_cc60-90_cuda118_20260717.sif\") ])" ContainerImageFullPath keeps the $$ and expands correctly at match time; ContainerImage does not, and the OSG glidein PREPARE_JOB prepare-hook is what reads it. Three trivial jobs on the IGWN pool (3 MB sif images staged on OSDF, keyed on TARGET.Memory so they match a CPU slot; condor_submit derives ContainerImage identically either way): 5926098 plain single osdf image -> ran, exit 0 5926099 rift_O4d form (full URLs) -> HELD: "PREPARE_JOB (prepare-hook) failed (reported status 001): Unable to download or build singularity image cutest_busybox_20260810.sif\") ])" 5926100 this fix -> ran, exit 0, MATCH_EXP_ContainerImage = "cutest_alpine_20260810.sif" Fix: - build_container_image_select() emits BASENAMES, so the selector holds no '/', condor_submit's derivation is a no-op, the whole $$ token reaches the job ad, and the schedd expands it at match time. - The matched image is delivered by the comma-free $$() transfer token, which container universe previously skipped (it assumed container_image would fetch). - MY.TransferInput is pinned to the same list so condor_submit does not append the basename selector to TransferInput as a bogus extra input file. - A family containing an in-place (CVMFS/local) image now raises ContainerManifestError under container universe: such an image can only be named by its full path, which reintroduces the truncation. Stage it at a URL, or use RIFT_CONTAINER_RUNTIME_SELECT=1. The shipped example manifest is updated to all-URL accordingly. CIP is untouched -- CPU-only, already a single plain image, which condor_submit handles correctly. Tests, docs/source/containers.rst and containers/README.md updated to the corrected contract and the live evidence. Reproducer: ~/LVK/IR1/demo_multi_container (validate_build.sh, live_check/). Co-Authored-By: Claude Opus 5 --- .../Code/RIFT/misc/container_manifest.py | 64 +++++++++++++---- .../Code/RIFT/misc/dag_utils.py | 24 +++++-- .../Code/test/test_container_manifest.py | 70 +++++++++++++++---- containers/README.md | 36 ++++++---- containers/rift_container_family.yaml | 14 +++- docs/source/containers.rst | 54 ++++++++------ 6 files changed, 191 insertions(+), 71 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py index 49076edd6..8559663a0 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py @@ -105,6 +105,14 @@ def _image_runtime_path(image): return image +def _image_basename(image): + """The bare file name an image has once transferred into the job scratch dir. + + Used by the container-universe selector, which MUST NOT contain a ``/``. + """ + return image.rstrip("/").split("/")[-1] + + def _fmt_cap(value): """Format a capability number for a ClassAd expression (e.g. 7.0 -> '7.0').""" return repr(float(value)) @@ -311,17 +319,36 @@ def build_container_image_select(manifest, request_gpu=True): GPU jobs (``request_gpu=True``, the default) get a per-machine selection: an unquoted ``$$([ ... ])`` token. ``$$()`` is HTCondor's *match-time machine-ad substitution* -- the schedd evaluates the bracketed expression against the - matched machine ad and substitutes a literal image string into - ``container_image`` before the job reaches the execution point. Unlike - :func:`build_singularity_image_expr` (an execute-side ClassAd expression that - OSPool glidein pilots read as a literal string and hold on), the pilot only - ever sees a literal URL. ``$$`` in ``container_image`` is HTCondor's - documented mechanism for per-GPU-capability image selection, and it works on - both the CIT-local pool and OSPool glideins. The branch value is the manifest - image *verbatim* (an ``osdf://`` URL the container-universe file-transfer - plugin fetches, or a CVMFS/local path used in place) -- NOT a ``./basename`` - rewrite. ``container_image`` is a single submit command (not a comma list), - so the comma-bearing ``ifThenElse`` form is fine. + matched machine ad and substitutes a literal string before the job reaches the + execution point. Unlike :func:`build_singularity_image_expr` (an execute-side + ClassAd expression that OSPool glidein pilots read as a literal string and hold + on), the pilot only ever sees a literal image name. ``container_image`` is a + single submit command (not a comma list), so the comma-bearing ``ifThenElse`` + form is fine here. + + **The branch values are BASENAMES, not full URLs.** ``condor_submit`` parses + ``container_image`` *before* any ``$$`` expansion and derives the job ad's + ``ContainerImage`` -- the name the image will have in the job scratch dir -- as + the text after the **last** ``/``. A selector containing full paths therefore + gets cut in half, and what survives is not even a valid image name. This is not + theoretical: submitting the full-URL form to the IGWN pool holds the job at the + execute point with:: + + PREPARE_JOB (prepare-hook) failed (reported status 001): + Unable to download or build singularity image cutest_busybox_...sif") ]) + + With no ``/`` in the value, that derivation is a no-op, the whole ``$$`` token + survives into ``ContainerImage``, and the schedd expands it at match time + (``MATCH_EXP_ContainerImage = "rift_container_modern.sif"``) -- verified end to + end on an OSPool glidein. + + Because the selector now names only basenames, the caller MUST also deliver the + matched image itself: add :func:`build_transfer_input_expr` (the comma-free + ternary over the full URLs) to ``transfer_input_files`` **and** emit it as + ``MY.TransferInput`` so it overrides the entry ``condor_submit`` would otherwise + derive from ``container_image``. All images in the family must therefore be + transferable URLs; a family that references an image in place (CVMFS/local path) + cannot be selected this way and raises :class:`ContainerManifestError`. **Non-GPU jobs (``request_gpu=False``) collapse to a SINGLE fixed container**: the plain ``fallback`` image (a literal ``container_image``, no ``$$()``). @@ -339,9 +366,20 @@ def build_container_image_select(manifest, request_gpu=True): by_label = {c["label"]: c for c in manifest["containers"]} fb_image = by_label[manifest["fallback"]]["image"] if not request_gpu: - # Single fixed container: no capability, no $$() -- a plain literal. + # Single fixed container: no capability, no $$() -- a plain literal. This is + # the ordinary single-image path condor_submit handles correctly (it derives + # ContainerImage as the basename, which is exactly right). return fb_image - selector = _build_selector(manifest, lambda c: '"{}"'.format(c["image"])) + in_place = [c["label"] for c in manifest["containers"] if not _image_needs_transfer(c["image"])] + if in_place: + raise ContainerManifestError( + "container universe per-machine selection requires every image in the family " + "to be a transferable URL (e.g. osdf://), because the selector may not contain " + "a '/' -- condor_submit would truncate it. In-place image(s): {}. Either " + "stage those images at a URL, or use RIFT_CONTAINER_RUNTIME_SELECT=1 instead " + "of RIFT_CONTAINER_UNIVERSE=1.".format(", ".join(sorted(in_place))) + ) + selector = _build_selector(manifest, lambda c: '"{}"'.format(_image_basename(c["image"]))) return "$$([ {} ])".format(selector) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py index 070a85d36..e558a2d2e 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py @@ -933,11 +933,14 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, # Selective transfer: only the matched osdf image is fetched (via the # $$() token, which is comma-free so it survives transfer_input_files # comma-splitting). CVMFS/local images are referenced in place and - # never transferred, so the whole family is never pulled. In container- - # universe mode the image is delivered via container_image itself; in - # runtime-select mode the wrapper self-fetches. In both cases do NOT add - # the match-time transfer token. - if singularity_transfer_expr and not singularity_container_universe and not singularity_runtime_select: + # never transferred, so the whole family is never pulled. + # + # Container universe needs this token too: its container_image selector + # names BASENAMES (it may not contain a '/', or condor_submit truncates + # it -- see build_container_image_select), so the image itself must be + # delivered by file transfer. Runtime-select mode self-fetches inside + # the wrapper, so it is the only mode that skips the token. + if singularity_transfer_expr and not singularity_runtime_select: extra_files += [singularity_transfer_expr] elif singularity_image: if 'osdf:' in singularity_image: @@ -1252,6 +1255,15 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, fname_str=fname_str.strip() ile_job.add_condor_cmd('transfer_input_files', fname_str) ile_job.add_condor_cmd('should_transfer_files','YES') + if singularity_container_universe: + # condor_submit APPENDS the container_image value to the derived + # TransferInput. Our selector names basenames (it may not contain a + # '/'), so that appended entry would ask the execute point to fetch a + # bare file name from the access point and fail. Set TransferInput + # directly -- emitted after transfer_input_files, it wins -- so the + # list is exactly ours, with the matched image supplied by the + # comma-free $$() ternary already in extra_files. + ile_job.add_condor_cmd('MY.TransferInput', '"' + fname_str.replace('"', '\\"') + '"') if not transfer_output_files is None: if not isinstance(transfer_output_files, list): @@ -1260,7 +1272,7 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, fname_str = ','.join(transfer_output_files) fname_str=fname_str.strip() ile_job.add_condor_cmd('transfer_output_files', fname_str) - + # Periodic remove: kill jobs running longer than max runtime # https://stackoverflow.com/questions/5900400/maximum-run-time-in-condor if not(max_runtime_minutes is None): diff --git a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py index 436fb5663..b911e8b7d 100644 --- a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py +++ b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py @@ -58,6 +58,22 @@ ) +ALL_OSDF_MANIFEST = textwrap.dedent( + """ + version: 1 + fallback: ancient + containers: + - label: ancient + image: osdf:///igwn/sw/rift_ancient_cuda11.sif + cuda_capability_min: 3.0 + cuda_capability_max: 7.0 + - label: modern + image: osdf:///igwn/sw/rift_modern_cuda12.sif + cuda_capability_min: 7.0 + """ +) + + def _write(tmp_path, text, name="fam.yaml"): p = tmp_path / name p.write_text(text) @@ -158,7 +174,8 @@ def test_selectors_are_not_undefined_guarded(tmp_path): m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) assert "=?= undefined" not in cm.build_singularity_image_expr(m) assert "=?= undefined" not in cm.build_transfer_input_expr(m) - assert "=?= undefined" not in cm.build_container_image_select(m) + assert "=?= undefined" not in cm.build_container_image_select( + cm.load_container_manifest(_write(tmp_path, ALL_OSDF_MANIFEST, "osdf.yaml"))) def test_capability_defined_requirement(tmp_path): @@ -268,23 +285,40 @@ def test_backward_compat_single_sif(tmp_path, monkeypatch): # --------------------------------------------------------------------------- def test_container_image_select_expression(tmp_path): - m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + m = cm.load_container_manifest(_write(tmp_path, ALL_OSDF_MANIFEST)) expr = cm.build_container_image_select(m) - # a $$() match-time substitution token with VERBATIM image values (osdf URL - # fetched by container universe; cvmfs path used in place) -- NOT a ./basename - # rewrite, and NOT undefined-guarded (Requirements exclusion is used instead) + # A $$() match-time substitution token over BASENAMES. condor_submit derives the + # job ad's ContainerImage as the text after the LAST '/', *before* any $$ + # expansion, so a selector containing a path is truncated and the job holds at the + # execute point ("Unable to download or build singularity image ...sif\") ])", + # observed live on an OSPool glidein). With no '/' the token survives intact and + # the schedd expands it at match time. assert expr.startswith("$$([ ") and expr.endswith(" ])") + assert "/" not in expr # THE invariant assert "=?= undefined" not in expr # not a guess-guard assert "ifThenElse(TARGET.GPUs_Capability >= 7.0," in expr - assert '"osdf:///igwn/rift_modern_cuda12.sif"' in expr # raw osdf URL - assert '"/cvmfs/sw/rift_ancient_cuda11.sif"' in expr # fallback verbatim - assert "./rift_modern_cuda12.sif" not in expr # no basename rewrite + assert '"rift_modern_cuda12.sif"' in expr # basename branch + assert '"rift_ancient_cuda11.sif"' in expr # fallback basename + + +def test_container_image_select_rejects_in_place_images(tmp_path): + # An in-place (CVMFS/local) image can only be named by its full path, which would + # reintroduce the '/' truncation. Refuse loudly rather than emit a submit file + # that holds every job. + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + with pytest.raises(cm.ContainerManifestError) as exc: + cm.build_container_image_select(m) + assert "ancient" in str(exc.value) + # ... but the CPU-only single-image path is unaffected: it is a plain literal that + # condor_submit handles correctly. + assert cm.build_container_image_select(m, request_gpu=False) == "/cvmfs/sw/rift_ancient_cuda11.sif" def test_integration_container_universe(tmp_path, monkeypatch): - # Opt-in container-universe mode: per-machine image via $$()-substituted - # container_image; no MY.SingularityImage / BindCVMFS / $$() transfer token; - # universe=container; require_gpus floor still applied. + # Opt-in container-universe mode: per-machine image via a $$()-substituted + # container_image over BASENAMES, the matched image delivered by the comma-free + # $$() transfer token, no MY.SingularityImage / BindCVMFS, universe=container, + # require_gpus floor still applied. monkeypatch.setenv("RIFT_CONTAINER_UNIVERSE", "1") monkeypatch.delenv("RIFT_REQUIRE_GPUS", raising=False) monkeypatch.chdir(tmp_path) @@ -296,7 +330,7 @@ def test_integration_container_universe(tmp_path, monkeypatch): arg_str="--foo bar", transfer_files=["../all.net"], use_singularity=True, - singularity_image=_write(tmp_path, MIXED_MANIFEST), + singularity_image=_write(tmp_path, ALL_OSDF_MANIFEST), request_gpu=True, cache_file="local.cache", ) @@ -305,9 +339,19 @@ def test_integration_container_universe(tmp_path, monkeypatch): ci = cmds["container_image"] assert ci.startswith("$$([") # match-time substitution, unquoted assert not ci.startswith('"') + assert "/" not in ci # else condor_submit truncates it assert "MY.SingularityImage" not in cmds # the OSG-breaking attr is gone assert "MY.SingularityBindCVMFS" not in cmds - assert "$$([" not in cmds.get("transfer_input_files", "") # image via container_image, not transfer + + # container_image names only a basename, so the image must arrive by transfer: + # exactly one comma-free $$() token carrying the full URLs. + tif = cmds["transfer_input_files"] + assert tif.count("$$([") == 1 + assert "osdf:///igwn/sw/rift_modern_cuda12.sif" in tif + # ... and TransferInput is pinned, so condor_submit does not append the basename + # selector to it as a bogus extra input file. + assert cmds["MY.TransferInput"] == '"' + tif.replace('"', '\\"') + '"' + assert "Capability >= 3.0" in cmds["require_gpus"] # floor still steers GPUs # GPU family job: still excludes slots that don't advertise the capability attr assert "TARGET.GPUs_Capability =!= undefined" in cmds["requirements"] diff --git a/containers/README.md b/containers/README.md index fce584f00..1f45d1f36 100644 --- a/containers/README.md +++ b/containers/README.md @@ -188,20 +188,28 @@ for `igwn+osdf:`) by inspecting the manifest's image URLs — the single-image p keys off the `SINGULARITY_RIFT_IMAGE` string, which for a family is only a `.yaml` path. -> **Open item on the container-universe mode.** `condor_submit` parses -> `container_image` *before* any `$$` expansion and derives the job ad's -> `ContainerImage` (the name the image gets in the job scratch dir) as the text -> after the **last** `/`. A `$$([ ... ])` selection contains slashes, so that -> derivation cuts it in half. On `$CondorVersion: 25.11.1`, `condor_submit -> -dry-run` of a generated `ILE.sub` gives -> `ContainerImage="rift_container_default.sif\") ])"` while -> `ContainerImageFullPath` keeps the intact `$$` token. The full path and -> `transfer_input_files` expand correctly at match time; `ContainerImage` has no -> `$$` left, so nothing repairs it unless the schedd re-derives it after -> expansion -- unverified against a live GPU match. Check this on one real ILE -> job (`condor_q -l`) before a production OSPool campaign. -> `RIFT_CONTAINER_RUNTIME_SELECT=1` emits no container-universe attributes and is -> unaffected. +> **Why the container-universe selector names basenames, not URLs.** +> `condor_submit` parses `container_image` *before* any `$$` expansion and derives +> the job ad's `ContainerImage` -- the name the image gets in the job scratch dir -- +> as the text after the **last** `/`. A selector containing full paths is cut in +> half, and the fragment that survives is not a valid image name. Submitting that +> form to the IGWN pool holds the job at the execute point: +> `PREPARE_JOB (prepare-hook) failed: Unable to download or build singularity image +> cutest_busybox_...sif") ])`. +> +> So the selector emits **basenames only** (no `/`); the whole `$$` token survives +> into `ContainerImage` and the schedd expands it at match time +> (`MATCH_EXP_ContainerImage = "rift_container_modern.sif"`). The image itself +> arrives via the comma-free `$$()` transfer token, and `MY.TransferInput` is pinned +> so `condor_submit` does not append the basename selector to `TransferInput` as a +> bogus extra input file. Verified end to end on an OSPool glidein against +> `$CondorVersion: 25.11.1`. +> +> Consequence: **every image in a family used with container universe must be a +> transferable URL.** An in-place (CVMFS/local) image can only be named by its full +> path, which reintroduces the truncation, so `build_container_image_select()` +> raises `ContainerManifestError` for such a family. Stage those images at a URL, or +> use `RIFT_CONTAINER_RUNTIME_SELECT=1`. ### HTCondor GPU attribute names — important diff --git a/containers/rift_container_family.yaml b/containers/rift_container_family.yaml index 1a87b1a5e..fc324736f 100644 --- a/containers/rift_container_family.yaml +++ b/containers/rift_container_family.yaml @@ -34,11 +34,19 @@ capability_attr: GPUs_Capability # CPU-safe / most broadly compatible image. fallback: default +# NOTE for RIFT_CONTAINER_UNIVERSE=1 (the recommended OSG mode): every image below +# must be a transferable URL. condor_submit derives the job's ContainerImage as the +# text after the last "/" of container_image, BEFORE any $$ expansion, so the +# selector may not contain a path -- it names basenames, and the image is delivered +# by file transfer. An in-place CVMFS/local image cannot be named that way and is +# rejected with ContainerManifestError. It is still fine under the legacy and +# runtime-select modes; see containers/README.md. + containers: - # Default, broadly-compatible image for older machines. Referenced in place - # on CVMFS -- never transferred; CVMFS lazy-fetches it only when selected. + # Default, broadly-compatible image for older machines. Delivered via osdf: + # only the matched machine fetches it (selective $$() transfer). - label: default - image: /cvmfs/singularity.opensciencegrid.org/oshaughn/rift_container_default.sif + image: osdf:///igwn/staging/oshaughn/rift_containers/rift_container_default.sif cuda_capability_min: 3.5 cuda_capability_max: 8.0 note: "base=nvidia/cuda:11.8.0-runtime-ubuntu22.04, cupy-cuda11x" diff --git a/docs/source/containers.rst b/docs/source/containers.rst index 5fbb19909..7f99b0db4 100644 --- a/docs/source/containers.rst +++ b/docs/source/containers.rst @@ -124,7 +124,9 @@ For a manifest, the GPU ILE Condor submit files get: * **Selective transfer** — only ``osdf://`` images are fetched, and only on the machine that selected them, via a single HTCondor ``$$()`` match-time token appended to ``transfer_input_files``. CVMFS/local images are referenced in - place and never transferred, so the **whole family is never pulled**:: + place and never transferred, so the **whole family is never pulled**. Under + container universe this token is what actually delivers the image (the + ``container_image`` selector only names a basename — see the warning below):: $$([ (TARGET.GPUs_Capability >= 8.0 ? "osdf:///.../rift_container_modern.sif" : "") ]) @@ -210,24 +212,31 @@ Under asimov, set the variable from the blueprint rather than the shell: .. warning:: - **Open item on the container-universe mode.** ``condor_submit`` parses - ``container_image`` *before* any ``$$`` expansion, and derives the job ad's - ``ContainerImage`` (the name the image will have in the job scratch dir) as - the text after the **last** ``/``. A ``$$([ ... ])`` selection contains - slashes, so that derivation cuts the expression in half. On - ``$CondorVersion: 25.11.1``, ``condor_submit -dry-run`` of a generated - ``ILE.sub`` yields:: - - ContainerImage="rift_container_default.sif\") ])" - ContainerImageFullPath="$$([ ifThenElse(TARGET.GPUs_Capability >= 8.0, \"osdf:///...\", \"osdf:///...\") ])" - - ``ContainerImageFullPath`` and ``transfer_input_files`` keep the ``$$`` token - and expand correctly at match time; ``ContainerImage`` no longer contains a - ``$$``, so nothing repairs it unless the schedd re-derives it after expansion - — which has not been verified against a live GPU match. Confirm this on one - real ILE job (``condor_q -l`` the matched job) before running a production - OSPool campaign on this mode. ``RIFT_CONTAINER_RUNTIME_SELECT=1`` emits no - container-universe attributes at all and is not affected. + **Why the container-universe selector names basenames, not URLs.** + ``condor_submit`` parses ``container_image`` *before* any ``$$`` expansion and + derives the job ad's ``ContainerImage`` — the name the image will have in the + job scratch dir — as the text after the **last** ``/``. A selector containing + full paths is therefore cut in half, and the fragment that survives is not a + valid image name. Submitting that form to the IGWN pool holds the job at the + execute point:: + + PREPARE_JOB (prepare-hook) failed (reported status 001): + Unable to download or build singularity image cutest_busybox_...sif") ]) + + So the selector emits **basenames only** (no ``/``), the whole ``$$`` token + survives into ``ContainerImage``, and the schedd expands it at match time + (``MATCH_EXP_ContainerImage = "rift_container_modern.sif"``). The image itself + is delivered by the comma-free ``$$()`` transfer token, and ``MY.TransferInput`` + is pinned so ``condor_submit`` does not append the basename selector to + ``TransferInput`` as a bogus extra input file. Verified end to end on an OSPool + glidein against ``$CondorVersion: 25.11.1``. + + Consequence: **every image in a family used with container universe must be a + transferable URL.** An image referenced in place (a CVMFS or local path) can + only be named by its full path, which would reintroduce the truncation, so + :func:`~RIFT.misc.container_manifest.build_container_image_select` raises + ``ContainerManifestError`` for such a family. Stage those images at a URL, or + use ``RIFT_CONTAINER_RUNTIME_SELECT=1``. GPU attribute names @@ -254,9 +263,10 @@ Two different ClassAd namespaces are involved, and they are kept separate: attribute names, the ``require_gpus`` floor (matching a compatible GPU and excluding an incompatible one), the ``$$()`` match-time image selection, and tolerance of the empty-result case for a manifest that mixes CVMFS and - ``osdf`` entries. The expression-valued ``MY.SingularityImage`` is *not* - OSPool-safe (see :ref:`osg-container-modes`); use - ``RIFT_CONTAINER_UNIVERSE=1`` there. + ``osdf`` entries. The container-universe mode is additionally verified end to + end on an OSPool glidein (image selected, fetched and entered; job exit 0). The + expression-valued ``MY.SingularityImage`` is *not* OSPool-safe (see + :ref:`osg-container-modes`); use ``RIFT_CONTAINER_UNIVERSE=1`` there. Building a container family From e0a0cd037022f4129b45dfbe5479c0da06a28572 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Mon, 10 Aug 2026 02:39:41 -0700 Subject: [PATCH 4/4] containers/README: correct the delivery-mode table row for the basename selector The row still said container universe delivers the image via container_image with no transfer token. It does not: the selector names basenames, so the matched image arrives via the $$() transfer token with MY.TransferInput pinned, and every image in the family must be a transferable URL. (An earlier edit missed this line -- it matched on '--' where the file has an em dash.) Co-Authored-By: Claude Opus 5 --- containers/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/containers/README.md b/containers/README.md index 1f45d1f36..e623d3d34 100644 --- a/containers/README.md +++ b/containers/README.md @@ -169,7 +169,7 @@ selected by an environment variable at DAG-build time: | env var | behaviour | |---|---| | *(unset)* | legacy `universe = vanilla` + expression-valued `MY.SingularityImage`. Correct on a local/CIT pool; **not OSG-safe**. | -| `RIFT_CONTAINER_UNIVERSE=1` | **recommended for OSG.** `universe = container` + `container_image = $$([ ifThenElse(...) ])`. `$$()` is HTCondor's match-time (schedd-side) machine-ad substitution, so the pilot only ever sees a literal URL. No `MY.SingularityImage`, no `MY.SingularityBindCVMFS`, no `$$()` transfer token — the image arrives via `container_image`. GPU access is automatic under `request_gpus`. Works on CIT-local too. | +| `RIFT_CONTAINER_UNIVERSE=1` | **recommended for OSG.** `universe = container` + `container_image = $$([ ifThenElse(...) ])` over image BASENAMES. `$$()` is HTCondor's match-time (schedd-side) machine-ad substitution, so the pilot only ever sees a literal image name. No `MY.SingularityImage`, no `MY.SingularityBindCVMFS`; the matched image arrives via the `$$()` transfer token with `MY.TransferInput` pinned (see below). Requires every family image to be a transferable URL. GPU access is automatic under `request_gpus`. Works on CIT-local too. | | `RIFT_CONTAINER_RUNTIME_SELECT=1` | older ILE-only fallback: Condor runs a generated `rift_container_select.sh` on the bare node, which reads the real capability from `nvidia-smi`, fetches only the matching image (`stashcp`/`pelican`) and re-execs under `apptainer exec --nv`. | Under asimov set it from the blueprint, not the shell: