From b3d77518a6c54011c518b940ea1e4b17830b742f Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Thu, 11 Jun 2026 08:41:39 -0700 Subject: [PATCH 01/63] container family: OSG-safe per-machine image via container universe (opt-in) ALTERNATIVE to the runtime-wrapper approach (branch rift_O4d_osg_runtime_container_select): use HTCondor's container universe with container_image = $$([...]) instead of MY.SingularityImage = ifThenElse(...). Why it works on OSG: MY.SingularityImage=ifThenElse(...) is an execute-side ClassAd expression that OSPool glidein pilots read as a LITERAL string and hold the job on. container_image with a $$() token is resolved by HTCondor via match-time machine-ad substitution (in the schedd, against the matched machine ad) BEFORE the job reaches the EP, so the pilot only ever sees a literal image URL. $$ in container_image is HTCondor's *documented* mechanism for selecting a container image by GPU CUDA capability, and container universe is the current OSPool-standard (it deprecated +SingularityImage); osdf:// container images are supported and OSDF-cached; GPU access is automatic under request_gpus (no --nv needed). The same path also works on the CIT-local pool, so this unifies both pools (vs the ifThenElse path which is CIT-local-only). - container_manifest.build_container_image_select(manifest): returns the $$([ ifThenElse(attr =?= undefined, , ) ]) value. Image branches are the manifest images VERBATIM (osdf URL fetched by container universe, or cvmfs/local path in place) -- not a ./basename rewrite. The =?= undefined guard makes a CPU-only / non-advertising slot fall to the fallback image instead of an undefined $$() that would hold the job. - write_ILE_sub_simple: when RIFT_CONTAINER_UNIVERSE is set (and a family manifest + use_singularity), set universe=container, emit container_image = the $$() selector, and drop MY.SingularityImage / MY.SingularityBindCVMFS / the $$() transfer token (container universe transfers the image itself). The require_gpus floor is still applied. Default (env unset) behavior is unchanged: the existing ifThenElse MY.SingularityImage path for CIT-local runs. Tests: container_image select expression (undefined-safe, verbatim osdf URLs, fallback) and integration (universe=container, container_image=$$([...]), no MY.SingularityImage / no transfer token, floor present). Existing CIT-local and single-sif tests unchanged. Trade-off vs the wrapper branch: this is much smaller and uses native/documented HTCondor machinery, but relies on the matched slot advertising the capability attribute at match time; the wrapper detects the real GPU at job start instead. ILE-only for now (CIP/PSD/calibration still use the ifThenElse path). Open item to confirm on a real OSG GPU job: cvmfs bind + capability advertisement coverage across OSPool sites. Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/container_manifest.py | 337 ++ .../Code/RIFT/misc/dag_utils_generic.py | 4790 +++++++++++++++++ .../Code/test/test_container_manifest.py | 259 + 3 files changed, 5386 insertions(+) create mode 100644 MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py create mode 100644 MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py create mode 100644 MonteCarloMarginalizeCode/Code/test/test_container_manifest.py diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py new file mode 100644 index 000000000..f3cda0f00 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py @@ -0,0 +1,337 @@ +""" +container_manifest +================== + +Support for "container family" manifests used by the RIFT pipeline. + +Historically ``SINGULARITY_RIFT_IMAGE`` names a single ``.sif`` image (a local +path or an ``osdf://`` URL), and the ILE/CIP Condor jobs hard-code + + MY.SingularityImage = "" + +A *manifest* lets us instead advertise a *family* of images, each targeting a +different GPU compute capability, and let HTCondor pick the right one per matched +machine. When ``SINGULARITY_RIFT_IMAGE`` points at a ``.yaml``/``.yml`` file, +the job-submission code turns it into: + + * an expression-valued ``MY.SingularityImage`` -- a nested ``ifThenElse`` over + the matched machine's GPU capability attribute (default ``GPUs_Capability``) + that selects the highest-capability image the machine can run; and + * a ``require_gpus`` capability floor (the lowest capability any image in the + family supports), composed (``&&``) with any user-supplied + ``RIFT_REQUIRE_GPUS``; and + * for ``osdf://`` images, a *selective* ``transfer_input_files`` entry using + HTCondor ``$$()`` match-time substitution, so only the *matched* image is + transferred (CVMFS/local images are referenced in place and never + transferred). + +Single-``.sif`` behavior is completely unchanged: only ``.yaml``/``.yml`` values +exercise any of this. + +YAML schema +----------- + + version: 1 + capability_attr: GPUs_Capability # machine ClassAd attr the ifThenElse tests + fallback: ancient # label used as the innermost else-branch + containers: + - label: ancient + image: /cvmfs/.../rift_ancient_cuda11.sif # in-place (CVMFS/local) + cuda_capability_min: 3.0 # inclusive + cuda_capability_max: 7.0 # exclusive; null/omitted => open-ended + note: "cupy-cuda11x, ancient base" + - label: modern + image: osdf:///igwn/.../rift_modern_cuda12.sif # selectively transferred + cuda_capability_min: 7.0 + cuda_capability_max: null + note: "cupy-cuda12x, newer base" +""" + +import os + +__all__ = [ + "ContainerManifestError", + "is_container_manifest", + "load_container_manifest", + "build_singularity_image_expr", + "build_transfer_input_expr", + "build_require_gpus_floor", + "build_container_image_select", +] + +# Default machine ClassAd attribute advertising GPU compute capability. The +# user's pools advertise this via e.g. +# condor_status -constraint 'TotalGPUs > 0' -autoformat GPUs_DeviceName GPUs_Capability +DEFAULT_CAPABILITY_ATTR = "GPUs_Capability" + + +class ContainerManifestError(Exception): + """Raised for a missing/malformed container family manifest.""" + + +def is_container_manifest(value): + """Return True iff ``value`` (the ``SINGULARITY_RIFT_IMAGE`` string) names a + multi-container manifest rather than a single ``.sif``/``osdf://`` image. + + Pure string check (no filesystem access) so single-image callers pay zero + cost and their behavior is unchanged. + """ + if not value or not isinstance(value, str): + return False + return value.lower().endswith((".yaml", ".yml")) + + +def _image_needs_transfer(image): + """True iff ``image`` is a URL that must be fetched via Condor file transfer + (e.g. ``osdf://``). CVMFS/local paths (``/cvmfs/...``, ``./foo.sif``) are + resolved in place and return False. + """ + return "://" in image + + +def _image_runtime_path(image): + """The string used *inside* ``MY.SingularityImage`` for this image. + + Transferred (URL) images land in the job scratch dir under their basename, + so the pilot must reference ``./`` -- matching the existing + single-image osdf rewrite convention. In-place (CVMFS/local) images are + referenced verbatim. + """ + if _image_needs_transfer(image): + return "./{}".format(image.rstrip("/").split("/")[-1]) + return image + + +def _fmt_cap(value): + """Format a capability number for a ClassAd expression (e.g. 7.0 -> '7.0').""" + return repr(float(value)) + + +def load_container_manifest(path): + """Parse and validate a YAML container family manifest. + + Returns a dict ``{capability_attr, fallback, containers}`` where + ``containers`` is sorted by ``cuda_capability_min`` *descending* (containers + with no min sort last). + + Raises ``ContainerManifestError`` on a missing pyyaml, an unreadable or + malformed file, an empty container list, or an unknown ``fallback`` label. + """ + try: + import yaml + except ImportError as exc: # pragma: no cover - environment dependent + raise ContainerManifestError( + "PyYAML is required to read a container family manifest ({}); " + "install pyyaml or point SINGULARITY_RIFT_IMAGE at a single .sif".format(path) + ) from exc + + try: + with open(path, "r") as f: + data = yaml.safe_load(f) + except (IOError, OSError) as exc: + raise ContainerManifestError("Cannot read container manifest {}: {}".format(path, exc)) + except yaml.YAMLError as exc: + raise ContainerManifestError("Malformed container manifest {}: {}".format(path, exc)) + + if not isinstance(data, dict): + raise ContainerManifestError("Container manifest {} is not a mapping".format(path)) + + raw_containers = data.get("containers") + if not raw_containers or not isinstance(raw_containers, list): + raise ContainerManifestError( + "Container manifest {} must define a non-empty 'containers' list".format(path) + ) + + containers = [] + for idx, entry in enumerate(raw_containers): + if not isinstance(entry, dict): + raise ContainerManifestError( + "Container manifest {} entry #{} is not a mapping".format(path, idx) + ) + image = entry.get("image") + label = entry.get("label") + if not image: + raise ContainerManifestError( + "Container manifest {} entry #{} is missing 'image'".format(path, idx) + ) + if not label: + raise ContainerManifestError( + "Container manifest {} entry #{} is missing 'label'".format(path, idx) + ) + cap_min = entry.get("cuda_capability_min") + cap_max = entry.get("cuda_capability_max") + try: + cap_min = None if cap_min is None else float(cap_min) + cap_max = None if cap_max is None else float(cap_max) + except (TypeError, ValueError): + raise ContainerManifestError( + "Container manifest {} entry '{}' has non-numeric capability bounds".format( + path, label + ) + ) + containers.append( + { + "label": label, + "image": image, + "cuda_capability_min": cap_min, + "cuda_capability_max": cap_max, + "note": entry.get("note"), + } + ) + + # Sort by min capability descending; None mins (open-ended-low catch-alls) + # sort last. float('-inf') keeps them at the bottom. + containers.sort( + key=lambda c: (c["cuda_capability_min"] if c["cuda_capability_min"] is not None else float("-inf")), + reverse=True, + ) + + labels = {c["label"] for c in containers} + fallback = data.get("fallback") + if fallback is None: + # Default fallback = the most-compatible (lowest-min) container, i.e. the + # last one after the descending sort. This is the CPU-safe catch-all. + fallback = containers[-1]["label"] + elif fallback not in labels: + raise ContainerManifestError( + "Container manifest {} fallback '{}' is not one of {}".format( + path, fallback, sorted(labels) + ) + ) + + capability_attr = data.get("capability_attr") or DEFAULT_CAPABILITY_ATTR + + return { + "capability_attr": capability_attr, + "fallback": fallback, + "containers": containers, + } + + +def _capability_attr(manifest): + """Resolve the machine attribute used by the selection ifThenElse. + + Precedence: ``RIFT_GPU_CAPABILITY_ATTR`` env override > manifest + ``capability_attr`` > module default. + """ + return os.environ.get("RIFT_GPU_CAPABILITY_ATTR") or manifest["capability_attr"] + + +def _build_selector(manifest, value_fn, ternary=False): + """Build a nested capability selector over the family. + + ``value_fn(container)`` returns the ClassAd literal for a container branch + (already quoted as appropriate). The highest-min container is the outermost + test; the ``fallback`` container is the innermost else (catch-all, also used + when the capability attribute is ``undefined``). + + With ``ternary=False`` the selector uses ``ifThenElse(cond, a, b)`` (commas). + With ``ternary=True`` it uses the comma-free ClassAd ternary ``cond ? a : b`` + -- required when the result is embedded as one element of a comma-separated + ``transfer_input_files`` list, where internal commas would be mis-split. + """ + attr = _capability_attr(manifest) + containers = manifest["containers"] # sorted desc by min + by_label = {c["label"]: c for c in containers} + fb = by_label[manifest["fallback"]] + + # Containers that contribute a capability threshold test (exclude the + # fallback so it is not duplicated as both a branch and the else). + thresholds = [ + c + for c in containers + if c["cuda_capability_min"] is not None and c["label"] != fb["label"] + ] + # Fold ascending so the highest min ends up outermost. + thresholds.sort(key=lambda c: c["cuda_capability_min"]) + + expr = value_fn(fb) + for c in thresholds: + cond = "TARGET.{attr} >= {mn}".format(attr=attr, mn=_fmt_cap(c["cuda_capability_min"])) + if ternary: + expr = "({cond} ? {val} : {inner})".format(cond=cond, val=value_fn(c), inner=expr) + else: + expr = "ifThenElse({cond}, {val}, {inner})".format( + cond=cond, val=value_fn(c), inner=expr + ) + return expr + + +def build_singularity_image_expr(manifest): + """Return the unquoted ClassAd expression for ``MY.SingularityImage``. + + Each branch literal is the container's *runtime* path (CVMFS/local verbatim, + ``./`` for transferred images). + """ + return _build_selector( + manifest, lambda c: '"{}"'.format(_image_runtime_path(c["image"])) + ) + + +def build_transfer_input_expr(manifest): + """Return a single ``$$([ ... ])`` token for ``transfer_input_files`` that + fetches *only the matched* image, or ``None`` if no container in the family + needs transfer. + + Transfer branches yield the URL verbatim; in-place (CVMFS/local) branches + yield ``""`` (no transfer on those machines). Uses the comma-free ternary + form so the token survives comma-splitting of ``transfer_input_files``. + """ + if not any(_image_needs_transfer(c["image"]) for c in manifest["containers"]): + return None + + def value_fn(c): + return '"{}"'.format(c["image"]) if _image_needs_transfer(c["image"]) else '""' + + return "$$([ {} ])".format(_build_selector(manifest, value_fn, ternary=True)) + + +def build_container_image_select(manifest): + """Return an unquoted ``$$([ ... ])`` value for the HTCondor *container + universe* ``container_image`` submit command, selecting the per-machine image. + + Unlike :func:`build_singularity_image_expr` (an execute-side ClassAd + expression that OSPool glidein pilots read as a literal string and hold the + job on), this uses HTCondor ``$$()`` *match-time machine-ad substitution*: the + schedd evaluates the bracketed expression against the matched machine ad and + substitutes a literal image string into ``container_image`` before the job + reaches the execution point. ``$$`` in ``container_image`` is HTCondor's + documented mechanism for per-GPU-capability image selection, and it works on + both the CIT-local pool and OSPool glideins. + + The branch value is the manifest image *verbatim* -- an ``osdf://`` URL that + container universe's file-transfer plugin fetches, or a CVMFS/local path used + in place -- NOT a ``./basename`` rewrite (container universe handles the image + itself). ``container_image`` is a single submit command (not a comma list), + so the comma-bearing ``ifThenElse`` form is fine here. + + The expression is undefined-safe: if the matched machine does not advertise + the capability attribute (e.g. a CPU-only slot), it yields the ``fallback`` + image instead of an undefined ``$$()`` that would hold the job. + """ + attr = _capability_attr(manifest) + by_label = {c["label"]: c for c in manifest["containers"]} + fb_image = by_label[manifest["fallback"]]["image"] + selector = _build_selector(manifest, lambda c: '"{}"'.format(c["image"])) + guarded = 'ifThenElse(TARGET.{attr} =?= undefined, "{fb}", {sel})'.format( + attr=attr, fb=fb_image, sel=selector + ) + return "$$([ {} ])".format(guarded) + + +def build_require_gpus_floor(manifest): + """Return a ``require_gpus`` capability floor expression for the family, or + ``None``. + + The floor is the lowest ``cuda_capability_min`` across the family -- i.e. do + not match a GPU less capable than anything we ship. Uses the require_gpus + sub-ad attribute ``Capability`` (unprefixed -- *not* ``TARGET.`` and *not* + ``GPUs_Capability``). + + If any container has no min (open-ended-low catch-all), there is effectively + no lower bound and ``None`` is returned. + """ + mins = [c["cuda_capability_min"] for c in manifest["containers"]] + if any(m is None for m in mins) or not mins: + return None + return "Capability >= {}".format(_fmt_cap(min(mins))) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py new file mode 100644 index 000000000..d602cc722 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py @@ -0,0 +1,4790 @@ +# Copyright (C) 2013 Evan Ochsner +# +# This program is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the +# Free Software Foundation; either version 2 of the License, or (at your +# option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General +# Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +""" +Backend-neutral workflow / DAG utilities for RIFT. + +This module is a re-implementation of ``RIFT.misc.dag_utils`` whose original +form was tied directly to ``glue.pipeline`` (a piece of LIGO/glue infrastructure +that is fragile to install and not always available -- particularly on macOS). + +Architecture +============ + +The module separates three concerns: + +1. **A backend-neutral workflow data model** -- :class:`_GenericJob`, + :class:`_GenericNode`, :class:`_GenericDAG`, :class:`_GenericManJob`. These + are *plain data containers*: they record the executable, arguments, + resource requests, environment, files, queue count, dependencies, etc. + They expose the historical ``glue.pipeline.CondorDAGJob`` API + (``add_opt``, ``add_arg``, ``add_condor_cmd``, ``set_sub_file``, + ``write_sub_file``, ...) as a *facade* on top of that data model, so + existing RIFT code keeps working. The underlying state is fully generic; + nothing about it presupposes Condor. + +2. **A pluggable backend layer** -- :class:`WorkflowBackend` is an abstract + base class. Each backend implements ``emit_job(job, path)`` (turn a + :class:`_GenericJob` into a per-system submit description) and + ``emit_dag(dag, path)`` (turn a :class:`_GenericDAG` into a per-system + workflow driver). Three backends ship with this module: + + - :class:`HTCondorBackend` -- modern HTCondor python bindings + (``htcondor.Submit``). Emits ``.sub`` files and a HTCondor DAGMan + ``.dag`` file. + - :class:`GluePipelineBackend` -- legacy ``glue.pipeline``-based fallback, + useful where the htcondor python bindings are not available (e.g. some + macOS installs). Translates the generic job spec into ``glue.pipeline`` + calls at emit time. + - :class:`SlurmBackend` -- emits ``.sbatch`` scripts plus a shell driver + script that submits the jobs in dependency order (using + ``sbatch --dependency=afterok:JOBID``). + +3. **A backend registry** -- :func:`register_backend`, :func:`set_backend`, + :func:`get_backend`. Backends are looked up by name. New backends can be + registered by user code without touching this module. + +Backend selection +================= + +At import time this module performs an *auto-detection*: it tries the +HTCondor python bindings first, then ``glue.pipeline``. Slurm is never +auto-selected because no reliable purely-pythonic test for "we are on a Slurm +cluster" exists; pick it explicitly. + +The active backend can be controlled in three ways, in order of precedence: + +1. Calling :func:`set_backend("name")` from python. +2. The ``RIFT_DAG_BACKEND`` environment variable. Recognised values: + ``htcondor``, ``glue``, ``slurm``, ``auto`` (default). +3. Auto-detection (above). + +Custom backends +=============== + +To add a new execution-system backend (e.g. PBS/Torque, LSF, Kubernetes, ...):: + + from RIFT.misc.dag_utils_generic import ( + WorkflowBackend, register_backend, set_backend + ) + + class PBSBackend(WorkflowBackend): + name = "pbs" + + def emit_job(self, job, path): + ... + + def emit_dag(self, dag, path): + ... + + register_backend(PBSBackend()) + set_backend("pbs") + +Public API +========== + +For consumers of this module, the public surface is: + +- The factories ``CondorDAGJob``, ``CondorDAG``, ``CondorDAGNode``, + ``CondorDAGManJob`` (named for historical reasons; they return the + backend-neutral wrappers). A ``pipeline`` namespace exposes the same four + factories so code that previously did ``from glue import pipeline`` can do + ``from RIFT.misc.dag_utils_generic import pipeline`` instead. +- All the ``write_*_sub`` helpers (``write_CIP_sub``, ``write_ILE_sub_simple``, + ``write_consolidate_sub_simple``, ...) ported verbatim from the original + ``dag_utils.py``. Because they are written against the facade API, they + work under any backend. +- The utility helpers ``which``, ``mkdir``, ``quote_arguments``, + ``safely_quote_arg_str``, ``bilby_ish_string_to_dict``, + ``build_resolved_env``. +""" + +import os +import re +import sys +import abc +import shlex +from time import time +from hashlib import md5 + +import shutil + +import numpy as np +import configparser + +# Container family manifest support (Feature: multi-architecture container +# deployment). Import-guarded so that plain single-.sif runs never require +# pyyaml; only an actual .yaml/.yml manifest exercises this path. +try: + from RIFT.misc.container_manifest import ( + is_container_manifest, + load_container_manifest, + build_singularity_image_expr, + build_transfer_input_expr, + build_require_gpus_floor, + build_container_image_select, + ContainerManifestError, + ) + _HAVE_CONTAINER_MANIFEST = True +except ImportError: + _HAVE_CONTAINER_MANIFEST = False + + def is_container_manifest(value): + # Without the helper module available, never treat a value as a manifest. + return False + +__author__ = ( + "Evan Ochsner , " + "Chris Pankow " +) + + +# =========================================================================== +# Utility helpers +# =========================================================================== + +# getenv=True deprecated, will need workaround to explicitly pull extra environment variables +default_getenv_value = 'True' +default_getenv_osg_value = 'True' +if 'RIFT_GETENV' in os.environ: + default_getenv_value = os.environ['RIFT_GETENV'] +if 'RIFT_GETENV_OSG' in os.environ: + default_getenv_osg_value = os.environ['RIFT_GETENV_OSG'] + + +def is_exe(fpath): + return os.path.exists(fpath) and os.access(fpath, os.X_OK) + + +def which(program): + fpath, fname = os.path.split(program) + if fpath: + if is_exe(program): + return program + else: + for path in os.environ["PATH"].split(os.pathsep): + exe_file = os.path.join(path, program) + if is_exe(exe_file): + return exe_file + return None + + +def mkdir(dir_name): + try: + os.mkdir(dir_name) + except OSError: + pass + + +def generate_job_id(): + """Generate a unique md5 hash for use as a job ID.""" + t = str(int(time() * 1000)) + r = str(int(np.random.random() * 100000000000000000)) + return md5((t + r).encode("utf-8")).hexdigest() + + +def _double_up_quotes(instr): + return instr.replace("'", "''").replace('"', '""') + + +def quote_arguments(args): + """Quote a string or list of strings using the Condor submit-file 'new' argument-quoting rules.""" + if isinstance(args, str): + args_list = [args] + else: + args_list = args + quoted_args = [] + for a in args_list: + qa = _double_up_quotes(a) + if " " in qa or "'" in qa: + qa = "'" + qa + "'" + quoted_args.append(qa) + return " ".join(quoted_args) + + +def safely_quote_arg_str(arg_str): + if not ('"' in arg_str): + return quote_arguments + quote_breaks = arg_str.split('"') + if len(quote_breaks) != 3: + raise Exception(" Arg parsing: multiple quoted argument strings provided, not ready to handle ") + args0 = quote_arguments(quote_breaks[0].split()) + args2 = quote_arguments(quote_breaks[2].split()) + args1 = quote_arguments('"{}"'.format(quote_breaks[1])) + return "{} {} {}".format(args0, args1, args2) + + +def bilby_ish_string_to_dict(my_str): + items = my_str.replace('{', '').replace('}', '').strip().split(',') + items = [x for x in items if len(x) > 0] + pseudo_dict = {} + for item in items: + key, val = item.split(':') + pseudo_dict[key] = val + return pseudo_dict + + +def match_expr(my_list, my_expr): + list_out = [] + p = re.compile(my_expr.replace('*', '.*')) + for name in my_list: + if p.match(name): + list_out.append(name) + print(" RESOLVE: for {} found ".format(my_expr), list_out) + return list_out + + +def build_resolved_env(my_str): + env_dict = os.environ + str_out = "" + for pat in my_str.split(','): + print(" RESOLVE: building env for ", pat) + if ('*' in pat): + list_out = match_expr(list(env_dict.keys()), pat) + else: + list_out = [pat] if pat in env_dict else [] + for name in list_out: + str_out += (" {}={} ".format(name, env_dict[name])) + return str_out + + +default_resolved_env = None +default_resolved_osg_env = None +if 'RIFT_GETENV_RESOLVE' in os.environ: + default_resolved_env = build_resolved_env(default_getenv_value) + default_resolved_osg_env = build_resolved_env(default_getenv_osg_value) + + +# =========================================================================== +# Backend-neutral workflow data model +# =========================================================================== +# +# These classes are plain data containers. They expose the historical +# ``glue.pipeline.CondorDAGJob`` API as a facade so that existing RIFT code +# continues to work, but the underlying state is fully generic and contains +# no Condor-specific encoding. At ``write_sub_file()`` / ``write_concrete_dag()`` +# time, the active :class:`WorkflowBackend` is asked to translate this state +# into the appropriate per-system artefact (Condor submit file, sbatch script, +# ...). +# =========================================================================== + + +# Set of HTCondor "request_*" / submit-file commands that map to semantic +# resource concepts shared by most batch systems. When ``add_condor_cmd`` is +# called with one of these, we *also* record it as a structured resource so +# non-Condor backends can use it. +_RESOURCE_KEYS = { + "request_memory": "memory", + "request_disk": "disk", + "request_cpus": "cpus", + "request_gpus": "gpus", + "request_GPUs": "gpus", + "+MaxRunTimeMinutes": "runtime_minutes", + "MY.MaxRunTimeMinutes": "runtime_minutes", + "max_runtime_minutes": "runtime_minutes", +} + + +class _GenericJob(object): + """Backend-neutral description of a single job/task. + + Exposes the legacy ``glue.pipeline.CondorDAGJob`` API as a facade. All + ``add_*`` / ``set_*`` calls just record state; the actual on-disk artefact + is produced by the active :class:`WorkflowBackend` when + :meth:`write_sub_file` is called. + """ + + def __init__(self, universe="vanilla", executable=None): + # Generic, backend-neutral state + self.universe = universe # "vanilla", "local", ... -- backends interpret per-system + self.executable = executable + self.opts = [] # list of (name, value-or-None) for --opts + self.short_opts = [] # list of (name, value) for -opts + self.file_opts = [] # list of (name, value) for file--opts + self.var_opts = [] # variable opts (filled per-node from macros) + self.arguments = [] # positional arguments + self.environment = {} # explicit env: dict of name -> value + self.inherit_environment = False # condor's "getenv = True" / slurm's "--export=ALL" + self.condor_cmds = [] # raw Condor submit-file commands (escape hatch / passthrough) + self.resources = {} # structured: memory, disk, cpus, gpus, runtime_minutes + # Files + self.sub_file = None + self.log_file = None + self.stdout_file = None + self.stderr_file = None + # Replicas (Condor "queue N", Slurm "--array") + self.queue_count = 1 + + # ------------------------------------------------------------------ + # Compatibility shims: legacy "private" attribute access + # ------------------------------------------------------------------ + @property + def _CondorJob__queue(self): + return self.queue_count + + @_CondorJob__queue.setter + def _CondorJob__queue(self, value): + self.queue_count = int(value) + + @property + def _CondorJob__arguments(self): + return self.arguments + + @_CondorJob__arguments.setter + def _CondorJob__arguments(self, value): + self.arguments = list(value) + + # ------------------------------------------------------------------ + # Facade API (legacy CondorDAGJob method names) + # ------------------------------------------------------------------ + def set_universe(self, universe): + self.universe = universe + + def set_executable(self, executable): + self.executable = executable + + def set_sub_file(self, fname): + self.sub_file = fname + + def get_sub_file(self): + return self.sub_file + + def set_log_file(self, fname): + self.log_file = fname + + def set_stdout_file(self, fname): + self.stdout_file = fname + + def set_stderr_file(self, fname): + self.stderr_file = fname + + def add_opt(self, name, value=None): + self.opts.append((name, value)) + + def add_short_opt(self, name, value): + self.short_opts.append((name, value)) + + def add_var_opt(self, name): + self.var_opts.append(name) + + def add_file_opt(self, name, value): + self.file_opts.append((name, value)) + + def add_arg(self, arg): + self.arguments.append(arg) + + def add_condor_cmd(self, key, value): + """Record a HTCondor submit-file command. + + For commands that map to a *semantic* resource concept + (``request_memory``, ``request_disk``, ``request_cpus``, + ``request_gpus``, runtime limits) we also store the value as a + structured :attr:`resources` entry so non-Condor backends can use it. + Commands that have no portable equivalent (``MY.flock_local``, + ``+SingularityImage``, ...) are stored verbatim and only Condor-style + backends will emit them; other backends may emit them as comments. + """ + # Capture a few well-known semantic commands + if key == "getenv": + self.inherit_environment = (str(value).strip().lower() in ("true", "1", "yes")) + elif key == "environment": + # Condor environment string: a flat "K=V K=V" sequence (possibly quoted). + self._merge_environment_string(value) + else: + mapped = _RESOURCE_KEYS.get(key) + if mapped is not None: + self.resources[mapped] = value + self.condor_cmds.append((key, value)) + + def _merge_environment_string(self, value): + """Best-effort parse of a Condor 'environment = ...' string.""" + s = str(value).strip() + # Strip surrounding quotes if present + if len(s) >= 2 and s[0] == s[-1] and s[0] in ("'", '"'): + s = s[1:-1] + try: + tokens = shlex.split(s) + except ValueError: + tokens = s.split() + for tok in tokens: + if "=" in tok: + k, v = tok.split("=", 1) + self.environment[k] = v + + # ------------------------------------------------------------------ + # Emission + # ------------------------------------------------------------------ + def write_sub_file(self): + """Hand the job to the active backend, which writes the submit artefact.""" + if self.sub_file is None: + raise RuntimeError("write_sub_file: no sub-file path set") + get_backend().emit_job(self, self.sub_file) + + # Useful for tests / debugging + def to_dict(self): + return { + "universe": self.universe, + "executable": self.executable, + "opts": list(self.opts), + "short_opts": list(self.short_opts), + "file_opts": list(self.file_opts), + "var_opts": list(self.var_opts), + "arguments": list(self.arguments), + "environment": dict(self.environment), + "inherit_environment": self.inherit_environment, + "condor_cmds": list(self.condor_cmds), + "resources": dict(self.resources), + "sub_file": self.sub_file, + "log_file": self.log_file, + "stdout_file": self.stdout_file, + "stderr_file": self.stderr_file, + "queue_count": self.queue_count, + } + + +def _make_unique_node_name(prefix): + """Return a globally-unique node name of the form ``-``. + + Mirrors ``glue.pipeline.CondorDAGNode``: the md5 hash is generated from + ``time() * 1000`` plus a random integer, so two workflows constructed in + different processes (or the same process) cannot collide accidentally + when their ``.dag`` files are merged. ``generate_job_id()`` is exposed + as the helper that produces the hash. + """ + safe_prefix = re.sub(r"\W+", "_", prefix or "job") or "job" + return "{}-{}".format(safe_prefix, generate_job_id()) + + +class _GenericNode(object): + """A node in a workflow DAG. Bound to a :class:`_GenericJob`. + + Each node carries a globally-unique ``name`` (and equivalent + ``_CondorDAGNode__md5name``) so multiple workflows can be combined + without identifier collisions. The legacy ``_CondorDAGNode__md5name`` + private attribute is preserved so external code that reaches for that + name (e.g. when wiring up ``SCRIPT POST`` items by hand) keeps working. + """ + + def __init__(self, job): + self.job = job + self.macros = {} + self.category = None + self.retry = 0 + self.parents = [] + prefix = os.path.basename(getattr(job, "executable", None) or "") or "job" + self.name = _make_unique_node_name(prefix) + # glue.pipeline.CondorDAGNode stores its globally-unique id under + # ``self.__md5name`` (which Python name-mangles to + # ``_CondorDAGNode__md5name``). Mirror that exact attribute name so + # consumer code that hard-codes it keeps working. + self._CondorDAGNode__md5name = self.name + + # Compatibility names from glue.pipeline + def get_name(self): + return self.name + + def set_name(self, name): + """Override the node name (e.g. for legibility). Keeps the legacy + ``_CondorDAGNode__md5name`` attribute in sync.""" + self.name = name + self._CondorDAGNode__md5name = name + + def add_macro(self, key, value): + self.macros[str(key)] = value + + def set_category(self, category): + self.category = category + + def set_retry(self, n): + self.retry = int(n) + + def add_parent(self, parent_node): + self.parents.append(parent_node) + + +class _GenericSubdagNode(_GenericNode): + """A node that represents an external sub-DAG (Condor: ``SUBDAG EXTERNAL``).""" + + def __init__(self, subdag_file): + # Don't go through _GenericNode.__init__ -- there's no underlying job. + self.job = None + self.macros = {} + self.category = None + self.retry = 0 + self.parents = [] + self.subdag_file = subdag_file + self.name = _make_unique_node_name("subdag") + self._CondorDAGNode__md5name = self.name + + +class _GenericManJob(object): + """Wrapper for an external sub-DAG, the moral equivalent of + ``glue.pipeline.CondorDAGManJob``.""" + + def __init__(self, dag_file): + self.dag_file = dag_file + + def create_node(self): + return _GenericSubdagNode(self.dag_file) + + +class _GenericDAG(object): + """Backend-neutral workflow / DAG container. + + In addition to the basic node graph, the DAG carries the *control-logic* + overlays that HTCondor's DAGMan supports natively: + + * **per-node pre/post scripts** -- analogous to ``SCRIPT PRE`` / + ``SCRIPT POST`` directives. These run on the submit node before / + after the corresponding job. + * **abort-on hooks** -- analogous to ``ABORT-DAG-ON``: if the named + node returns the given exit code, the DAG aborts (optionally with + a specific success/failure return value). + * **dot visualisation file** -- ``DOT`` directive. + * **escape-hatch raw directives** -- arbitrary backend-native lines + appended verbatim by the active backend. + + Each backend is responsible for translating these into its native form: + HTCondor / glue.pipeline emit DAGMan directives unchanged; the Slurm + backend translates pre/post into pre-sbatch shell stanzas and + afterany-dependency wrapper jobs respectively, and emits the rest as + documented comments where there's no direct equivalent. + + The control-logic accessors can be called either *before* + :meth:`write_concrete_dag` (in which case they're emitted alongside + the rest of the DAG) or *after* (in which case the active backend is + asked to append them to the artefact it already produced -- this + matches the legacy pattern of ``open(dag_file, "a").write(...)`` that + older RIFT bin/ scripts use). + """ + + def __init__(self, log=None): + self.log = log + self.dag_file = None + self.nodes = [] + # Control-logic overlays + self.script_pre = [] # list of (node_or_name, exe, args_str) + self.script_post = [] # list of (node_or_name, exe, args_str) + self.abort_on = [] # list of (node_or_name, exit_code, return_value) + self.dot_file = None + self.extra_directives = [] # list of raw lines (escape hatch) + self._already_written = False + + def set_dag_file(self, name): + self.dag_file = name + + def add_node(self, node): + self.nodes.append(node) + + @staticmethod + def _node_name(node_or_name): + """Accept either a node instance or its bare name string.""" + if isinstance(node_or_name, str): + return node_or_name + # Prefer the legacy ``_CondorDAGNode__md5name`` attribute so we + # match exactly what HTCondor / glue.pipeline already use. + return ( + getattr(node_or_name, "_CondorDAGNode__md5name", None) + or getattr(node_or_name, "name", None) + or str(node_or_name) + ) + + def add_script_pre(self, node, executable, *args): + """Add a SCRIPT PRE hook for *node* (executable + args).""" + entry = (self._node_name(node), str(executable), + " ".join(str(a) for a in args)) + self.script_pre.append(entry) + if self._already_written: + get_backend().append_script_pre(self, *entry) + + def add_script_post(self, node, executable, *args): + """Add a SCRIPT POST hook for *node* (executable + args).""" + entry = (self._node_name(node), str(executable), + " ".join(str(a) for a in args)) + self.script_post.append(entry) + if self._already_written: + get_backend().append_script_post(self, *entry) + + def add_abort_on(self, node, exit_code, return_value=0): + """Add an ABORT-DAG-ON hook: if *node* returns *exit_code*, abort + the DAG with overall exit *return_value*.""" + entry = (self._node_name(node), int(exit_code), int(return_value)) + self.abort_on.append(entry) + if self._already_written: + get_backend().append_abort_on(self, *entry) + + def set_dot_file(self, path): + """Request a DAG visualisation file (DOT directive).""" + self.dot_file = path + if self._already_written: + get_backend().append_dot_file(self, path) + + def add_extra_directive(self, line): + """Append a raw backend-native directive line. Escape hatch for + anything the structured API doesn't cover.""" + self.extra_directives.append(line) + if self._already_written: + get_backend().append_extra_directive(self, line) + + def write_concrete_dag(self): + """Hand the DAG to the active backend, which writes the workflow artefact.""" + if self.dag_file is None: + raise RuntimeError("write_concrete_dag: no dag file set") + get_backend().emit_dag(self, self.dag_file) + self._already_written = True + + @property + def output_path(self): + """Path the active backend actually wrote to. + + Useful when external code wants to append further directives. For + HTCondor / glue this is ``.dag``; for Slurm it's the + ``_dag.sh`` driver script. + """ + if self.dag_file is None: + return None + return get_backend().output_path_for_dag(self.dag_file) + + +# =========================================================================== +# Backend abstraction +# =========================================================================== + + +class WorkflowBackend(abc.ABC): + """Abstract base class for execution-system backends. + + A backend takes the backend-neutral :class:`_GenericJob` / + :class:`_GenericDAG` containers and produces whatever artefacts the + target execution system expects (submit files, sbatch scripts, shell + drivers, ...). + """ + + #: Short identifier used to register the backend. + name = None + + @classmethod + def is_available(cls): + """Return ``True`` if the backend's runtime dependencies are installed. + + Used during auto-detection. Backends that have no python-importable + dependency (e.g. Slurm, which is detected by the presence of + ``sbatch`` on the host running the workflow) should return ``False`` + from auto-detect-time and be selected explicitly. + """ + return True + + @abc.abstractmethod + def emit_job(self, job, path): + """Write the submit artefact for *job* to *path*.""" + + @abc.abstractmethod + def emit_dag(self, dag, path): + """Write the workflow driver artefact for *dag* to *path*.""" + + # ------------------------------------------------------------------ + # DAG-level control-logic overlays. + # + # The default behaviour of every ``append_*`` method is to write a + # single backend-native line into ``output_path_for_dag(dag.dag_file)``. + # Subclasses override the formatters via ``format_*``; only Slurm + # needs to fully override ``append_script_post`` etc. because its + # control-logic equivalent isn't a single text line. + # ------------------------------------------------------------------ + + def output_path_for_dag(self, dag_file): + """Return the absolute path the backend wrote (or will write) + for *dag_file*. Default: append ``.dag`` if missing.""" + if dag_file.endswith(".dag"): + return dag_file + return dag_file + ".dag" + + def format_script_pre(self, name, exe, args_str): + return "SCRIPT PRE {} {} {}\n".format(name, exe, args_str).rstrip(" \n") + "\n" + + def format_script_post(self, name, exe, args_str): + return "SCRIPT POST {} {} {}\n".format(name, exe, args_str).rstrip(" \n") + "\n" + + def format_abort_on(self, name, exit_code, return_value): + return "ABORT-DAG-ON {} {} RETURN {}\n".format(name, exit_code, return_value) + + def format_dot_file(self, path): + return "DOT {}\n".format(path) + + def _append_to_output(self, dag, text): + path = self.output_path_for_dag(dag.dag_file) + with open(path, "a") as fh: + fh.write(text) + + def append_script_pre(self, dag, name, exe, args_str): + self._append_to_output(dag, self.format_script_pre(name, exe, args_str)) + + def append_script_post(self, dag, name, exe, args_str): + self._append_to_output(dag, self.format_script_post(name, exe, args_str)) + + def append_abort_on(self, dag, name, exit_code, return_value): + self._append_to_output(dag, self.format_abort_on(name, exit_code, return_value)) + + def append_dot_file(self, dag, path): + self._append_to_output(dag, self.format_dot_file(path)) + + def append_extra_directive(self, dag, line): + if not line.endswith("\n"): + line = line + "\n" + self._append_to_output(dag, line) + + def _emit_dag_control_overlays(self, dag, fh): + """Helper for emit_dag implementations that produce a single + text artefact: write all the captured control-logic overlays + appended to the workflow body.""" + for name, exe, args_str in dag.script_pre: + fh.write(self.format_script_pre(name, exe, args_str)) + for name, exe, args_str in dag.script_post: + fh.write(self.format_script_post(name, exe, args_str)) + for name, exit_code, return_value in dag.abort_on: + fh.write(self.format_abort_on(name, exit_code, return_value)) + if dag.dot_file is not None: + fh.write(self.format_dot_file(dag.dot_file)) + for line in dag.extra_directives: + if not line.endswith("\n"): + line = line + "\n" + fh.write(line) + + # Defaults useful for sub-classes + @staticmethod + def _build_argument_string(job, var_ref): + """Render a job's argument list as a single command-line string. + + *var_ref* is a callable mapping a variable-opt name to whatever string + should be substituted at submit time (e.g. ``"$(macroevent)"`` for + Condor, ``"${event}"`` for Slurm). + """ + parts = [] + for name, value in job.opts: + if value is None or value == "": + parts.append("--{}".format(name)) + else: + parts.append("--{}={}".format(name, value)) + for name, value in job.short_opts: + parts.append("-{} {}".format(name, value)) + for name, value in job.file_opts: + parts.append("--{}={}".format(name, value)) + for name in job.var_opts: + parts.append("--{}={}".format(name, var_ref(name))) + for arg in job.arguments: + parts.append(str(arg)) + return " ".join(parts) + + +# --------------------------------------------------------------------------- +# HTCondor backend (modern python bindings) +# --------------------------------------------------------------------------- + +class HTCondorBackend(WorkflowBackend): + """HTCondor backend using the modern ``htcondor`` python bindings. + + Submit descriptions are constructed via :class:`htcondor.Submit` (which + validates the result) and rendered to text. DAG files are emitted as + plain text in HTCondor DAGMan format. + """ + + name = "htcondor" + + @classmethod + def is_available(cls): + try: + import htcondor # noqa: F401 + return True + except Exception: + return False + + def __init__(self): + try: + import htcondor + self._htcondor = htcondor + except Exception: + self._htcondor = None + + @staticmethod + def _var_ref(name): + return "$(macro{})".format(name.replace("-", "_")) + + def _build_submit_dict(self, job): + sub = {} + if job.universe is not None: + sub["universe"] = job.universe + if job.executable is not None: + sub["executable"] = str(job.executable) + args = self._build_argument_string(job, self._var_ref) + if args: + # Condor "new arguments syntax": the entire argument string must + # be wrapped in double quotes. glue.pipeline.CondorDAGJob's + # write_sub_file() does this, so the legacy implementation + # emitted `arguments = "..."`. Without the outer quotes Condor + # falls back to the old arguments syntax, which parses brackets, + # commas, and embedded single-quoted tokens differently -- e.g. + # `'[0.2,0.2499]'` reaches the executable as a different argv + # string. Re-wrap here so we exactly match the glue behaviour. + if not (args.startswith('"') and args.endswith('"')): + args = '"' + args + '"' + sub["arguments"] = args + if job.log_file: + sub["log"] = job.log_file + if job.stdout_file: + sub["output"] = job.stdout_file + if job.stderr_file: + sub["error"] = job.stderr_file + # condor_cmds includes both well-known and custom commands; pass them all through. + for key, value in job.condor_cmds: + sub[key] = "" if value is None else str(value) + return sub + + def emit_job(self, job, path): + sub_dict = self._build_submit_dict(job) + text = None + if self._htcondor is not None: + try: + submit = self._htcondor.Submit(sub_dict) + text = str(submit) + if not text.endswith("\n"): + text += "\n" + text += "queue {}\n".format(job.queue_count) + except Exception: + text = None + if text is None: + text = self._render_submit_text(sub_dict, job.queue_count) + with open(path, "w") as fh: + fh.write(text) + + def _render_submit_text(self, sub_dict, queue_count): + ordered = ["universe", "executable", "arguments", "log", "output", "error"] + lines, seen = [], set() + for k in ordered: + if k in sub_dict: + lines.append("{} = {}".format(k, sub_dict[k])) + seen.add(k) + for k, v in sub_dict.items(): + if k in seen: + continue + lines.append("{} = {}".format(k, v)) + lines.append("queue {}".format(queue_count)) + return "\n".join(lines) + "\n" + + def emit_dag(self, dag, path): + if not path.endswith(".dag"): + path = path + ".dag" + with open(path, "w") as fh: + for node in dag.nodes: + if isinstance(node, _GenericSubdagNode): + fh.write("SUBDAG EXTERNAL {} {}\n".format(node.name, node.subdag_file)) + else: + sub = node.job.get_sub_file() + if sub is None: + raise RuntimeError( + "node {} references a job with no sub-file".format(node.name) + ) + fh.write("JOB {} {}\n".format(node.name, sub)) + if node.macros: + items = " ".join('{}="{}"'.format(k, v) for k, v in node.macros.items()) + fh.write("VARS {} {}\n".format(node.name, items)) + if node.category: + fh.write("CATEGORY {} {}\n".format(node.name, node.category)) + if node.retry: + fh.write("RETRY {} {}\n".format(node.name, node.retry)) + for node in dag.nodes: + for parent in node.parents: + fh.write("PARENT {} CHILD {}\n".format(parent.name, node.name)) + # Control-logic overlays (SCRIPT POST, ABORT-DAG-ON, DOT, ...) + self._emit_dag_control_overlays(dag, fh) + + +# --------------------------------------------------------------------------- +# glue.pipeline fallback backend +# --------------------------------------------------------------------------- + +class GluePipelineBackend(WorkflowBackend): + """Legacy fallback that drives ``glue.pipeline`` to produce submit / DAG + files. + + The generic :class:`_GenericJob` / :class:`_GenericDAG` state is replayed + onto a freshly-constructed ``glue.pipeline.CondorDAGJob`` / + ``glue.pipeline.CondorDAG`` at emit time. This means consumers see the + same backend-neutral surface as under HTCondor, but the actual file I/O + is handled by ``glue``. + """ + + name = "glue" + + @classmethod + def is_available(cls): + try: + from glue import pipeline # noqa: F401 + return True + except Exception: + return False + + def __init__(self): + from glue import pipeline as _p + self._pipeline = _p + # Translation cache so a node in emit_dag can find the glue object + # that was created to back its underlying _GenericJob, and so an + # already-submitted glue node isn't rebuilt. + self._glue_jobs_by_id = {} + self._glue_nodes_by_id = {} + + def _make_glue_job(self, job): + gid = id(job) + if gid in self._glue_jobs_by_id: + return self._glue_jobs_by_id[gid] + gjob = self._pipeline.CondorDAGJob(universe=job.universe, executable=job.executable) + if job.sub_file is not None: + gjob.set_sub_file(job.sub_file) + # glue.pipeline insists that log/stdout/stderr files all be set, + # raising CondorSubmitError otherwise; htcondor and slurm tolerate + # their absence. Smooth over the difference by deriving defaults + # from the sub-file whenever the caller didn't supply one. + if job.sub_file is not None: + base, _ = os.path.splitext(job.sub_file) + else: + base = None + log_file = job.log_file + if log_file is None and base is not None: + log_file = base + ".log" + if log_file is not None: + gjob.set_log_file(log_file) + stdout_file = job.stdout_file + if stdout_file is None and base is not None: + stdout_file = base + ".out" + if stdout_file is not None: + gjob.set_stdout_file(stdout_file) + stderr_file = job.stderr_file + if stderr_file is None and base is not None: + stderr_file = base + ".err" + if stderr_file is not None: + gjob.set_stderr_file(stderr_file) + for n, v in job.opts: + gjob.add_opt(n, v) + for n, v in job.short_opts: + gjob.add_short_opt(n, v) + for n, v in job.file_opts: + gjob.add_file_opt(n, v) + for n in job.var_opts: + gjob.add_var_opt(n) + for arg in job.arguments: + gjob.add_arg(arg) + for k, v in job.condor_cmds: + gjob.add_condor_cmd(k, v) + try: + # glue.pipeline stores the queue count in this private attribute. + gjob._CondorJob__queue = job.queue_count + except Exception: + pass + self._glue_jobs_by_id[gid] = gjob + return gjob + + def emit_job(self, job, path): + gjob = self._make_glue_job(job) + # Make sure sub-file path matches what the caller asked for + gjob.set_sub_file(path) + gjob.write_sub_file() + + def emit_dag(self, dag, path): + gdag = self._pipeline.CondorDAG(log=(dag.log or os.getcwd())) + # Build glue nodes + glue_nodes = {} + for node in dag.nodes: + if isinstance(node, _GenericSubdagNode): + # glue's CondorDAGManJob/create_node() + man = self._pipeline.CondorDAGManJob(node.subdag_file) + gnode = man.create_node() + else: + gjob = self._make_glue_job(node.job) + gnode = self._pipeline.CondorDAGNode(gjob) + for k, v in node.macros.items(): + gnode.add_macro(k, v) + if node.category: + gnode.set_category(node.category) + if node.retry: + gnode.set_retry(node.retry) + glue_nodes[id(node)] = gnode + # Wire up parents + for node in dag.nodes: + gnode = glue_nodes[id(node)] + for parent in node.parents: + if id(parent) in glue_nodes: + gnode.add_parent(glue_nodes[id(parent)]) + for node in dag.nodes: + gdag.add_node(glue_nodes[id(node)]) + # glue.pipeline.CondorDAG.write_concrete_dag() always appends a + # ".dag" suffix. If the caller already passed a path ending in + # ".dag" we'd get "wf.dag.dag" otherwise; strip the suffix so the + # final file lands at the path we were asked for. + if path.endswith(".dag"): + glue_path = path[:-4] + else: + glue_path = path + gdag.set_dag_file(glue_path) + gdag.write_concrete_dag() + # Append the control-logic overlays directly to the file glue + # produced (glue.pipeline does not expose its own SCRIPT POST / + # ABORT-DAG-ON / DOT API). + final_path = self.output_path_for_dag(path) + with open(final_path, "a") as fh: + self._emit_dag_control_overlays(dag, fh) + + def output_path_for_dag(self, dag_file): + # glue.pipeline always lands the dag at ``.dag``. + return dag_file if dag_file.endswith(".dag") else dag_file + ".dag" + + +# --------------------------------------------------------------------------- +# Slurm backend +# --------------------------------------------------------------------------- + +class SlurmBackend(WorkflowBackend): + """Slurm backend. + + Each :class:`_GenericJob` is emitted as an ``sbatch`` script. Resource + requests, queue counts, environments, etc. are translated into + ``#SBATCH`` directives where a portable mapping exists. A workflow + :class:`_GenericDAG` is emitted as a shell driver script that submits + its jobs in topological order with ``sbatch --dependency=afterok:...`` + chains. + + Things that don't have a portable Slurm equivalent (e.g. + ``MY.flock_local``, ``+SingularityImage``, condor universe ``local``) + are emitted as ``# HTCondor: ...`` comments in the sbatch script so the + information is preserved but inert under Slurm. + """ + + name = "slurm" + + @classmethod + def is_available(cls): + # We can run on a login node without sbatch installed (we'll just + # write the scripts), so don't gate on `sbatch`. However, we don't + # want auto-detection to pick Slurm by default since that would + # silently override Condor on misconfigured machines. + return False + + def emit_job(self, job, path): + lines = ["#!/bin/bash", ""] + + # Map structured resources / common semantics to #SBATCH directives. + directives = [] + if job.resources.get("memory"): + directives.append("--mem={}".format(self._coerce_mem(job.resources["memory"]))) + if job.resources.get("cpus"): + directives.append("--cpus-per-task={}".format(job.resources["cpus"])) + if job.resources.get("gpus"): + directives.append("--gres=gpu:{}".format(job.resources["gpus"])) + if job.resources.get("runtime_minutes"): + directives.append("--time={}".format(self._coerce_time(job.resources["runtime_minutes"]))) + if job.resources.get("disk"): + directives.append("--tmp={}".format(self._coerce_mem(job.resources["disk"]))) + if job.queue_count and int(job.queue_count) > 1: + directives.append("--array=0-{}".format(int(job.queue_count) - 1)) + if job.stdout_file: + directives.append("--output={}".format(job.stdout_file)) + if job.stderr_file: + directives.append("--error={}".format(job.stderr_file)) + # Job name from the executable, useful in squeue + if job.executable: + directives.append("--job-name={}".format(os.path.basename(str(job.executable)))) + # Map condor's "accounting_group" / "accounting_group_user" if present + for key, value in job.condor_cmds: + if key == "accounting_group": + directives.append("--account={}".format(value)) + elif key == "+SlurmPartition" or key == "MY.SlurmPartition": + directives.append("--partition={}".format(self._unquote(value))) + elif key == "+SlurmQOS" or key == "MY.SlurmQOS": + directives.append("--qos={}".format(self._unquote(value))) + + for d in directives: + lines.append("#SBATCH {}".format(d)) + + # Environment + if job.inherit_environment: + lines.append("#SBATCH --export=ALL") + elif job.environment: + kvs = ",".join("{}={}".format(k, v) for k, v in job.environment.items()) + lines.append("#SBATCH --export={}".format(kvs)) + + # Preserve all condor commands as comments so the information isn't lost + if job.condor_cmds: + lines.append("") + lines.append("# Original HTCondor submit-file commands (preserved for reference):") + for k, v in job.condor_cmds: + lines.append("# {} = {}".format(k, v)) + + # When queue_count > 1, the per-task variable opt substitution should + # use SLURM_ARRAY_TASK_ID; when there's only one task we still pass + # macros via per-job environment variables (the dag driver script + # will set them via --export when submitting). + def slurm_var(name): + ev = "SLURM_VAR_" + re.sub(r"\W+", "_", name).upper() + return "${" + ev + "}" + + args = self._build_argument_string(job, slurm_var) + lines.append("") + if not job.executable: + raise RuntimeError("SlurmBackend.emit_job: job has no executable") + if args: + lines.append("exec {} {}".format(job.executable, args)) + else: + lines.append("exec {}".format(job.executable)) + text = "\n".join(lines) + "\n" + with open(path, "w") as fh: + fh.write(text) + try: + os.chmod(path, 0o755) + except OSError: + pass + + def output_path_for_dag(self, dag_file): + """For Slurm, the workflow lands in a shell driver, not a .dag. + + Mapping: + ``foo.dag`` → ``foo_dag.sh`` + ``foo.sh`` → ``foo.sh`` + otherwise → ``foo.sh`` + """ + if dag_file.endswith(".sh"): + return dag_file + if dag_file.endswith(".dag"): + return dag_file[:-4] + "_dag.sh" + return dag_file + ".sh" + + def emit_dag(self, dag, path): + """Emit a shell driver that submits the DAG via sbatch dependency chains. + + High-level DAGMan control directives are translated as follows: + + * ``RETRY`` per node → ``--requeue`` on the corresponding sbatch + * ``VARS k="v"`` per node → ``--export=ALL,SLURM_VAR_K=v`` + * ``SCRIPT PRE`` per node → bash command run before sbatch + * ``SCRIPT POST`` per node → ``sbatch --dependency=afterany: --wrap=...`` + scheduled to run regardless of success/failure + * ``ABORT-DAG-ON`` per node → bash check that inspects the named + job's exit code, scancels future + dependents, and exits with the + requested return value + * ``DOT`` directive → comment (Slurm has no equivalent) + * Extra raw directives → comment (escape hatch); preserved verbatim + """ + path = self.output_path_for_dag(path) + + # Topological sort so we can emit submissions in dependency order + order = self._topological_sort(dag.nodes) + var_for_node_id = {} + var_for_node_name = {} + for i, node in enumerate(order): + var = "JOBID_{}".format(i) + var_for_node_id[id(node)] = var + var_for_node_name[node.name] = var + # Index pre/post hooks per node-name for quick lookup. + pre_by_name = {} + post_by_name = {} + for name, exe, args in dag.script_pre: + pre_by_name.setdefault(name, []).append((exe, args)) + for name, exe, args in dag.script_post: + post_by_name.setdefault(name, []).append((exe, args)) + + lines = [ + "#!/bin/bash", + "# Slurm driver script for workflow", + "# Submits each job via sbatch and chains them with --dependency=afterok", + "set -euo pipefail", + "", + ] + # Track post-script jobs we submitted so subsequent abort-on checks + # can wait for them too. + for node in order: + var = var_for_node_id[id(node)] + # SCRIPT PRE -- run synchronously before sbatch. + for exe, args in pre_by_name.get(node.name, []): + lines.append("# SCRIPT PRE for {}: {} {}".format(node.name, exe, args)) + lines.append("{} {}".format(exe, args).rstrip()) + + if isinstance(node, _GenericSubdagNode): + lines.append("# Sub-DAG: {}".format(node.subdag_file)) + lines.append('echo "Slurm backend: SUBDAG nodes are not supported natively; ' + 'driving sub-script {} via bash" >&2'.format(node.subdag_file)) + deps = self._dep_clause(node, var_for_node_id) + lines.append("{}=$(sbatch{} --wrap=\"bash {}\" | awk '{{print $4}}')".format( + var, deps, node.subdag_file)) + else: + sub = node.job.get_sub_file() + if sub is None: + raise RuntimeError( + "Slurm: node {} has no sbatch script".format(node.name)) + # Per-node variable opts get exported via --export=ALL,VAR=val + export_args = "" + if node.macros: + kvs = ",".join("SLURM_VAR_{}={}".format( + re.sub(r"\W+", "_", k).upper(), v) + for k, v in node.macros.items()) + export_args = " --export=ALL,{}".format(kvs) + deps = self._dep_clause(node, var_for_node_id) + retry_clause = "" + if node.retry: + retry_clause = " --requeue" + lines.append("{}=$(sbatch{}{}{} {} | awk '{{print $4}}')".format( + var, deps, export_args, retry_clause, sub)) + lines.append('echo "Submitted {} as ${{{}}}"'.format(node.name, var)) + + # SCRIPT POST -- a separate sbatch with afterany dependency so + # the post-script runs regardless of success / failure. + for j, (exe, args) in enumerate(post_by_name.get(node.name, [])): + post_var = "{}_post_{}".format(var, j) + lines.append("# SCRIPT POST for {}: {} {}".format(node.name, exe, args)) + lines.append( + "{}=$(sbatch --dependency=afterany:${{{}}} --wrap=\"{} {}\" " + "| awk '{{print $4}}')".format(post_var, var, exe, args.rstrip()) + ) + lines.append('echo "Post-script for {} as ${{{}}}"'.format(node.name, post_var)) + + # ABORT-DAG-ON -- block waiting for the named job, then if its + # exit code matches, scancel everything else and exit with the + # requested return value. Scheduled at the end of the driver + # (after all sbatch calls have been issued) so we have all jobids. + if dag.abort_on: + lines.append("") + lines.append("# ABORT-DAG-ON checks: monitor the named jobs and abort the workflow") + lines.append("# if any of them returns the matching exit code.") + for name, exit_code, return_value in dag.abort_on: + if name not in var_for_node_name: + lines.append("# (skipped abort-on for unknown node {})".format(name)) + continue + target_var = var_for_node_name[name] + lines.append("(") + lines.append(" sleep_until_finished_var=${{{}}}".format(target_var)) + lines.append(' while squeue -j "${sleep_until_finished_var}" -h ' + '-o "%i" 2>/dev/null | grep -q .; do sleep 30; done') + lines.append(' code=$(sacct -j "${sleep_until_finished_var}" -X ' + '--format=ExitCode --noheader | head -1 | awk -F: ' + "'{print $1}')") + lines.append(' if [[ "${{code}}" -eq {} ]]; then'.format(exit_code)) + lines.append(' echo "ABORT-DAG-ON: node {} returned exit ${{code}};' + ' scancelling remaining jobs and exiting {}" >&2' + .format(name, return_value)) + lines.append(' scancel ' + ' '.join( + '${{{}}}'.format(v) for v in var_for_node_id.values() if v != target_var + )) + lines.append(' exit {}'.format(return_value)) + lines.append(' fi') + lines.append(") &") + lines.append("wait") + + # DOT directive -- best-effort: emit a graphviz file describing the + # dependency graph, since Slurm has no native equivalent. + if dag.dot_file is not None: + lines.append("") + lines.append("# DOT visualisation requested by DAG; written to {}".format(dag.dot_file)) + lines.append("cat <<'__DOT_EOF__' > '{}'".format(dag.dot_file)) + lines.append("digraph workflow {") + for node in order: + lines.append(' "{}";'.format(node.name)) + for node in order: + for parent in node.parents: + lines.append(' "{}" -> "{}";'.format(parent.name, node.name)) + lines.append("}") + lines.append("__DOT_EOF__") + + # Escape-hatch raw directives -- emit each as a comment so the + # information is preserved (these are usually condor-specific and + # don't have a Slurm equivalent). + if dag.extra_directives: + lines.append("") + lines.append("# Original DAG-language directives preserved as comments:") + for line in dag.extra_directives: + lines.append("# " + line.rstrip("\n")) + + text = "\n".join(lines) + "\n" + with open(path, "w") as fh: + fh.write(text) + try: + os.chmod(path, 0o755) + except OSError: + pass + + # ------------------------------------------------------------------ + # Post-emission append hooks. + # + # For backends that emit a single DAG file (HTCondor / glue) the + # default ``WorkflowBackend._append_to_output`` is fine -- it just + # appends a ``SCRIPT POST ...\n`` line to the .dag and DAGMan picks + # it up. For Slurm the .sh driver is a bash program; we can't just + # append a foreign DAG-language line to it. So if the caller is + # adding control logic *after* ``write_concrete_dag()`` we re-emit + # the whole driver, picking up the new state captured on ``dag``. + # ------------------------------------------------------------------ + def _reemit(self, dag): + # Re-run emit_dag on the same target. We can do this safely + # because ``dag`` already holds the updated state. + self.emit_dag(dag, dag.dag_file) + + def append_script_pre(self, dag, name, exe, args_str): + self._reemit(dag) + + def append_script_post(self, dag, name, exe, args_str): + self._reemit(dag) + + def append_abort_on(self, dag, name, exit_code, return_value): + self._reemit(dag) + + def append_dot_file(self, dag, path): + self._reemit(dag) + + def append_extra_directive(self, dag, line): + self._reemit(dag) + + @staticmethod + def _dep_clause(node, var_for_node): + if not node.parents: + return "" + ids = ":".join("${" + var_for_node[id(p)] + "}" for p in node.parents if id(p) in var_for_node) + if not ids: + return "" + return " --dependency=afterok:" + ids + + @staticmethod + def _topological_sort(nodes): + # Kahn's algorithm + in_edges = {id(n): set(id(p) for p in n.parents) for n in nodes} + node_by_id = {id(n): n for n in nodes} + out = [] + ready = [n for n in nodes if not in_edges[id(n)]] + while ready: + n = ready.pop(0) + out.append(n) + for m in nodes: + if id(n) in in_edges[id(m)]: + in_edges[id(m)].remove(id(n)) + if not in_edges[id(m)]: + ready.append(m) + if len(out) != len(nodes): + # Cycle, just emit in input order + return list(nodes) + return out + + @staticmethod + def _coerce_mem(value): + # Accept "2048M", "2G", "1024", 1024 + s = str(value).strip() + # Condor sometimes uses just numbers (interpreted as MB); Slurm uses M/G suffix. + if re.fullmatch(r"\d+", s): + return s + "M" + return s + + @staticmethod + def _coerce_time(minutes): + try: + m = int(minutes) + except (TypeError, ValueError): + return str(minutes) + h, mm = divmod(m, 60) + return "{}:{:02d}:00".format(h, mm) + + @staticmethod + def _unquote(s): + s = str(s) + if len(s) >= 2 and s[0] == s[-1] and s[0] in ('"', "'"): + s = s[1:-1] + return s + + +# =========================================================================== +# Backend registry & selection +# =========================================================================== + +# Registry of backend instances, keyed by name. +_BACKENDS = {} +_ACTIVE_BACKEND_NAME = None + + +def register_backend(backend): + """Register a :class:`WorkflowBackend` instance. + + Subsequent calls to :func:`set_backend` (and the auto-detection path) can + reference *backend* by its ``name`` attribute. + """ + if backend.name is None: + raise ValueError("backend instance must have a non-None .name attribute") + _BACKENDS[backend.name] = backend + + +def set_backend(name): + """Select the active backend by name. Raises ``KeyError`` if unknown.""" + if name not in _BACKENDS: + raise KeyError( + "Unknown workflow backend: {!r}. Registered backends: {}".format( + name, sorted(_BACKENDS.keys()) + ) + ) + global _ACTIVE_BACKEND_NAME + _ACTIVE_BACKEND_NAME = name + + +def get_backend(): + """Return the active :class:`WorkflowBackend` instance. + + Raises :class:`RuntimeError` if no backend has been selected and none is + available for auto-selection. + """ + if _ACTIVE_BACKEND_NAME is None: + raise RuntimeError( + "No workflow backend is selected. Install one of " + "`htcondor` or `lscsoft-glue`, or call set_backend('slurm') / " + "set_backend('your_backend_name')." + ) + return _BACKENDS[_ACTIVE_BACKEND_NAME] + + +def current_backend_name(): + """Return the name of the active backend, or ``None`` if not selected.""" + return _ACTIVE_BACKEND_NAME + + +def _auto_select_backend(): + # Honor explicit override via environment variable + env_choice = os.environ.get("RIFT_DAG_BACKEND", "auto").lower() + if env_choice not in ("auto", "", "none"): + if env_choice in _BACKENDS: + set_backend(env_choice) + return + else: + print( + "[dag_utils_generic] RIFT_DAG_BACKEND={!r} not registered; " + "falling back to auto-detection".format(env_choice), + file=sys.stderr, + ) + # Auto: prefer HTCondor python bindings, then glue.pipeline. + for name in ("htcondor", "glue"): + if name in _BACKENDS: + try: + set_backend(name) + return + except Exception: + pass + + +# Register the bundled backends. +# +# We *always* register HTCondorBackend and SlurmBackend because both have a +# pure-python emission path with no library dependency (HTCondor falls back +# to a built-in submit-file text renderer when ``htcondor.Submit`` isn't +# available; Slurm only emits shell text). GluePipelineBackend is the +# exception: its emit_* paths construct real glue.pipeline objects, so we +# can only register it when glue is actually importable. +try: + register_backend(HTCondorBackend()) +except Exception as _exc: + print( + "[dag_utils_generic] WARNING: could not register HTCondorBackend: {}" + .format(_exc), + file=sys.stderr, + ) +if GluePipelineBackend.is_available(): + try: + register_backend(GluePipelineBackend()) + except Exception: + pass +register_backend(SlurmBackend()) + +_auto_select_backend() + + +# =========================================================================== +# Public factories (preserve historical names) + pipeline namespace shim +# =========================================================================== + +def CondorDAGJob(universe="vanilla", executable=None): + """Return a backend-neutral job container. + + Despite the legacy name, the returned object is *not* Condor-specific; it + just exposes the historical ``glue.pipeline.CondorDAGJob`` API. The + actual submit artefact emitted by :meth:`_GenericJob.write_sub_file` is + determined by the active :class:`WorkflowBackend`. + """ + return _GenericJob(universe=universe, executable=executable) + + +def CondorDAG(log=None): + return _GenericDAG(log=log) + + +def CondorDAGNode(job): + return _GenericNode(job) + + +def CondorDAGManJob(dag_file): + return _GenericManJob(dag_file) + + +# Expose a ``pipeline``-flavored namespace so consumers that previously did +# ``from glue import pipeline`` can switch to +# ``from RIFT.misc.dag_utils_generic import pipeline`` with no other changes. +class _PipelineNamespace(object): + CondorDAGJob = staticmethod(CondorDAGJob) + CondorDAG = staticmethod(CondorDAG) + CondorDAGNode = staticmethod(CondorDAGNode) + CondorDAGManJob = staticmethod(CondorDAGManJob) + + +pipeline = _PipelineNamespace() + + +# Legacy module-level constant. Kept for any external code that might check +# it (e.g. tests). Set lazily so it reflects the active backend rather than +# a static "auto-detected at import time" value. +def _legacy_BACKEND(): + return _ACTIVE_BACKEND_NAME + + +_BACKEND = _ACTIVE_BACKEND_NAME # snapshot for backwards-compat + + + +# --------------------------------------------------------------------------- +# Ported write_*_sub helpers +# --------------------------------------------------------------------------- +# +# Everything below is the original logic from RIFT.misc.dag_utils, copied +# verbatim except that ``pipeline.CondorDAGJob`` / ``pipeline.CondorDAG`` / +# ``pipeline.CondorDAGNode`` / ``pipeline.CondorDAGManJob`` references have +# been rewritten to use the backend-neutral classes defined above. This means +# every helper here works under both the htcondor python bindings backend and +# the glue.pipeline fallback backend. +# --------------------------------------------------------------------------- + +def write_integrate_likelihood_extrinsic_grid_sub(tag='integrate', exe=None, log_dir=None, ncopies=1, **kwargs): + """ + Write a submit file for launching jobs to marginalize the likelihood over + extrinsic parameters. + Like the other case (below), but modified to use the sim_xml + and loop over 'event' + + Inputs: + - 'tag' is a string to specify the base name of output files. The output + submit file will be named tag.sub, and the jobs will write their + output to tag-ID.out, tag-ID.err, tag.log, where 'ID' is a unique + identifier for each instance of a job run from the sub file. + - 'cache' is the path to a cache file which gives the location of the + data to be analyzed. + - 'sim' is the path to the XML file with the grid + - 'channelH1/L1/V1' is the channel name to be read for each of the + H1, L1 and V1 detectors. + - 'psdH1/L1/V1' is the path to an XML file specifying the PSD of + each of the H1, L1, V1 detectors. + - 'ncopies' is the number of runs with identical input parameters to + submit per condor 'cluster' + + Outputs: + - An instance of the CondorDAGJob that was generated for ILE + """ + + assert len(kwargs["psd_file"]) == len(kwargs["channel_name"]) + + exe = exe or which("integrate_likelihood_extrinsic") + ile_job = CondorDAGJob(universe="vanilla", executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + if "output_file" in kwargs and kwargs["output_file"] is not None: + # + # Need to modify the output file so it's unique + # + ofname = kwargs["output_file"].split(".") + ofname, ext = ofname[0], ".".join(ofname[1:]) + ile_job.add_file_opt("output-file", "%s-%s.%s" % (ofname, uniq_str, ext)) + del kwargs["output_file"] + if "save_samples" in kwargs and kwargs["save_samples"] is True: + ile_job.add_opt("save-samples", None) + del kwargs["save_samples"] + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + # + # Macro based options + # + ile_job.add_var_opt("event") + + if default_resolved_env: + ile_job.add_condor_cmd('environment', default_resolved_env) + else: + ile_job.add_condor_cmd('getenv', default_getenv_value) + ile_job.add_condor_cmd('request_memory', '2048M') + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + + ### + ### SUGGESTION FROM STUART (for later) + # request_memory = ifthenelse( (LastHoldReasonCode=!=34 && LastHoldReasonCode=!=26), InitialRequestMemory, int(1.5 * NumJobStarts * MemoryUsage) ) + # periodic_release = ((HoldReasonCode =?= 34) || (HoldReasonCode =?= 26)) + # This will automatically release a job that is put on hold for using too much memory with a 50% increased memory request each tim.e + + + return ile_job, ile_sub_name + + +# FIXME: Keep in sync with arguments of integrate_likelihood_extrinsic +def write_integrate_likelihood_extrinsic_sub(tag='integrate', exe=None, log_dir=None, ncopies=1, **kwargs): + """ + Write a submit file for launching jobs to marginalize the likelihood over + extrinsic parameters. + + Inputs: + - 'tag' is a string to specify the base name of output files. The output + submit file will be named tag.sub, and the jobs will write their + output to tag-ID.out, tag-ID.err, tag.log, where 'ID' is a unique + identifier for each instance of a job run from the sub file. + - 'cache' is the path to a cache file which gives the location of the + data to be analyzed. + - 'coinc' is the path to a coincident XML file, from which masses and + times will be drawn FIXME: remove this once it's no longer needed. + - 'channelH1/L1/V1' is the channel name to be read for each of the + H1, L1 and V1 detectors. + - 'psdH1/L1/V1' is the path to an XML file specifying the PSD of + each of the H1, L1, V1 detectors. + - 'ncopies' is the number of runs with identical input parameters to + submit per condor 'cluster' + + Outputs: + - An instance of the CondorDAGJob that was generated for ILE + """ + + assert len(kwargs["psd_file"]) == len(kwargs["channel_name"]) + + exe = exe or which("integrate_likelihood_extrinsic") + ile_job = CondorDAGJob(universe="vanilla", executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + if "output_file" in kwargs and kwargs["output_file"] is not None: + # + # Need to modify the output file so it's unique + # + ofname = kwargs["output_file"].split(".") + ofname, ext = ofname[0], ".".join(ofname[1:]) + ile_job.add_file_opt("output-file", "%s-%s.%s" % (ofname, uniq_str, ext)) + del kwargs["output_file"] + if "save_samples" in kwargs and kwargs["save_samples"] is True: + ile_job.add_opt("save-samples", None) + del kwargs["save_samples"] + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + # + # Macro based options + # + ile_job.add_var_opt("mass1") + ile_job.add_var_opt("mass2") + + if default_resolved_env: + ile_job.add_condor_cmd('environment', default_resolved_env) + else: + ile_job.add_condor_cmd('getenv', default_getenv_value) + ile_job.add_condor_cmd('request_memory', '2048M') + + return ile_job, ile_sub_name + +def write_result_coalescence_sub(tag='coalesce', exe=None, log_dir=None, output_dir="./", use_default_cache=True): + """ + Write a submit file for launching jobs to coalesce ILE output + """ + + exe = exe or which("ligolw_sqlite") + sql_job = CondorDAGJob(universe="vanilla", executable=exe) + + sql_sub_name = tag + '.sub' + sql_job.set_sub_file(sql_sub_name) + + # + # Logging options + # + uniq_str = "$(cluster)-$(process)" + sql_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + sql_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + sql_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + if use_default_cache: + sql_job.add_opt("input-cache", "ILE_$(macromassid).cache") + else: + sql_job.add_arg("$(macrofiles)") + #sql_job.add_arg("*$(macromassid)*.xml.gz") + sql_job.add_opt("database", "ILE_$(macromassid).sqlite") + #if os.environ.has_key("TMPDIR"): + #tmpdir = os.environ["TMPDIR"] + #else: + #print >>sys.stderr, "WARNING, TMPDIR environment variable not set. Will default to /tmp/, but this could be dangerous." + #tmpdir = "/tmp/" + tmpdir = "/dev/shm/" + sql_job.add_opt("tmp-space", tmpdir) + sql_job.add_opt("verbose", None) + + if default_resolved_env: + sql_job.add_condor_cmd('environment', default_resolved_env) + else: + sql_job.add_condor_cmd('getenv', default_getenv_value) + sql_job.add_condor_cmd('request_memory', '1024') + + return sql_job, sql_sub_name + +def write_posterior_plot_sub(tag='plot_post', exe=None, log_dir=None, output_dir="./"): + """ + Write a submit file for launching jobs to coalesce ILE output + """ + + exe = exe or which("plot_like_contours") + plot_job = CondorDAGJob(universe="vanilla", executable=exe) + + plot_sub_name = tag + '.sub' + plot_job.set_sub_file(plot_sub_name) + + # + # Logging options + # + uniq_str = "$(cluster)-$(process)" + plot_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + plot_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + plot_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + plot_job.add_opt("show-points", None) + plot_job.add_opt("dimension1", "mchirp") + plot_job.add_opt("dimension2", "eta") + plot_job.add_opt("input-cache", "ILE_all.cache") + plot_job.add_opt("log-evidence", None) + + plot_job.add_condor_cmd('getenv', default_getenv_value) + plot_job.add_condor_cmd('request_memory', '1024') + + return plot_job, plot_sub_name + +def write_tri_plot_sub(tag='plot_tri', injection_file=None, exe=None, log_dir=None, output_dir="./"): + """ + Write a submit file for launching jobs to coalesce ILE output + """ + + exe = exe or which("make_triplot") + plot_job = CondorDAGJob(universe="vanilla", executable=exe) + + plot_sub_name = tag + '.sub' + plot_job.set_sub_file(plot_sub_name) + + # + # Logging options + # + uniq_str = "$(cluster)-$(process)" + plot_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + plot_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + plot_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + plot_job.add_opt("output", "ILE_triplot_$(macromassid).png") + if injection_file is not None: + plot_job.add_opt("injection", injection_file) + plot_job.add_arg("ILE_$(macromassid).sqlite") + + plot_job.add_condor_cmd('getenv', default_getenv_value) + #plot_job.add_condor_cmd('request_memory', '2048M') + + return plot_job, plot_sub_name + +def write_1dpos_plot_sub(tag='1d_post_plot', exe=None, log_dir=None, output_dir="./"): + """ + Write a submit file for launching jobs to coalesce ILE output + """ + + exe = exe or which("postprocess_1d_cumulative") + plot_job = CondorDAGJob(universe="vanilla", executable=exe) + + plot_sub_name = tag + '.sub' + plot_job.set_sub_file(plot_sub_name) + + # + # Logging options + # + uniq_str = "$(cluster)-$(process)" + plot_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + plot_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + plot_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + plot_job.add_opt("save-sampler-file", "ILE_$(macromassid).sqlite") + plot_job.add_opt("disable-triplot", None) + plot_job.add_opt("disable-1d-density", None) + + plot_job.add_condor_cmd('getenv', default_getenv_value) + plot_job.add_condor_cmd('request_memory', '2048M') + + return plot_job, plot_sub_name + + +def write_CIP_single_iteration_subdag(cip_worker_job,it,unique_postfix,subdag_dir,n_retries=3,n_explode=1): + # Assume subdag_dir exists, we will append onto filename + dag = CondorDAG(log=os.getcwd()) + for indx in range(n_explode): + worker_node =CondorDAGNode(cip_worker_job) + worker_node.add_macro("macroiteration", it) + worker_node.add_macro("macroiterationnext", it+1) + worker_node.set_category("CIP_worker") + worker_node.set_retry(n_retries) + dag.add_node(worker_node) + dag_name=subdag_dir+"/subdag_CIP_{}".format(unique_postfix) + dag.set_dag_file(dag_name) + dag.write_concrete_dag() + return dag_name + ".dag" + + + +def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output-ILE-samples',universe="vanilla",out_dir=None,log_dir=None, use_eos=False,ncopies=1,arg_str=None,request_memory=8192,request_memory_flex=False, arg_vals=None, no_grid=False,request_disk=False, transfer_files=None,transfer_output_files=None,use_singularity=False,use_osg=False,use_oauth_files=False,use_simple_osg_requirements=False,singularity_image=None,max_runtime_minutes=None,condor_commands=None,**kwargs): + """ + Write a submit file for launching jobs to marginalize the likelihood over intrinsic parameters. + + Inputs: + Outputs: + - An instance of the CondorDAGJob that was generated for ILE + """ + + if use_singularity and (singularity_image == None) : + print(" FAIL : Need to specify singularity_image to use singularity ") + sys.exit(0) + if use_singularity and (transfer_files == None) : + print(" FAIL : Need to specify transfer_files to use singularity at present! (we will append the prescript; you should transfer any PSDs as well as the grid file ") + sys.exit(0) + + singularity_image_used = "{}".format(singularity_image) # make copy + extra_files = [] + # Container family manifest support (see write_ILE_sub_simple). CIP jobs do + # not request GPUs, so no require_gpus floor is added here; on a CPU-only + # slot TARGET.GPUs_Capability is undefined and the selection expression + # collapses to the fallback image, which must be the CPU-safe one. + singularity_is_family = False + singularity_image_expr = None + singularity_transfer_expr = None + if singularity_image and is_container_manifest(singularity_image): + singularity_is_family = True + _manifest = load_container_manifest(singularity_image) + singularity_image_expr = build_singularity_image_expr(_manifest) + singularity_transfer_expr = build_transfer_input_expr(_manifest) + if singularity_transfer_expr: + extra_files += [singularity_transfer_expr] + elif singularity_image: + if 'osdf:' in singularity_image: + singularity_image_used = "./{}".format(singularity_image.split('/')[-1]) + extra_files += [singularity_image] + + exe = exe or which("util_ConstructIntrinsicPosterior_GenericCoordinates.py") + if use_singularity: + exe_base = os.path.basename(exe) +# path_split = exe.split("/") +# print((" Executable: name breakdown ", path_split, " from ", exe)) + singularity_base_exe_path = "/usr/bin/" # should not hardcode this ...! + if 'SINGULARITY_BASE_EXE_DIR_HYPERPIPE' in list(os.environ.keys()) : # allow a DIFFERENT exe to be used here for hyperpipe : CIP used for remote + singularity_base_exe_path = os.environ['SINGULARITY_BASE_EXE_DIR_HYPERPIPE'] + elif 'SINGULARITY_BASE_EXE_DIR' in list(os.environ.keys()) : + singularity_base_exe_path = os.environ['SINGULARITY_BASE_EXE_DIR'] + exe=singularity_base_exe_path + exe_base + if exe_base == 'true': # special universal path for /bin/true, don't override it! + exe = "/usr/bin/true" + ile_job = CondorDAGJob(universe=universe, executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # + # Add options en mass, by brute force + # + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line +# ile_job.add_opt(arg_str[2:],'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + + ile_job.add_opt("fname", input_net) + ile_job.add_opt("fname-output-samples", out_dir+"/"+output) + ile_job.add_opt("fname-output-integral", out_dir+"/"+output) + + # + # Macro based options. + # - select EOS from list (done via macro) + # - pass spectral parameters + # +# ile_job.add_var_opt("event") + if use_eos: + ile_job.add_var_opt("using-eos") + + + # + # Logging options + # + uniq_str = "$(macroevent)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + if "fname_output_samples" in kwargs and kwargs["fname_output_samples"] is not None: + # + # Need to modify the output file so it's unique + # + ofname = kwargs["fname_output_samples"].split(".") + ofname, ext = ofname[0], ".".join(ofname[1:]) + ile_job.add_file_opt("output-file", "%s-%s.%s" % (ofname, uniq_str, ext)) + if "fname_output_integral" in kwargs and kwargs["fname_output_integral"] is not None: + # + # Need to modify the output file so it's unique + # + ofname = kwargs["fname_output_integral"].split(".") + ofname, ext = ofname[0], ".".join(ofname[1:]) + ile_job.add_file_opt("output-file", "%s-%s.%s" % (ofname, uniq_str, ext)) + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + if not use_osg: + if default_resolved_env: + ile_job.add_condor_cmd('environment', default_resolved_env) + else: + ile_job.add_condor_cmd('getenv', default_getenv_value) + if not(request_memory_flex): + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + if request_memory_flex: + ile_job.add_condor_cmd("MY.InitialRequestMemory",str(request_memory)) + ile_job.add_condor_cmd('periodic_release', "HoldReasonCode =?= 34") + ile_job.add_condor_cmd('request_memory', 'ifthenelse( LastHoldReasonCode=!=34, InitialRequestMemory, int(1.5 * MemoryUsage) )') + if not(request_disk is False): + ile_job.add_condor_cmd('request_disk', str(request_disk)) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + requirements = [] + if use_singularity: + # Compare to https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py + ile_job.add_condor_cmd('request_CPUs', str(1)) + ile_job.add_condor_cmd('transfer_executable', 'False') + ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') + if singularity_is_family: + # Expression-valued: emit raw, NO surrounding double quotes. + ile_job.add_condor_cmd("MY.SingularityImage", singularity_image_expr) + else: + ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') + requirements.append("HAS_SINGULARITY=?=TRUE") + + if use_oauth_files: + # we are using some authentication to retrieve files from the file transfer list, for example, from distributed hosts, not just submit. eg urls provided + ile_job.add_condor_cmd('use_oauth_services',use_oauth_files) + if use_osg: + # avoid black-holing jobs to specific machines that consistently fail. Uses history attribute for ad + # https://htcondor.readthedocs.io/en/latest/classad-attributes/job-classad-attributes.html + ile_job.add_condor_cmd('periodic_release','((HoldReasonCode == 45) && (HoldReasonSubCode == 0)) || (HoldReasonCode == 13)') + ile_job.add_condor_cmd('job_machine_attrs','Machine') + ile_job.add_condor_cmd('job_machine_attrs_history_length','4') +# for indx in [1,2,3,4]: +# requirements.append("TARGET.GLIDEIN_ResourceName=!=MY.MachineAttrGLIDEIN_ResourceName{}".format(indx)) + if "OSG_DESIRED_SITES" in os.environ: + ile_job.add_condor_cmd('+DESIRED_SITES',os.environ["OSG_DESIRED_SITES"]) + if "OSG_UNDESIRED_SITES" in os.environ: + ile_job.add_condor_cmd('+UNDESIRED_SITES',os.environ["OSG_UNDESIRED_SITES"]) + # Some options to automate restarts, acts on top of RETRY in dag + if use_singularity or use_osg: + # Set up file transfer options + ile_job.add_condor_cmd("when_to_transfer_output",'ON_EXIT') + + # Stream log info + if not ('RIFT_NOSTREAM_LOG' in os.environ): + ile_job.add_condor_cmd("stream_error",'True') + ile_job.add_condor_cmd("stream_output",'True') + + if use_osg and ( 'RIFT_BOOLEAN_LIST' in os.environ): + extra_requirements = [ "{} =?= TRUE".format(x) for x in os.environ['RIFT_BOOLEAN_LIST'].split(',')] + requirements += extra_requirements + + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + # Stream log info: always stream CIP error, it is a critical bottleneck + if True: # not ('RIFT_NOSTREAM_LOG' in os.environ): + ile_job.add_condor_cmd("stream_error",'True') + ile_job.add_condor_cmd("stream_output",'True') + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + if not transfer_files is None: + if not isinstance(transfer_files, list): + fname_str=transfer_files + ' '.join(extra_files) + else: + fname_str = ','.join(transfer_files + extra_files) + fname_str=fname_str.strip() + ile_job.add_condor_cmd('transfer_input_files', fname_str) + ile_job.add_condor_cmd('should_transfer_files','YES') + + # Periodic remove: kill jobs running longer than max runtime + # https://stackoverflow.com/questions/5900400/maximum-run-time-in-condor + if not(max_runtime_minutes is None): + remove_str = 'JobStatus =?= 2 && (CurrentTime - JobStartDate) > ( {})'.format(60*max_runtime_minutes) + ile_job.add_condor_cmd('periodic_remove', remove_str) + + + ### + ### SUGGESTION FROM STUART (for later) + # request_memory = ifthenelse( (LastHoldReasonCode=!=34 && LastHoldReasonCode=!=26), InitialRequestMemory, int(1.5 * NumJobStarts * MemoryUsage) ) + # periodic_release = ((HoldReasonCode =?= 34) || (HoldReasonCode =?= 26)) + # This will automatically release a job that is put on hold for using too much memory with a 50% increased memory request each tim.e + if condor_commands is not None: + for cmd, value in condor_commands.items(): + ile_job.add_condor_cmd(cmd, value) + + + return ile_job, ile_sub_name + + +def write_puff_sub(tag='puffball', exe=None, base=None,input_net='output-ILE-samples',output='puffball',universe="vanilla",out_dir=None,log_dir=None, use_eos=False,ncopies=1,arg_str=None,request_memory=1024,arg_vals=None, no_grid=False,extra_text='',**kwargs): + """ + Perform puffball calculation + Inputs: + Outputs: + - An instance of the CondorDAGJob that was generated for ILE + """ + + exe = exe or which("util_ParameterPuffball.py") + # Create executable if needed (using extra_text as flag for now) + base_str = '' + if len(extra_text) > 0: + if not (base is None): + base_str = ' ' + base +"/" + + cmdname = "puff_sub.sh" + + with open(cmdname,'w') as f: + f.write("#! /usr/bin/env bash\n") + f.write(extra_text+"\n") + extra_args = '' + f.write( exe + " $@ \n") + + st = os.stat(cmdname) + import stat + os.chmod(cmdname, st.st_mode | stat.S_IEXEC) + + exe = base_str + "puff_sub.sh" + + + + ile_job = CondorDAGJob(universe=universe, executable=exe) + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # + # Add options en mass, by brute force + # + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + + if not(input_net is None): + ile_job.add_opt("inj-file", input_net) # using this double-duty for FETCH, other use cases + if not(output is None): + ile_job.add_opt("inj-file-out", output) + + + # + # Logging options + # + uniq_str = "$(macroevent)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + if default_resolved_env: + ile_job.add_condor_cmd('environment', default_resolved_env) + else: + ile_job.add_condor_cmd('getenv', default_getenv_value) + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + return ile_job, ile_sub_name + + +def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False,simple_unique=False,ncopies=1,arg_str=None,request_memory=4096,request_gpu=False,request_cross_platform=False,request_disk=False,arg_vals=None, transfer_files=None,transfer_output_files=None,use_singularity=False,use_osg=False,use_simple_osg_requirements=False,singularity_image=None,use_cvmfs_frames=False,use_oauth_files=False,frames_dir=None,cache_file=None,fragile_hold=False,max_runtime_minutes=None,condor_commands=None,**kwargs): + """ + Write a submit file for launching jobs to marginalize the likelihood over intrinsic parameters. + + Inputs: + Outputs: + - An instance of the CondorDAGJob that was generated for ILE + """ + if use_singularity and (singularity_image == None) : + print(" FAIL : Need to specify singularity_image to use singularity ") + sys.exit(0) + if use_singularity and (frames_dir == None) and (cache_file == None) : + print(" FAIL : Need to specify frames_dir or cache_file to use singularity (at present) ") + sys.exit(0) + if use_singularity and (transfer_files == None) : + print(" FAIL : Need to specify transfer_files to use singularity at present! (we will append the prescript; you should transfer any PSDs as well as the grid file ") + sys.exit(0) + + singularity_image_used = "{}".format(singularity_image) # make copy + extra_files = [] + # Container family manifest support: if singularity_image points at a + # .yaml/.yml manifest, build an expression-valued MY.SingularityImage plus a + # selective ($$()) transfer entry and a require_gpus capability floor. A + # plain .sif / osdf:// value keeps the legacy single-image behavior below. + singularity_is_family = False + singularity_container_universe = False + singularity_container_image_select = None + singularity_image_expr = None + singularity_transfer_expr = None + singularity_require_gpus_floor = None + if singularity_image and is_container_manifest(singularity_image): + singularity_is_family = True + _manifest = load_container_manifest(singularity_image) + singularity_image_expr = build_singularity_image_expr(_manifest) + singularity_transfer_expr = build_transfer_input_expr(_manifest) + singularity_require_gpus_floor = build_require_gpus_floor(_manifest) + # OSG-safe alternative (opt-in: RIFT_CONTAINER_UNIVERSE). The execute-side + # ifThenElse MY.SingularityImage below works on the CIT-local pool but + # OSPool glidein pilots read SingularityImage as a LITERAL string and hold + # the job. Container universe with container_image = $$([...]) instead + # uses HTCondor match-time machine-ad substitution: the schedd resolves it + # to a literal image before the job reaches the EP. $$ in container_image + # is HTCondor's documented per-GPU-capability selection, and works on both + # CIT-local and OSPool. Requires use_singularity. + singularity_container_universe = bool(use_singularity and os.environ.get('RIFT_CONTAINER_UNIVERSE')) + if singularity_container_universe: + singularity_container_image_select = build_container_image_select(_manifest) + # Selective transfer: only the matched osdf image is fetched (via the + # $$() token, which is comma-free so it survives transfer_input_files + # comma-splitting). CVMFS/local images are referenced in place and + # never transferred, so the whole family is never pulled. In container- + # universe mode the image is delivered via container_image itself, so we + # do NOT add the transfer token. + if singularity_transfer_expr and not singularity_container_universe: + extra_files += [singularity_transfer_expr] + elif singularity_image: + if 'osdf:' in singularity_image: + singularity_image_used = "./{}".format(singularity_image.split('/')[-1]) + extra_files += [singularity_image] + + + exe = exe or which("integrate_likelihood_extrinsic") + frames_local = None + if use_singularity: + exe_base = os.path.basename(exe) +# print((" Executable: name breakdown ", path_split, " from ", exe)) + singularity_base_exe_path = "/opt/lscsoft/rift/MonteCarloMarginalizeCode/Code/" # should not hardcode this ...! + if 'SINGULARITY_BASE_EXE_DIR' in list(os.environ.keys()) : + singularity_base_exe_path = os.environ['SINGULARITY_BASE_EXE_DIR'] + else: +# singularity_base_exe_path = "/opt/lscsoft/rift/MonteCarloMarginalizeCode/Code/" # should not hardcode this ...! + singularity_base_exe_path = "/usr/bin/" # should not hardcode this ...! + exe=singularity_base_exe_path + exe_base + if not(frames_dir is None): + frames_local = frames_dir.split("/")[-1] + elif use_osg: # NOT using singularity! + if not(frames_dir is None): + frames_local = frames_dir.split("/")[-1] + exe = os.path.basename(exe) + exe_here = 'my_wrapper.sh' + if transfer_files is None: + transfer_files = [] + transfer_files += ['../my_wrapper.sh'] + with open(exe_here,'w') as f: + f.write("#! /bin/bash \n") + f.write(r""" +#!/bin/bash +# Modules and scripts run directly from repository +# Note the repo and branch are self-referential ! Not a robust solution long-term +# Exit on failure: +# set -e +export INSTALL_DIR=research-projects-RIT +export ILE_DIR=${INSTALL_DIR}/MonteCarloMarginalizeCode/Code +export PATH=${PATH}:${ILE_DIR} +export PYTHONPATH=${PYTHONPATH}:${ILE_DIR} +export GW_SURROGATE=gwsurrogate +git clone https://git.ligo.org/richard-oshaughnessy/research-projects-RIT.git +pushd ${INSTALL_DIR} +git checkout temp-RIT-Tides-port_master-GPUIntegration +popd + +ls +cat local.cache +echo Starting ... +./research-projects-RIT/MonteCarloMarginalizeCode/Code/""" + exe + " $@ \n") + os.system("chmod a+x "+exe_here) + exe = exe_here # update executable + + + # Container universe (opt-in) runs the job inside container_image directly; + # otherwise stay vanilla + (optional) condor singularity. + ile_job = CondorDAGJob(universe=("container" if singularity_container_universe else "vanilla"), executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # + # Add options en mass, by brute force + # + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + if '"' in arg_str: + arg_str = safely_quote_arg_str(arg_str) + #arg_str = arg_str.replace('"','""') # double quote for condor - weird but true + ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line +# ile_job.add_opt(arg_str[2:],'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + + # + # Macro based options. + # - select EOS from list (done via macro) + # - pass spectral parameters + # +# ile_job.add_var_opt("event") + if use_eos: + ile_job.add_var_opt("using-eos") + + + requirements =[] + # + # Logging options + # + uniq_str = "$(macroevent)-$(cluster)-$(process)" + if simple_unique: + uniq_str = "$(macroevent)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + # Add lame initial argument + + if "output_file" in kwargs and kwargs["output_file"] is not None: + # + # Need to modify the output file so it's unique + # + ofname = kwargs["output_file"].split(".") + ofname, ext = ofname[0], ".".join(ofname[1:]) + ile_job.add_file_opt("output-file", "%s-%s.%s" % (ofname, uniq_str, ext)) + del kwargs["output_file"] + if "save_samples" in kwargs and kwargs["save_samples"] is True: + ile_job.add_opt("save-samples", None) + del kwargs["save_samples"] + + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + if cache_file: + ile_job.add_opt("cache-file",cache_file) + + ile_job.add_var_opt("event") + + if not use_osg: + ile_job.add_condor_cmd('getenv', default_getenv_value) + else: + env_statement="*RIFT*" + if 'RIFT_GETENV_OSG' in os.environ: + env_statement = os.environ['RIFT_GETENV_OSG'] # for example use NUMBA_CACHE_DIR=/tmp; see https://git.ligo.org/computing/helpdesk/-/issues/4616 + # special-purpose environment variable to help tweak remote execution/driver issues + if 'CUDA_LAUNCH_BLOCKING' in os.environ: + env_statement+= ",CUDA_LAUNCH_BLOCKING" + if default_resolved_env: + new_resolved_env = build_resolved_env(env_statement) + ile_job.add_condor_cmd('environment', new_resolved_env) + else: + ile_job.add_condor_cmd('getenv', env_statement) # retrieve any RIFT commands -- specifically RIFT_LOWLATENCY + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + if not(request_disk is False): + ile_job.add_condor_cmd('request_disk', str(request_disk)) + nGPUs =0 + requirements = [] + if request_gpu: + nGPUs=1 + if request_cross_platform: + # recipe from https://opensciencegrid.atlassian.net/browse/HTCONDOR-2200 + nGPUs = 'countMatches(RequireGPUs, AvailableGPUs) >= 1 ? 1 : 0' + ile_job.add_condor_cmd('rank', 'RequestGPUs') + ile_job.add_condor_cmd('request_GPUs', str(nGPUs)) +# Claim we don't need to make this request anymore to avoid out-of-memory errors. Also, no longer in 'requirements' +# requirements.append("CUDAGlobalMemoryMb >= 2048") + if use_singularity: + # Compare to https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py + ile_job.add_condor_cmd('request_CPUs', str(1)) + ile_job.add_condor_cmd('transfer_executable', 'False') + if singularity_container_universe: + # Container universe: the per-machine image is delivered via + # container_image, a $$()-substituted (match-time) literal -- emit it + # raw/unquoted (a $$() value must not be wrapped in quotes), with NO + # MY.SingularityImage / MY.SingularityBindCVMFS. GPU access is + # automatic under request_gpus; the executable runs inside the image. + ile_job.add_condor_cmd("container_image", singularity_container_image_select) + else: + ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') + if singularity_is_family: + # Expression-valued: emit the ifThenElse raw, with NO surrounding + # double quotes (a classad expression must not be quoted). + ile_job.add_condor_cmd("MY.SingularityImage", singularity_image_expr) + else: + ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') + ile_job.add_condor_cmd("MY.flock_local",'true') # jobs can match to local pool ! + requirements.append("HAS_SINGULARITY=?=TRUE") +# if not(use_simple_osg_requirements): +# requirements.append("HAS_CVMFS_LIGO_CONTAINERS=?=TRUE") + #ile_job.add_condor_cmd("requirements", ' (IS_GLIDEIN=?=True) && (HAS_LIGO_FRAMES=?=True) && (HAS_SINGULARITY=?=TRUE) && (HAS_CVMFS_LIGO_CONTAINERS=?=TRUE)') + + if use_oauth_files: + # we are using some authentication to retrieve files from the file transfer list, for example, from distributed hosts, not just submit. eg urls provided + ile_job.add_condor_cmd('use_oauth_services',use_oauth_files) + if use_cvmfs_frames: + requirements.append("HAS_LIGO_FRAMES=?=TRUE") + if 'LIGO_OATH_SCOPE' in os.environ: + ile_job.add_condor_cmd('use_oauth_services','igwn') + ile_job.add_condor_cmd('igwn_oauth_permissions',os.environ['LIGO_OATH_SCOPE']) + else: + ile_job.add_condor_cmd('use_x509userproxy','True') + if 'X509_USER_PROXY' in list(os.environ.keys()): + print(" Storing copy of X509 user proxy -- beware expiration! ") + cwd = os.getcwd() + fname_proxy = cwd +"/my_proxy" # this can get overwritten, that's fine - just renews, feature not bug + os.system("cp ${X509_USER_PROXY} " + fname_proxy) + # ile_job.add_condor_cmd('x509userproxy',os.environ['X509_USER_PROXY']) + ile_job.add_condor_cmd('x509userproxy',fname_proxy) + + if use_osg: +# if not(use_simple_osg_requirements): +# requirements.append("IS_GLIDEIN=?=TRUE") + # avoid black-holing jobs to specific machines that consistently fail. Uses history attribute for ad + ile_job.add_condor_cmd('periodic_release','((HoldReasonCode == 45) && (HoldReasonSubCode == 0)) || (HoldReasonCode == 13)') + ile_job.add_condor_cmd('job_machine_attrs','Machine') + ile_job.add_condor_cmd('job_machine_attrs_history_length','4') +# for indx in [1,2,3,4]: +# requirements.append("TARGET.GLIDEIN_ResourceName=!=MY.MachineAttrGLIDEIN_ResourceName{}".format(indx)) + if "OSG_DESIRED_SITES" in os.environ: + ile_job.add_condor_cmd('+DESIRED_SITES',os.environ["OSG_DESIRED_SITES"]) + if "OSG_UNDESIRED_SITES" in os.environ: + ile_job.add_condor_cmd('+UNDESIRED_SITES',os.environ["OSG_UNDESIRED_SITES"]) + # Some options to automate restarts, acts on top of RETRY in dag + if fragile_hold: + ile_job.add_condor_cmd("periodic_release","(NumJobStarts < 5) && ((CurrentTime - EnteredCurrentStatus) > 600)") + ile_job.add_condor_cmd("on_exit_hold","(ExitBySignal == True) || (ExitCode != 0)") + if use_singularity or use_osg: + # Set up file transfer options + ile_job.add_condor_cmd("when_to_transfer_output",'ON_EXIT') + + # Stream log info + if not ('RIFT_NOSTREAM_LOG' in os.environ): + ile_job.add_condor_cmd("stream_error",'True') + ile_job.add_condor_cmd("stream_output",'True') + + if use_osg and ( 'RIFT_BOOLEAN_LIST' in os.environ): + extra_requirements = [ "{} =?= TRUE".format(x) for x in os.environ['RIFT_BOOLEAN_LIST'].split(',')] + requirements += extra_requirements + + # Create prescript command to set up local.cache, only if frames are needed + # if we have CVMFS frames, we should be copying local.cache over directly, with it already populated ! + if not(frames_local is None) and not(use_cvmfs_frames): # should be required for singularity or osg + try: + lalapps_path2cache=os.environ['LALAPPS_PATH2CACHE'] + except KeyError: + print("Variable LALAPPS_PATH2CACHE is unset, assume default lal_path2cache is appropriate") + lalapps_path2cache="lal_path2cache" + cmdname = 'ile_pre.sh' + if transfer_files is None: + transfer_files = [] + transfer_files += [frames_dir] + # Test if we *need* ile_pre.sh : are path names already relative? DOES NOT WORK + pre_needed = True + if False: # try: + with open('local.cache', 'r') as f: + lines = f.readlines() + fnames = [x.split()[-1] for x in lines] + fnames_no_prefix = [x.replace('file:/','').replace('osdf:/','') for x in fnames] + for name in fnames_no_prefix: + if name[0] == '/': + pre_needed =True + else: # except: + print(" WARNING: local.cache file not present, reverting to ile_pre.sh ") + if not(pre_needed): + transfer_files += ['../local.cache'] + else: + transfer_files += ["../ile_pre.sh"] # assuming default working directory setup + with open(cmdname,'w') as f: + f.write("#! /bin/bash -xe \n") + f.write( "ls "+frames_local+" | {lalapps_path2cache} 1> local.cache \n".format(lalapps_path2cache=lalapps_path2cache)) # Danger: need user to correctly specify local.cache directory + # Rewrite cache file to use relative paths, not a file:// operation + f.write(" cat local.cache | awk '{print $1, $2, $3, $4}' > local_stripped.cache \n") + f.write("for i in `ls " + frames_local + "`; do echo "+ frames_local + "/$i; done > base_paths.dat \n") + f.write("paste local_stripped.cache base_paths.dat > local_relative.cache \n") + f.write("cp local_relative.cache local.cache \n") + f.write('{exe} "$@" '.format(exe=exe)) + os.system("chmod a+x ile_pre.sh") + ile_job.set_executable("ile_pre.sh") # transferred, used as executable +# ile_job.add_condor_cmd('+PreCmd', '"ile_pre.sh"') + + +# if use_osg: +# ile_job.add_condor_cmd("MY.OpenScienceGrid",'True') +# if use_cvmfs_frames: +# transfer_files += ["../local.cache"] + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + # Avoid undesirable hosts in RIFT_AVOID_HOSTS + if 'RIFT_AVOID_HOSTS' in os.environ: + line = os.environ['RIFT_AVOID_HOSTS'] + line = line.rstrip() + if line: + name_list = line.split(',') + for name in name_list: + requirements.append('TARGET.Machine =!= "{}" '.format(name)) + + # Write requirements + # From https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + if not transfer_files is None: + if not isinstance(transfer_files, list): + fname_str=transfer_files + ' '.join(extra_files) + else: + fname_str = ','.join(transfer_files+extra_files) + fname_str=fname_str.strip() + ile_job.add_condor_cmd('transfer_input_files', fname_str) + ile_job.add_condor_cmd('should_transfer_files','YES') + + if not transfer_output_files is None: + if not isinstance(transfer_output_files, list): + fname_str=transfer_output_files + else: + fname_str = ','.join(transfer_output_files) + fname_str=fname_str.strip() + ile_job.add_condor_cmd('transfer_output_files', fname_str) + + # Periodic remove: kill jobs running longer than max runtime + # https://stackoverflow.com/questions/5900400/maximum-run-time-in-condor + if not(max_runtime_minutes is None): + remove_str = 'JobStatus =?= 2 && (CurrentTime - JobStartDate) > ( {})'.format(60*max_runtime_minutes) + ile_job.add_condor_cmd('periodic_remove', remove_str) + + # require_gpus: compose the user's RIFT_REQUIRE_GPUS (used today to block + # incompatible hosts by DeviceName) with the container family's capability + # floor (lowest capability any image in the family supports). Both apply; + # neither is silently dropped. (new convention 'require_gpus = ' specifies + # conditions on GPU properties) + require_gpus_terms = [] + if 'RIFT_REQUIRE_GPUS' in os.environ: + require_gpus_terms.append('({})'.format(os.environ['RIFT_REQUIRE_GPUS'])) + if singularity_is_family and singularity_require_gpus_floor: + require_gpus_terms.append('({})'.format(singularity_require_gpus_floor)) + if require_gpus_terms: + ile_job.add_condor_cmd('require_gpus', ' && '.join(require_gpus_terms)) + + + ### + ### SUGGESTION FROM STUART (for later) + # request_memory = ifthenelse( (LastHoldReasonCode=!=34 && LastHoldReasonCode=!=26), InitialRequestMemory, int(1.5 * NumJobStarts * MemoryUsage) ) + # periodic_release = ((HoldReasonCode =?= 34) || (HoldReasonCode =?= 26)) + # This will automatically release a job that is put on hold for using too much memory with a 50% increased memory request each tim.e + if condor_commands is not None: + for cmd, value in condor_commands.items(): + ile_job.add_condor_cmd(cmd, value) + + return ile_job, ile_sub_name + + + +def write_consolidate_sub_simple(tag='consolidate', exe=None, base=None,target=None,universe="vanilla",arg_str=None,log_dir=None, use_eos=False,ncopies=1,no_grid=False, max_runtime_minutes=120,extra_text='',**kwargs): + """ + Write a submit file for launching a consolidation job + util_ILEdagPostprocess.sh # suitable for ILE consolidation. + arg_str # add argument (used for NR postprocessing, to identify group) + + + """ + + exe = exe or which("util_ILEdagPostprocess.sh") + + # Create executable if needed (using extra_text as flag for now) + # Note 'base' refers to the working diretory here, so we need to back up + base_str = '' + if len(extra_text) > 0: + if not (base is None): + base_0 = base[0] + remove_last_path = '/'.join(base.split('/')[:-1]) + base_str = ' ' + remove_last_path +"/" + + cmdname = "con_sub.sh" + + with open(cmdname,'w') as f: + f.write("#! /usr/bin/env bash\n") + f.write(extra_text+"\n") + extra_args = '' + f.write( exe + " $@ \n") + + st = os.stat(cmdname) + import stat + os.chmod(cmdname, st.st_mode | stat.S_IEXEC) + + exe = base_str + "con_sub.sh" + + + ile_job = CondorDAGJob(universe=universe, executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # Add manual options for input, output + ile_job.add_arg(base) # what directory to load + ile_job.add_arg(target) # where to put the output (label), in CWD + ile_job.add_arg(arg_str) + # + # NO OPTIONS + # +# arg_str = arg_str.lstrip() # remove leading whitespace and minus signs +# arg_str = arg_str.lstrip('-') +# ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line +# ile_job.add_opt(arg_str[2:],'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + + + # + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + + ### + ### SUGGESTION FROM STUART (for later) + # request_memory = ifthenelse( (LastHoldReasonCode=!=34 && LastHoldReasonCode=!=26), InitialRequestMemory, int(1.5 * NumJobStarts * MemoryUsage) ) + # periodic_release = ((HoldReasonCode =?= 34) || (HoldReasonCode =?= 26)) + # This will automatically release a job that is put on hold for using too much memory with a 50% increased memory request each tim.e + + # Periodic remove: kill jobs running longer than max runtime + # https://stackoverflow.com/questions/5900400/maximum-run-time-in-condor + if not(max_runtime_minutes is None): + remove_str = 'JobStatus =?= 2 && (CurrentTime - JobStartDate) > ( {})'.format(60*max_runtime_minutes) + ile_job.add_condor_cmd('periodic_remove', remove_str) + + + return ile_job, ile_sub_name + + + +def write_calpilot_sub(tag='calpilot', exe=None, log_dir=None, universe="vanilla", + working_directory=None, ile_args_file=None, top_fraction=0.05, + max_points=32, request_memory=4096, request_gpu=True, + singularity_image=None, max_runtime_minutes=300, + use_osg=False, use_singularity=False, frames_dir=None, + use_oauth_files=False, transfer_files=None, **kwargs): + """Submit file for a calibration PILOT stage (Option C; see + RIFT/calmarg/DESIGN_adaptive_driver.md): harvest top-lnL points from iteration + $(macroiteration)'s composite, run ILE --calibration-dump-responsibilities on them, + fit + consolidate a cal proposal that seeds wide_{N+1}. Runs util_CalPilotStage.py. + + Macros expected at instantiation: macroiteration, macroiterationprev. + Produces /cal_consolidated_$(macroiteration).npz (consumed by wide_{N+1} ILE via + --calibration-proposal-breadcrumb). + + OSG/container (use_osg/use_singularity): the CALPILOT job runs ILE internally, so it + needs the SAME input set as a wide ILE job -- mirrors write_ILE_sub_simple: + - runs in the singularity image (exe at SINGULARITY_BASE_EXE_DIR); + - a prescript (calpilot_pre.sh) rebuilds local.cache from the transferred frames; + - transfer_input_files = transfer_files (PSD + cal envelopes) + frames_dir + the + composite + args_ile.txt; transfer_output_files = the consolidated breadcrumb; + - the stage args reference BASENAMES (the worker has no shared filesystem). + Refinement (--prev-breadcrumb) is skipped on OSG: the previous breadcrumb is produced + at runtime so it cannot be reliably listed for transfer (esp. iteration 0); each OSG + pilot is an independent cold start (safe -- the fit shrinks toward the prior). + NOTE: untested off-CIT; validate the container + transfer on a real OSG run. + """ + exe = exe or which("util_CalPilotStage.py") + wd = working_directory + on_osg = bool(use_osg or use_singularity) + exe_base = os.path.basename(exe) + frames_local = os.path.basename(frames_dir) if frames_dir else None + if transfer_files is None: + transfer_files = [] + transfer_files = list(transfer_files) + + if use_singularity: + base = os.environ.get('SINGULARITY_BASE_EXE_DIR', '/usr/bin/') + exe = base.rstrip('/') + '/' + exe_base + + # On OSG/container there is no shared FS: the stage's inputs are transferred FLAT into + # the job scratch dir, so reference them by basename and run in '.'; on a local shared + # FS use absolute paths. + if on_osg: + composite_arg = "consolidated_$(macroiteration).composite" + ile_args_arg = os.path.basename(ile_args_file) if ile_args_file else "args_ile.txt" + out_arg = "cal_consolidated_$(macroiteration).npz" + workdir_arg = "." + else: + composite_arg = wd + "/consolidated_$(macroiteration).composite" + ile_args_arg = ile_args_file + out_arg = wd + "/cal_consolidated_$(macroiteration).npz" + workdir_arg = wd + + # Prescript: on OSG, rebuild local.cache (relative paths) from the transferred frames, + # then exec the stage. Mirrors write_ILE_sub_simple's ile_pre.sh. + if on_osg and frames_local: + lalapps_path2cache = os.environ.get('LALAPPS_PATH2CACHE', 'lal_path2cache') + pre = 'calpilot_pre.sh' + with open(pre, 'w') as f: + f.write("#! /bin/bash -xe \n") + f.write("ls {0} | {1} 1> local.cache \n".format(frames_local, lalapps_path2cache)) + f.write("cat local.cache | awk '{print $1, $2, $3, $4}' > local_stripped.cache \n") + f.write("for i in `ls {0}`; do echo {0}/$i; done > base_paths.dat \n".format(frames_local)) + f.write("paste local_stripped.cache base_paths.dat > local_relative.cache \n") + f.write("cp local_relative.cache local.cache \n") + f.write('{0} "$@" \n'.format(exe)) + os.system("chmod a+x " + pre) + transfer_files += [os.path.abspath(pre), frames_dir] + exe = pre + + job = pipeline.CondorDAGJob(universe="vanilla", executable=exe) + sub_name = tag + '.sub' + job.set_sub_file(sub_name) + + # arguments (per-iteration files via condor macros) + job.add_opt("composite", composite_arg) + job.add_opt("ile-args-file", ile_args_arg) + job.add_opt("iteration", "$(macroiteration)") + job.add_opt("output-breadcrumb", out_arg) + if not on_osg: + job.add_opt("prev-breadcrumb", wd + "/cal_consolidated_$(macroiterationprev).npz") + job.add_opt("top-fraction", str(top_fraction)) + job.add_opt("max-points", str(max_points)) + job.add_opt("workdir", workdir_arg) + + uniq_str = "$(macroiteration)-$(cluster)-$(process)" + job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + if default_resolved_env: + job.add_condor_cmd('environment', default_resolved_env) + else: + job.add_condor_cmd('getenv', default_getenv_value) + job.add_condor_cmd('request_memory', str(request_memory) + "M") + if request_gpu: + job.add_condor_cmd('request_GPUs', '1') # the pilot runs ILE (GPU path) + + requirements = [] + if use_singularity and singularity_image: + job.add_condor_cmd('transfer_executable', 'False') + job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') + job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image + '"') + job.add_condor_cmd("MY.flock_local", 'true') + requirements.append("HAS_SINGULARITY=?=TRUE") + elif singularity_image: + job.add_condor_cmd("+SingularityImage", '"' + singularity_image + '"') + if use_oauth_files: + job.add_condor_cmd('use_oauth_services', use_oauth_files) + + if on_osg: + # absolute paths -> condor transfers each to the worker scratch dir by basename, + # which is what the stage args (basenames) reference. + transfer_files += [wd + "/consolidated_$(macroiteration).composite", ile_args_file] + job.add_condor_cmd('transfer_input_files', ','.join(transfer_files)) + job.add_condor_cmd('should_transfer_files', 'YES') + job.add_condor_cmd('when_to_transfer_output', 'ON_EXIT') + job.add_condor_cmd('transfer_output_files', 'cal_consolidated_$(macroiteration).npz') + if requirements: + job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + job.add_condor_cmd('accounting_group', os.environ['LIGO_ACCOUNTING']) + job.add_condor_cmd('accounting_group_user', os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. Add manually to %s !" % sub_name) + if not (max_runtime_minutes is None): + remove_str = 'JobStatus =?= 2 && (CurrentTime - JobStartDate) > ( {})'.format(60 * max_runtime_minutes) + job.add_condor_cmd('periodic_remove', remove_str) + return job, sub_name + + +def write_extrconsolidate_sub(tag='extrconsolidate', exe=None, log_dir=None, universe="local", + working_directory=None, request_memory=2048, select="lnL", + max_runtime_minutes=30, **kwargs): + """Submit file for the EXTRINSIC handoff consolidation stage (see + RIFT/calmarg/DESIGN_extrinsic_handoff.md). Runs util_ExtrinsicConsolidate.py. + + After iteration $(macroiteration)'s wide ILE jobs each drop a per-event extrinsic + proposal breadcrumb (extr_proposal_$(macroiteration)_.npz, written by ILE + --extrinsic-proposal-output), this job picks the single most representative one and + writes /extr_consolidated_$(macroiteration).npz , which SEEDS the wide ILE jobs of + iteration $(macroiteration)+1 via --extrinsic-proposal-breadcrumb. + + Runs in the LOCAL universe on the submit node: it is pure-python file selection (no GPU, + no ILE, no container, no frames). On OSG the per-event ILE outputs are transferred back + to /iteration__ile on the submit node (ILE's transfer_output_files default), so a + local-universe job reads them directly from the shared FS -- no per-event input transfer + (which condor cannot glob anyway). The output breadcrumb is ALWAYS written (empty if no + valid input), so the next iteration's seed/transfer never fails. + + Macros expected at instantiation: macroiteration. + """ + exe = exe or which("util_ExtrinsicConsolidate.py") + wd = working_directory + job = pipeline.CondorDAGJob(universe=universe, executable=exe) + sub_name = tag + '.sub' + job.set_sub_file(sub_name) + + # per-event proposals land in the ILE initialdir; pick best -> consolidated breadcrumb in + # wd (where wide_{N+1} ILE's --extrinsic-proposal-breadcrumb path points). + job.add_opt("input-glob", wd + "/iteration_$(macroiteration)_ile/extr_proposal_$(macroiteration)_*.npz") + job.add_opt("output", wd + "/extr_consolidated_$(macroiteration).npz") + job.add_opt("iteration", "$(macroiteration)") + job.add_opt("select", select) + + uniq_str = "$(macroiteration)-$(cluster)-$(process)" + job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + if default_resolved_env: + job.add_condor_cmd('environment', default_resolved_env) + else: + job.add_condor_cmd('getenv', default_getenv_value) + job.add_condor_cmd('request_memory', str(request_memory) + "M") + + try: + job.add_condor_cmd('accounting_group', os.environ['LIGO_ACCOUNTING']) + job.add_condor_cmd('accounting_group_user', os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. Add manually to %s !" % sub_name) + if not (max_runtime_minutes is None): + remove_str = 'JobStatus =?= 2 && (CurrentTime - JobStartDate) > ( {})'.format(60 * max_runtime_minutes) + job.add_condor_cmd('periodic_remove', remove_str) + return job, sub_name + + +def write_unify_sub_simple(tag='unify', exe=None, base=None,target=None,universe="vanilla",arg_str=None,log_dir=None, use_eos=False,ncopies=1,no_grid=False, max_runtime_minutes=60,extra_text='',**kwargs): + """ + Write a submit file for launching a consolidation job + util_ILEdagPostprocess.sh # suitable for ILE consolidation. + arg_str # add argument (used for NR postprocessing, to identify group) + + + """ + + exe = exe or which("util_CleanILE.py") # like cat, but properly accounts for *independent* duplicates. (Danger if identical). Also strips large errors + + # Write unify.sh + # - problem of globbing inside condor commands + # - problem that *.composite files from intermediate results will generally NOT be present + cmdname ='unify.sh' + base_str = '' + if not (base is None): + base_str = ' ' + base +"/" + with open(cmdname,'w') as f: + f.write("#! /usr/bin/env bash\n") + if len(extra_text) > 0: + f.write(extra_text+"\n") + f.write( "ls " + base_str+"*.composite 1>&2 \n") # write filenames being concatenated to stderr + # Sometimes we need to pass --eccentricity or --tabular-eos-file etc to util_CleanILE.py + extra_args = '' + if arg_str: + extra_args = arg_str + f.write( exe + extra_args+ base_str+ "*.composite \n") + # Backstop code for untify.sh + f.write("""ret_value=$? +if [ $ret_value -eq 0 ]; then + exit 0 +else + cat {} +fi +""".format(base_str+"*.composite")) + st = os.stat(cmdname) + import stat + os.chmod(cmdname, st.st_mode | stat.S_IEXEC) + + + ile_job = CondorDAGJob(universe=universe, executable=base_str+cmdname) # force full prefix + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # Add manual options for input, output +# ile_job.add_arg('*.composite') # what to do + + # + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file(target) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + # Periodic remove: kill jobs running longer than max runtime + # https://stackoverflow.com/questions/5900400/maximum-run-time-in-condor + if not(max_runtime_minutes is None): + remove_str = 'JobStatus =?= 2 && (CurrentTime - JobStartDate) > ( {})'.format(60*max_runtime_minutes) + ile_job.add_condor_cmd('periodic_remove', remove_str) + + return ile_job, ile_sub_name + +def write_convert_sub(tag='convert', exe=None, file_input=None,file_output=None,universe="vanilla",arg_str='',log_dir=None, use_eos=False,ncopies=1, no_grid=False,max_runtime_minutes=120,**kwargs): + """ + Write a submit file for launching a 'convert' job + convert_output_format_ile2inference + + """ + + exe = exe or which("convert_output_format_ile2inference") # like cat, but properly accounts for *independent* duplicates. (Danger if identical). Also strips large errors + + ile_job = CondorDAGJob(universe=universe, executable=exe) + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + if not(arg_str is None or len(arg_str)<2): + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line +# ile_job.add_opt(arg_str[2:],'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + ile_job.add_arg(file_input) + + # + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file(file_output) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + # no grid + if no_grid: # very aggressively enforce staying on the current filesystem! + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"none"') + ile_job.add_condor_cmd("MY.flock_local",'true') + try: + os.system("condor_config_val UID_DOMAIN > uid_domain.txt") + with open("uid_domain.txt", 'r') as f: + uid_domain = f.readline().strip() + requirements.append(' UidDomain =?= "{}"'.format(uid_domain)) + except: + True + + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + # Periodic remove: kill jobs running longer than max runtime + # https://stackoverflow.com/questions/5900400/maximum-run-time-in-condor + if not(max_runtime_minutes is None): + remove_str = 'JobStatus =?= 2 && (CurrentTime - JobStartDate) > ( {})'.format(60*max_runtime_minutes) + ile_job.add_condor_cmd('periodic_remove', remove_str) + + return ile_job, ile_sub_name + + +def write_test_sub(tag='converge', exe=None,samples_files=None, base=None,target=None,universe="target",arg_str=None,log_dir=None, use_eos=False,ncopies=1, no_grid=False,**kwargs): + """ + Write a submit file for launching a convergence test job + + """ + + exe = exe or which("convergence_test_samples.py") + + ile_job = CondorDAGJob(universe=universe, executable=exe) + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line +# ile_job.add_opt(arg_str[2:],'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + + # Add options for two parameter files + for name in samples_files: +# ile_job.add_opt("samples",name) # do not add in usual fashion, because otherwise the key's value is overwritten + ile_job.add_opt("samples " + name,'') + + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file(target) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + try: + os.system("condor_config_val UID_DOMAIN > uid_domain.txt") + with open("uid_domain.txt", 'r') as f: + uid_domain = f.readline().strip() + requirements.append(' UidDomain =?= "{}"'.format(uid_domain)) + except: + True + + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + return ile_job, ile_sub_name + +def write_refine_sub(tag='refine', exe=None, input_net=None,input_grid=None,output=None,universe="vanilla",out_dir=None,log_dir=None, use_eos=False,ncopies=1,arg_str=None,request_memory=1024,arg_vals=None, target=None,no_grid=False,**kwargs): + """ + Write a submit file for creating a refined CIP grid for NR-based runs. + """ + + exe = exe or which("util_TestSpokesIO.py") + + ile_job = CondorDAGJob(universe="vanilla", executable=exe) + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + # ile_job.add_opt(arg_str[2:],'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + + ile_job.add_opt("fname-dat", input_net) + ile_job.add_opt("fname", input_grid) + ile_job.add_opt("save-refinement-fname", output) + + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file(target) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to refine.sub !") + + return ile_job, ile_sub_name + +def write_plot_sub(tag='converge', exe=None,samples_files=None, base=None,target=None,arg_str=None,log_dir=None, use_eos=False,ncopies=1, **kwargs): + """ + Write a submit file for launching a final plot. Note the user can in principle specify several samples (e.g., several iterations, if we want to diagnose them) + + """ + + exe = exe or which("plot_posterior_corner.py") + + ile_job = CondorDAGJob(universe="vanilla", executable=exe) + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line +# ile_job.add_opt(arg_str[2:],'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + + # Add options for two parameter files + for name in samples_files: +# ile_job.add_opt("samples",name) # do not add in usual fashion, because otherwise the key's value is overwritten + ile_job.add_opt("posterior-file " + name,'') + + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file(target) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + return ile_job, ile_sub_name + + + + +def write_init_sub(tag='gridinit', exe=None,arg_str=None,log_dir=None, use_eos=False,ncopies=1, **kwargs): + """ + Write a submit file for launching a grid initialization job. + Note this routine MUST create whatever files are needed by the ILE iteration + + """ + + exe = exe or which("util_ManualOverlapGrid.py") + + ile_job = CondorDAGJob(universe="vanilla", executable=exe) + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line +# ile_job.add_opt(arg_str[2:],'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + return ile_job, ile_sub_name + + + +def write_psd_sub_BW_monoblock(tag='PSD_BW_mono', exe=None, log_dir=None, ncopies=1,arg_str=None,request_memory=4096,arg_vals=None, transfer_files=None,transfer_output_files=None,use_singularity=False,use_osg=False,singularity_image=None,frames_dir=None,cache_file=None,psd_length=4,srate=4096,data_start_time=None,event_time=None,universe='local',no_grid=False,**kwargs): + """ + Write a submit file for constructing the PSD using BW + Modern argument syntax for BW + Note that *all ifo-specific results must be set outside this loop*, to work sensibly, and passed as an argument + + Inputs: + - channel_dict['H1'] = [channel_name, flow_ifo] + Outputs: + - An instance of the CondorDAGJob that was generated for BW + """ + exe = exe or which("BayesWave") + if exe is None: + print(" BayesWave not available, hard fail ") + sys.exit(0) + frames_local = None + + ile_job = CondorDAGJob(universe=universe, executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + + + + requirements =[] + # + # Logging options + # + uniq_str = "$(macroevent)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + + # + # Loop over IFOs + # You should only have one, in the workflow for which this is intended + # Problem: + ile_job.add_arg("$(macroargument0)") + + + # + # Add mandatory options + ile_job.add_opt('Niter', '1000100') + ile_job.add_opt('Nchain', '20') + ile_job.add_opt('Dmax', '200') # limit number of dimensions in model + ile_job.add_opt('resume', '') + ile_job.add_opt('progress', '') + ile_job.add_opt('checkpoint', '') + ile_job.add_opt('bayesLine', '') + ile_job.add_opt('cleanOnly', '') + ile_job.add_opt('updateGeocenterPSD', '') + ile_job.add_opt('dataseed', '1234') # make reproducible + + ile_job.add_opt('trigtime', str(event_time)) + ile_job.add_opt('psdstart', str(event_time-(psd_length-2))) + ile_job.add_opt('segment-start', str(event_time-(psd_length-2))) + ile_job.add_opt('seglen', str(psd_length)) + ile_job.add_opt('psdlength', str(psd_length)) + ile_job.add_opt('srate', str(srate)) + ile_job.add_opt('outputDir', 'output_$(ifo)') + + + + + + # Add lame initial argument + if "output_file" in kwargs and kwargs["output_file"] is not None: + # + # Need to modify the output file so it's unique + # + ofname = kwargs["output_file"].split(".") + ofname, ext = ofname[0], ".".join(ofname[1:]) + ile_job.add_file_opt("output-file", "%s-%s.%s" % (ofname, uniq_str, ext)) + del kwargs["output_file"] + if "save_samples" in kwargs and kwargs["save_samples"] is True: + ile_job.add_opt("save-samples", None) + del kwargs["save_samples"] + + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + + # Write requirements + # From https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + return ile_job, ile_sub_name + + +def write_psd_sub_BW_step1(tag='PSD_BW_post', exe=None, log_dir=None, ncopies=1,arg_str=None,request_memory=4096,arg_vals=None, transfer_files=None,transfer_output_files=None,use_singularity=False,use_osg=False,singularity_image=None,frames_dir=None,cache_file=None,channel_dict=None,psd_length=4,srate=4096,data_start_time=None,event_time=None,**kwargs): + """ + Write a submit file for launching jobs to marginalize the likelihood over intrinsic parameters. + + Inputs: + - channel_dict['H1'] = [channel_name, flow_ifo] + Outputs: + - An instance of the CondorDAGJob that was generated for ILE + """ + exe = exe or which("BayesWavePost") + if exe is None: + print(" BayesWavePost not available, hard fail ") + import sys + sys.exit(0) + frames_local = None + + ile_job = CondorDAGJob(universe="vanilla", executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + + requirements =[] + # + # Logging options + # + uniq_str = "$(macroevent)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + # + # Add mandatory options + ile_job.add_opt('checkpoint', '') + ile_job.add_opt('bayesLine', '') + ile_job.add_opt('cleanOnly', '') + ile_job.add_opt('updateGeocenterPSD', '') + ile_job.add_opt('Nchain', '20') + ile_job.add_opt('Niter', '4000000') + ile_job.add_opt('Nbayesline', '2000') + ile_job.add_opt('dataseed', '1234') # make reproducible + + ile_job.add_opt('trigtime', str(event_time)) + ile_job.add_opt('psdstart', str(event_time-(psd_length-2))) + ile_job.add_opt('segment-start', str(event_time-(psd_length-2))) + ile_job.add_opt('seglen', str(psd_length)) + ile_job.add_opt('srate', str(srate)) + + + + # + # Loop over IFOs + # Not needed, can do one job per PSD +# ile_job.add_opt("ifo","$(ifo)") +# ile_job.add_opt("$(ifo)-cache",cache_file) + for ifo in channel_dict: + channel_name, channel_flow = channel_dict[ifo] + ile_job.add_arg("--ifo "+ ifo) # need to prevent overwriting! + ile_job.add_opt(ifo+"-channel", ifo+":"+channel_name) + ile_job.add_opt(ifo+"-cache", cache_file) + ile_job.add_opt(ifo+"-flow", str(channel_flow)) + + # Add lame initial argument + if "output_file" in kwargs and kwargs["output_file"] is not None: + # + # Need to modify the output file so it's unique + # + ofname = kwargs["output_file"].split(".") + ofname, ext = ofname[0], ".".join(ofname[1:]) + ile_job.add_file_opt("output-file", "%s-%s.%s" % (ofname, uniq_str, ext)) + del kwargs["output_file"] + if "save_samples" in kwargs and kwargs["save_samples"] is True: + ile_job.add_opt("save-samples", None) + del kwargs["save_samples"] + + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + + # Write requirements + # From https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + return ile_job, ile_sub_name + + +def write_psd_sub_BW_step0(tag='PSD_BW', exe=None, log_dir=None, ncopies=1,arg_str=None,request_memory=4096,arg_vals=None, transfer_files=None,transfer_output_files=None,use_singularity=False,use_osg=False,singularity_image=None,frames_dir=None,cache_file=None,channel_dict=None,psd_length=4,srate=4096,data_start_time=None,event_time=None,**kwargs): + """ + Write a submit file for launching jobs to marginalize the likelihood over intrinsic parameters. + + Inputs: + - channel_dict['H1'] = [channel_name, flow_ifo] + Outputs: + - An instance of the CondorDAGJob that was generated for ILE + """ + exe = exe or which("BayesWave") + if exe is None: + print(" BayesWave not available, hard fail ") + sys.exit(0) + frames_local = None + + ile_job = CondorDAGJob(universe="vanilla", executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + + requirements =[] + # + # Logging options + # + uniq_str = "$(macroevent)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + # + # Add mandatory options + ile_job.add_opt('checkpoint', '') + ile_job.add_opt('bayesLine', '') + ile_job.add_opt('cleanOnly', '') + ile_job.add_opt('updateGeocenterPSD', '') + ile_job.add_opt('Nchain', '20') + ile_job.add_opt('Niter', '4000000') + ile_job.add_opt('Nbayesline', '2000') + ile_job.add_opt('dataseed', '1234') # make reproducible + + ile_job.add_opt('trigtime', str(event_time)) + ile_job.add_opt('psdstart', str(event_time-(psd_length-2))) + ile_job.add_opt('segment-start', str(event_time-(psd_length-2))) + ile_job.add_opt('seglen', str(psd_length)) + ile_job.add_opt('srate', str(srate)) + + + + # + # Loop over IFOs + for ifo in channel_dict: + channel_name, channel_flow = channel_dict[ifo] + ile_job.add_arg("--ifo " + ifo) + ile_job.add_opt(ifo+"-channel", ifo+":"+channel_name) + ile_job.add_opt(ifo+"-cache", cache_file) + ile_job.add_opt(ifo+"-flow", str(channel_flow)) + ile_job.add_opt(ifo+"-timeslide", str(0.0)) + + + # Add lame initial argument + if "output_file" in kwargs and kwargs["output_file"] is not None: + # + # Need to modify the output file so it's unique + # + ofname = kwargs["output_file"].split(".") + ofname, ext = ofname[0], ".".join(ofname[1:]) + ile_job.add_file_opt("output-file", "%s-%s.%s" % (ofname, uniq_str, ext)) + del kwargs["output_file"] + if "save_samples" in kwargs and kwargs["save_samples"] is True: + ile_job.add_opt("save-samples", None) + del kwargs["save_samples"] + + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + + # Write requirements + # From https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + return ile_job, ile_sub_name + + +def write_resample_sub(tag='resample', exe=None, file_input=None,file_output=None,universe="vanilla",arg_str='',log_dir=None, use_eos=False,ncopies=1, no_grid=False,**kwargs): + """ + Write a submit file for launching a 'resample' job + util_ResampleILEOutputWithExtrinsic.py + + """ + + exe = exe or which("util_ResampleILEOutputWithExtrinsic.py") # like cat, but properly accounts for *independent* duplicates. (Danger if identical). Also strips large errors + + ile_job = CondorDAGJob(universe=universe, executable=exe) + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + if not(arg_str is None or len(arg_str)<2): + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + ile_job.add_opt(arg_str,'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line +# ile_job.add_opt(arg_str[2:],'') # because we must be idiotic in how we pass arguments, I strip off the first two elements of the line + ile_job.add_opt('fname',file_input) + ile_job.add_opt('fname-out',file_output) + + # + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file(file_output) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + return ile_job, ile_sub_name + + + +def write_cat_sub(tag='cat', exe=None, file_prefix=None,file_postfix=None,file_output=None,universe="vanilla",arg_str='',log_dir=None, use_eos=False,ncopies=1, no_grid=False,**kwargs): + """ + Write a submit file for launching a 'resample' job + util_ResampleILEOutputWithExtrinsic.py + + """ + + exe = exe or which("find") # like cat, but properly accounts for *independent* duplicates. (Danger if identical). Also strips large errors + exe_switch = which("switcheroo") # tool for patterend search-replace, to fix first line of output file + + cmdname = 'catjob.sh' + with open(cmdname,'w') as f: + f.write("#! /bin/bash\n") + f.write(exe+" . -name '"+file_prefix+"*"+file_postfix+r"' -exec cat {} \; | sort -r | uniq > "+file_output+";\n") + f.write(exe_switch + " 'm1 ' '# m1 ' "+file_output) # add standard prefix + os.system("chmod a+x "+cmdname) + + ile_job = CondorDAGJob(universe=universe, executable='catjob.sh') + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + try: + os.system("condor_config_val UID_DOMAIN > uid_domain.txt") + with open("uid_domain.txt", 'r') as f: + uid_domain = f.readline().strip() + requirements.append(' UidDomain =?= "{}"'.format(uid_domain)) + except: + True + + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + +# ile_job.add_arg(" . -name '" + file_prefix + "*" +file_postfix+"' -exec cat {} \; ") + + # + # Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + return ile_job, ile_sub_name + + + +def write_consolidate_distance_grids_sub(tag='consolidate_dgrid', exe=None, + input_glob=None, file_output=None, + search_dir='.', universe='local', + log_dir=None, no_grid=False, **kwargs): + """Consolidate per-event .dgrid (Plan A) / .dslice (Plan B) files. + + Wraps ``util_ConsolidateDistanceGrids.py``: writes a thin shell driver + that runs the consolidator over ``input_glob`` (a find-style pattern, e.g. + ``EXTR_out.xml_*_.dgrid``) in ``search_dir`` and emits the concatenated + table at ``file_output``. Mirrors ``write_cat_sub`` so it slots into the + same post-extrinsic part of the DAG. + """ + exe = exe or which("util_ConsolidateDistanceGrids.py") + if not exe: + exe = "util_ConsolidateDistanceGrids.py" + + cmdname = tag + '.sh' + with open(cmdname, 'w') as f: + f.write("#! /bin/bash\n") + f.write("set -e\n") + f.write("cd " + search_dir + "\n") + # --allow-empty keeps the post-extrinsic job from failing the DAG if a + # re-run already consumed the per-event files or none were produced. + f.write(exe + " --input-glob '" + input_glob + "'" + " --output " + file_output + " --allow-empty\n") + os.system("chmod a+x " + cmdname) + + ile_job = CondorDAGJob(universe=universe, executable=cmdname) + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES", '"nogrid"') + ile_job.add_condor_cmd("MY.flock_local", 'true') + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + uniq_str = "$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + try: + ile_job.add_condor_cmd('accounting_group', os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user', os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + return ile_job, ile_sub_name + + +def write_convertpsd_sub(tag='convert_psd', exe=None, ifo=None,file_input=None,target_dir=None,arg_str='',log_dir=None, universe='local',**kwargs): + """ + Write script to convert PSD from one format to another. Needs to be called once per PSD file being used. + """ + + exe = exe or which("convert_psd_ascii2xml") # like cat, but properly accounts for *independent* duplicates. (Danger if identical). Also strips large errors + ile_job = CondorDAGJob(universe=universe, executable=exe) + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + ile_job.add_opt("fname-psd-ascii",file_input) + ile_job.add_opt("ifo",ifo) + ile_job.add_arg("--conventional-postfix") + + # + # Logging options + # + uniq_str = "$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + if not (target_dir is None): + # Copy output PSD into place + ile_job.add_condor_cmd("MY.PostCmd", '" cp '+ifo+'-psd.xml.gz ' + target_dir +'"') + + ile_job.add_condor_cmd('getenv', default_getenv_value) + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + return ile_job, ile_sub_name + + +def write_joingrids_sub(tag='join_grids', exe=None, universe='vanilla', input_pattern=None,target_dir=None,output_base=None,log_dir=None,n_explode=1, gzip="/usr/bin/gzip", old_add=False, old_style_add=False,no_grid=False,extra_text='', **kwargs): + """ + Write script to merge CIP 'overlap-grid-(iteration)-*.xml.gz results. Issue is that + """ + default_add = "util_RandomizeOverlapOrder.py" + if old_style_add: + default_add = "igwn_ligolw_add" + + exe = exe or which(default_add) + if not(exe): + exe = "igwn_ligolw_add" # go back to fallback if there is a weird disaster -- eg we are using an old-style install before this was updated + + working_dir = log_dir.replace("/logs", '') # assumption about workflow/naming! Danger! + + fname_out =target_dir + "/" +output_base + ".xml.gz" + if n_explode ==1: # we are really doing a glob match + fname_out = fname_out.replace('$(macroiteration)','$1') + fname_out = fname_out.replace('$(macroiterationnext)','$2') + alt_work_dir = working_dir.replace('$(macroiteration)','$1') + alt_out = output_base.replace('$(macroiterationnext)','$2') + extra_arg = '' + if old_add: + extra_arg = " --ilwdchar-compat " # should never be used anymore + with open("join_grids.sh",'w') as f: + f.write("#! /bin/bash \n") + f.write(r""" +# merge using glob command called from shell +{} +{} {} --output {} {}/{}*.xml.gz +""".format(extra_text,exe,extra_arg,fname_out,alt_work_dir,alt_out)) + os.system("chmod a+x join_grids.sh") + exe = target_dir + "/join_grids.sh" + + ile_job = CondorDAGJob(universe=universe, executable=exe) + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + try: + os.system("condor_config_val UID_DOMAIN > uid_domain.txt") + with open("uid_domain.txt", 'r') as f: + uid_domain = f.readline().strip() + requirements.append(' UidDomain =?= "{}"'.format(uid_domain)) + except: + True + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + if n_explode > 1: + ile_job.add_arg("--output="+fname_out) + + + # + # Logging options + # + uniq_str = "$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) +# ile_job.set_stdout_file(fname_out) + +# ile_job.add_condor_cmd("MY.PostCmd", ' "' + gzip + ' ' +fname_out + '"') + + explode_str = "" + explode_str += " {}/{}.xml.gz ".format(working_dir,output_base) # base result from fitting job + if n_explode >1: + for indx in np.arange(n_explode): + explode_str+= " {}/{}-{}.xml.gz ".format(working_dir,output_base,indx) + ile_job.add_arg(explode_str) + else: + ile_job.add_arg(" $(macroiteration) $(macroiterationnext) ") +# explode_str += " {}/{}-*.xml.gz ".format(working_dir,output_base) # if n_explode is 1 or 0, use a matching pattern +# ile_job.add_arg("overlap-grid*.xml.gz") # working in our current directory + + if old_add and n_explode > 1: + ile_job.add_opt("ilwdchar-compat",'') # needed? + + ile_job.add_condor_cmd('getenv', default_getenv_value) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + return ile_job, ile_sub_name + + + + + +def write_subdagILE_sub(tag='subdag_ile', full_path_name=True, exe=None, universe='vanilla', submit_file=None,input_pattern=None,target_dir=None,output_suffix=None,log_dir=None,sim_xml=None, **kwargs): + + """ + Write script to convert PSD from one format to another. Needs to be called once per PSD file being used. + """ + exe = exe or which("create_ile_sub_dag.py") + subfile = submit_file or 'ILE.sub' + if full_path_name and target_dir: + if subfile[0]!= '/': # if not already a full path + subfile = target_dir + "/"+subfile + + ile_job = CondorDAGJob(universe=universe, executable=exe) + + ile_sub_name = tag + '.sub' +# if full_path_name and target_dir: +# ile_sub_name = target_dir +"/" + ile_sub_name + ile_job.set_sub_file(ile_sub_name) + + ile_job.add_arg("--target-dir "+target_dir) + ile_job.add_arg("--output-suffix "+output_suffix) + ile_job.add_arg("--submit-script "+subfile) + ile_job.add_arg("--macroiteration $(macroiteration)") + ile_job.add_arg("--sim-xml "+sim_xml) + + working_dir = log_dir.replace("/logs", '') # assumption about workflow/naming! Danger! + + # + # Logging options + # + uniq_str = "$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) +# ile_job.set_stdout_file(fname_out) + +# ile_job.add_condor_cmd("MY.PostCmd", ' "' + gzip + ' ' +fname_out + '"') + + ile_job.add_condor_cmd('getenv', default_getenv_value) + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + return ile_job, ile_sub_name + + +def write_calibration_uncertainty_reweighting_sub(tag='Calib_reweight', exe=None, log_dir=None, ncopies=1,request_memory=8192,time_marg=True,pickle_file=None,posterior_file=None,universe='vanilla',no_grid=False,ile_args=None,n_cal=100,use_osg=False,use_oauth_files=False,use_singularity=False,singularity_image=None,transfer_files=None,**kwargs): + """ + Write a submit file for launching jobs to reweight final posterior samples due to calibration uncertainty + + Inputs: + - posterior samples, event pickle file (generated by Bilby) + Outputs: + - reweighted samples due to calibration uncertainty and corresponding weights + """ + if use_singularity and (singularity_image == None) : + print(" FAIL : Need to specify singularity_image to use singularity ") + sys.exit(0) + if use_singularity and (transfer_files == None) : + print(" FAIL : Need to specify transfer_files to use singularity at present! (we will append the prescript; you should transfer any PSDs as well as the grid file ") + sys.exit(0) + + singularity_image_used = "{}".format(singularity_image) # make copy + extra_files = [] + if singularity_image: + if 'osdf:' in singularity_image: + singularity_image_used = "./{}".format(singularity_image.split('/')[-1]) + extra_files += [singularity_image] + + + + exe = exe or which("calibration_reweighting.py") + if exe is None: + print(" Calibration Reweighting code not available. ") + sys.exit(0) + if use_singularity: + exe_base = os.path.basename(exe) +# print((" Executable: name breakdown ", path_split, " from ", exe)) + singularity_base_exe_path = "/opt/lscsoft/rift/MonteCarloMarginalizeCode/Code/" # should not hardcode this ...! + if 'SINGULARITY_BASE_EXE_DIR' in list(os.environ.keys()) : + singularity_base_exe_path = os.environ['SINGULARITY_BASE_EXE_DIR'] + else: +# singularity_base_exe_path = "/opt/lscsoft/rift/MonteCarloMarginalizeCode/Code/" # should not hardcode this ...! + singularity_base_exe_path = "/usr/bin/" # should not hardcode this ...! + exe=singularity_base_exe_path + exe_base + + ile_job = CondorDAGJob(universe="vanilla", executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) +# if not(request_disk is False): +# ile_job.add_condor_cmd('request_disk', str(request_disk)) + + + requirements =[] + # Containerization basics + if use_singularity: + # Compare to https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py + ile_job.add_condor_cmd('request_CPUs', str(1)) + ile_job.add_condor_cmd('transfer_executable', 'False') + ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') + ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') + ile_job.add_condor_cmd("transfer_output_files", "weight_files") + requirements.append("HAS_SINGULARITY=?=TRUE") + print(" WARNING: cal reweighting requires bilby. Directories are moved to cal_evelopes") +# os.system("condor_config_val UID_DOMAIN > uid_domain.txt") +# with open("uid_domain.txt", 'r') as f: +# uid_domain = f.readline().strip() +# requirements.append(' UidDomain =?= "{}"'.format(uid_domain)) + if use_oauth_files: + # we are using some authentication to retrieve files from the file transfer list, for example, from distributed hosts, not just submit. eg urls provided + ile_job.add_condor_cmd('use_oauth_services',use_oauth_files) + if use_singularity or use_osg: + # Set up file transfer options + ile_job.add_condor_cmd("when_to_transfer_output",'ON_EXIT') + + # + # Logging options + # + uniq_str = "$(macroevent)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + # + # Add mandatory options + pickle_file_arg = str(pickle_file) + post_file_arg = str(posterior_file) + if use_osg: + transfer_files += [pickle_file_arg , post_file_arg] + pickle_file_arg = os.path.basename(pickle_file_arg) + post_file_arg = os.path.basename(post_file_arg) + if os.path.exists('cal_envelopes'): + transfer_files += ['./cal_envelopes'] # note initial dir configured so this will work + ile_job.add_arg(" --use_local_cal_files ") + ile_job.add_opt('data_dump_file', str(pickle_file_arg)) + ile_job.add_opt('posterior_sample_file', str(post_file_arg)) + ile_job.add_opt('number_of_calibration_curves', str(n_cal)) + ile_job.add_opt('reevaluate_likelihood', 'True') + ile_job.add_opt('use_rift_samples', 'True') + if time_marg: + # problem with this argument: 'False' is often parsed as 'True' by argparsing (weird). Default is 'false' + ile_job.add_opt('time_marginalization', str(time_marg)) + + lmax=None + approx=None + if ile_args: + ile_args_split = ile_args.split('--') + fmin_list = [] + fmin_template = None + for line in ile_args_split: + line_split = line.split() + if len(line_split)>1: + if line_split[0] == 'fmin-ifo': + fmin_list += [line_split[1]] + if line_split[0] == 'fmin-template': + fmin_template = line_split[1] + elif line_split[0] == 'l-max': + lmax = int(line_split[1]) + elif line_split[0] == 'approx': + approx = line_split[1] +# fmin = np.min(fmin_list) + if fmin_template: + ile_job.add_arg(" --fmin {} ".format(fmin_template)) # code will fail without this, and it is always written anyways, but + if lmax: + ile_job.add_arg(" --l-max {} ".format(lmax)) + if approx: + ile_job.add_arg(" --waveform_approximant {} ".format(approx)) + # + # Add normal arguments + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + getenv_calmarg = 'PATH,PYTHONPATH,LIBRARY_PATH,LD_LIBRARY_PATH,*RIFT*' # local ! + if not(default_getenv_value == 'True'): + ile_job.add_condor_cmd('getenv', default_getenv_value) + else: + ile_job.add_condor_cmd('getenv', getenv_calmarg) + # use a smaller request initially, then increase. Should improve throughput + ile_job.add_condor_cmd('request_memory', str(request_memory/2)+"M") + ile_job.add_condor_cmd('retry_request_memory', str(request_memory)+"M") + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + + # Write requirements + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + # Write transfer file list. Will handle any surrogates + pickle/container files. + if not transfer_files is None: + if not isinstance(transfer_files, list): + fname_str=transfer_files + ' '.join(extra_files) + else: + fname_str = ','.join(transfer_files+extra_files) + fname_str=fname_str.strip() + ile_job.add_condor_cmd('transfer_input_files', fname_str) + ile_job.add_condor_cmd('should_transfer_files','YES') + + # Stream log info + if not ('RIFT_NOSTREAM_LOG' in os.environ): + ile_job.add_condor_cmd("stream_error",'True') + ile_job.add_condor_cmd("stream_output",'True') + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + return ile_job, ile_sub_name + +def bilby_prior_dict_string_from_mc_q(mc_range,dmax_Mpc): + out_str = """chirp-mass: bilby.gw.prior.UniformInComponentsChirpMass(minimum={}, maximum={}, name='chirp_mass', boundary=None), mass-ratio: bilby.gw.prior.UniformInComponentsMassRatio(minimum=0.05, maximum=1.0, name='mass_ratio', latex_label='$q$', unit=None, boundary=None), mass-1: Constraint(minimum=1, maximum=1000, name='mass_1', latex_label='$m_1$', unit=None), mass-2: Constraint(minimum=1, maximum=1000, name='mass_2', latex_label='$m_2$', unit=None), a-1: Uniform(minimum=0, maximum=0.99, name='a_1', latex_label='$a_1$', unit=None, boundary=None), a-2: Uniform(minimum=0, maximum=0.99, name='a_2', latex_label='$a_2$', unit=None, boundary=None), tilt-1: Sine(minimum=0, maximum=3.141592653589793, name='tilt_1'), tilt-2: Sine(minimum=0, maximum=3.141592653589793, name='tilt_2'), phi-12: Uniform(minimum=0, maximum=6.283185307179586, name='phi_12', boundary='periodic'), phi-jl: Uniform(minimum=0, maximum=6.283185307179586, name='phi_jl', boundary='periodic'), luminosity-distance: PowerLaw(alpha=2, minimum=10, maximum={}, name='luminosity_distance', latex_label='$d_L$', unit='Mpc', boundary=None), theta-jn: Sine(minimum=0, maximum=3.141592653589793, name='theta_jn'), psi: Uniform(minimum=0, maximum=3.141592653589793, name='psi', boundary='periodic'), phase: Uniform(minimum=0, maximum=6.283185307179586, name='phase', boundary='periodic'), dec: Cosine(name='dec'), ra: Uniform(name='ra', minimum=0, maximum=2 * np.pi, boundary='periodic') +""".format(mc_range[0],mc_range[1],dmax_Mpc) + out_str = "{" + out_str.rstrip() + "}" + return out_str + +def write_bilby_pickle_sub(tag='Bilby_pickle', exe=None, universe='local', log_dir=None, ncopies=1,request_memory=4096,bilby_ini_file=None,no_grid=False,frames_dir=None,cache_file=None,ile_args=None,**kwargs): + """ + Write a submit file for launching a job to generate a pickle file based off a bilby ini file; needed for reweight final posterior samples due to calibration uncertainty + + Inputs: + - bilby ini file + Outputs: + - pickle file of event settings; needed as input for calibration reweighting + + Notes: + - local universe is generally safer: we need access to frame files in a standard location (typically datafind returns cvmfs, etc). That may not be available on remote nodes. + """ + exe = exe or which("bilby_pipe_generation") + if exe is None: + print(" Pickle generation code unavailable. ") + sys.exit(0) + ile_job = CondorDAGJob(universe=universe, executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # + #Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + + # Add manual options for input, output. Hopefully this all happens in order as needed, if not we will just concatenate + # + ile_job.add_arg(str(bilby_ini_file)) # needs to be a bilby ini file for the particular event being analyzed + ile_job.add_arg(' --data-dump-file calmarg/data/calmarg_data_dump.pickle') + + # Problem: bilby ini file may not have 'data-dict', in which case we need to backstop it with data from 'frames_dir' or 'cache_file' + # Problem: bilby ini file does not have sections. + # Workaround: https://stackoverflow.com/questions/2885190/using-configparser-to-read-a-file-without-section-name + config = configparser.ConfigParser() + config.optionxform=str # force preserve case! Important for --choose-data-LI-seglen + with open(bilby_ini_file) as stream: + config.read_string("[top]\n" + stream.read()) + bilby_items = dict(config["top"]) + # Backstop horrible parsing situations where it returns a string and not dict + if not(isinstance(bilby_items['channel-dict'], dict)): + # Safer string parsing - in case no comma at end, etc + bilby_items['channel-dict'] = bilby_ish_string_to_dict(bilby_items['channel-dict']) + # base_list=bilby_items['channel-dict'][1:-1].split(',')[:-1] + # base_dict = {} + # for item in base_list: + # if item: + # key,value =item.split(':') + # key = key.lstrip() + # base_dict[key] = value + # bilby_items['channel-dict'] = base_dict + ifo_list = list(bilby_items['channel-dict']) # PSDs must be listed, implicitly provides all ifos + # remove entries with the None keyword, as misleading + dict_names = list(bilby_items) + for name in dict_names: + if bilby_items[name] == 'None': + del bilby_items[name] + if not('data-dict' in bilby_items): + bilby_data_dict = {} + if cache_file: + print(" calmarg: bilby ini file does not have data_dict, attempting to identify data from (host) cache file: {} ".format(cache_file)) + cache_lines = np.loadtxt(cache_file,dtype=str) + if len(ifo_list)==1 and len(cache_lines.shape)==1: + ifo = cache_lines[0] + '1' + bilby_data_dict[ifo] = cache_lines[-1].replace('file://localhost','') + else: + if len(cache_lines) <= len(ifo_list): + for indx in np.arange(len(cache_lines)): + ifo = cache_lines[indx][0]+"1" + bilby_data_dict[ifo] = cache_lines[indx][-1].replace('file://localhost','') + else: + import glob + print(" WARNING: cache file ideallly contain one line per IFO to identify files in this approach") + if not(frames_dir) or not os.path.exists('./frames_dir'): + print(" WARNING: Backstop method being applied - regenerating frames into frames_dir") + shutil.copyfile(cache_file, 'local.cache') + os.system("util_ForOSG_MakeTruncatedLocalFramesDir.sh .") + fnames_gwf = list(glob.glob(frames_dir+"/*.gwf") ) + # get dictionary matching files + for name in fnames_gwf: + this_frame_ifo = None + for ifo in ifo_list: + if name.startswith(frames_dir+"/{}-".format(ifo)): + this_frame_ifo=ifo + bilby_data_dict[ifo] = this_frame_ifo + elif frames_dir: # Danger : this directory might be EMPTY and generated at runtile + import glob + print(" calmarg: bilby ini file does not have data_dict, attempting to identify data from directory: {} ".format(frames_dir)) + fnames_gwf = list(glob.glob(frames_dir+"/*.gwf") ) + # get dictionary matching files + for name in fnames_gwf: + this_frame_ifo = None + for ifo in ifo_list: + if name.startswith(frames_dir+"/{}-".format(ifo)): + this_frame_ifo=ifo + bilby_data_dict[ifo] = this_frame_ifo + if len(list(bilby_data_dict)) ==0 : + print(" Failed to find files in frames_dir, warning! ") + else: + print(" ==== WARNING FALLTHROUGH : calmarg attempting to identify correct frame files to use but falling back to 'magic' options from bilby ===") + # add to command-line arguments, IF NONEMPTY. Otherwise we're stuck, and we have to hope magic works + if len(list(bilby_data_dict))>0: + data_argstr = '{}'.format(bilby_data_dict) + data_argstr = ' --data-dict ""{}"" '.format(data_argstr.replace(' ','')) # double "" because we are in a condor submit script! Annoying but seemt to be correct + ile_job.add_arg(data_argstr) + else: + print(" ==== WARNING FALLTHROUGH : calmarg failed to pull out options ===",bilby_data_dict,bilby_items) + + # make LOCAL COPIES OF CAL ENVELOPES with STANDARD NAMES - facilitate remote/OSG use + if 'spline-calibration-envelope-dict' in bilby_items: + spline_dict = bilby_ish_string_to_dict(bilby_items['spline-calibration-envelope-dict']) + if not os.path.exists('cal_envelopes'): + os.mkdir('cal_envelopes') + for ifo in spline_dict: + shutil.copyfile(spline_dict[ifo], 'cal_envelopes/{}.txt'.format(ifo)) + + # Other required settings from ILE + # approximant: if ile_args present, ALWAYS parse it and set it that way, so we are consistent with our own analysis + if ile_args: + approx = bilby_items['waveform-approximant'] + ile_args_split = ile_args.split('--') + start_time =None + end_time=None + trigger_time =None + rift_window_shape=None # remember this is a dimensionless number, not a time + rift_srate =None + fmin_list = [] + channel_list=[] + fmax=None + for line in ile_args_split: + line_split = line.split() + if len(line_split)>1: + if line_split[0]=='approx': + approx = line_split[1] + elif line_split[0] == 'event-time': + event_time = float(line_split[1]) + elif line_split[0] == 'data-start-time': + start_time = float(line_split[1]) + elif line_split[0] == 'data-end-time': + end_time = float(line_split[1]) + elif line_split[0] == 'window-shape': + rift_window_shape = float(line_split[1]) + elif line_split[0] == 'srate': + rift_srate = int(float(line_split[1])) # safety + elif line_split[0] == 'fmin-ifo': + fmin_list += [line_split[1]] + elif line_split[0] == 'fmax': + fmax = int(float(line_split[1])) # safety + elif line_split[0] == 'channel-name': + channel_list += [line_split[1]] + ile_job.add_arg(" --waveform-approximant {} ".format(approx)) + if rift_srate: + ile_job.add_arg(" --sampling-frequency {} ".format(rift_srate)) + if event_time: + ile_job.add_arg(" --trigger-time {} ".format(event_time)) + # t_tukey + t_tukey = (end_time-start_time)*rift_window_shape/2 # basically the fraction of time not in the window; see formula in helper + ile_job.add_arg(" --tukey-roll-off {} ".format(t_tukey)) + # channel list + channel_dict ={} + for channel_id in channel_list: + if '=' in channel_id: + ifo, channel_name = channel_id.split('=') + channel_dict[ifo] = channel_name + channel_argstr = '{}'.format(channel_dict) + channel_argstr = ' --channel-dict ""{}"" '.format(channel_argstr.replace(' ','')) + ile_job.add_arg(channel_argstr) + # fmin + if len(fmin_list)>0: + fmin_dict = {} + for fmin_id in fmin_list: + if '=' in fmin_id: + ifo, fmin = fmin_id.split('=') + fmin_dict[ifo] = float(fmin) + fmin_argstr = '{}'.format(fmin_dict) + fmin_argstr = ' --minimum-frequency ""{}"" '.format(fmin_argstr.replace(' ','')) # inside condor + ile_job.add_arg(fmin_argstr) + # fmax. Use previous to get ifo list + if fmax: + fmax_dict = {} + for ifo in fmin_dict: + fmax_dict[ifo] =fmax + fmax_argstr = '{}'.format(fmax_dict) + fmax_argstr = ' --maximum-frequency ""{}"" '.format(fmax_argstr.replace(' ','')) + ile_job.add_arg(fmax_argstr) + + # Add outdir, label so we can control filename for output + ile_job.add_arg(" --outdir calmarg ") + ile_job.add_arg(" --label calmarg ") + + # + # Add normal arguments + # Note these need to appear *after* the bilby ini file + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + + # getenv: use default, BUT want to check for datafind so it is passed! + # + pickle_getenv_value = str(default_getenv_value) # force re-create + if not(pickle_getenv_value == 'True') and not('DATAFIND' in pickle_getenv_value): + names_datafind = [] + for name in os.environ: + if 'DATAFIND' in name: + names_datafind.append(name) + if len(names_datafind)>0: + pickle_getenv_value= default_getenv_value +',' + ",".join(names_datafind) + ile_job.add_condor_cmd('getenv', pickle_getenv_value) + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + try: + os.system("condor_config_val UID_DOMAIN > uid_domain.txt") + with open("uid_domain.txt", 'r') as f: + uid_domain = f.readline().strip() + requirements.append(' UidDomain =?= "{}"'.format(uid_domain)) + except: + True + + # Write requirements + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + return ile_job, ile_sub_name + +def write_comov_distance_reweighting_sub(tag='Comov_dist', comov_distance_reweighting_exe=None, reweight_location=None, universe='vanilla', log_dir=None, ncopies=1,request_memory=4096,posterior_file=None,no_grid=False,**kwargs): + """ + Write a submit file for launching a job to generate reweight posterior samples to reflect a comoving distance prior + + Inputs: + - posterior samples in h5 format + Outputs: + - reweighted samples in h5 format + """ + exe = comov_distance_reweighting_exe or which("make_uni_comov_skymap.py") + if exe is None: + print(" Comoving distance reweighting code unavailable. ") + sys.exit(0) + ile_job = CondorDAGJob(universe=universe, executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # Add manual options for input, output + ile_job.add_opt('resampled-file', str(reweight_location)) + ile_job.add_arg(str(posterior_file)) # needs to be a bilby ini file for the particular event being analyzed + + # + #Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + + + # + # Add normal arguments + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + + + # Write requirements + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + return ile_job, ile_sub_name + +def write_convert_ascii_to_h5_sub(tag='Convert_ascii2h5', convert_ascii_to_h5_exe=None,output_file=None, universe='vanilla', log_dir=None, ncopies=1,request_memory=4096,posterior_file=None,no_grid=False,**kwargs): + """ + Converts posterior samples file from ascii to h5 format + + Inputs: + - posterior samples in ascii format + Outputs: + - posterior samples in h5 format + """ + exe = convert_ascii_to_h5_exe or which("convert_output_format_ascii2h5.py") + if exe is None: + print(" Converting code unavailable. ") + sys.exit(0) + ile_job = CondorDAGJob(universe=universe, executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # Add manual options for input, output + ile_job.add_opt('output-file', str(output_file)) + ile_job.add_opt('posterior-file', str(posterior_file)) +# ile_job.add_arg(str(posterior_file)) # needs to be a bilby ini file for the particular event being analyzed + + # + #Logging options + # + uniq_str = "$(macromassid)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + + + # + # Add normal arguments + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + ile_job.add_condor_cmd('getenv', default_getenv_value) + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + try: + os.system("condor_config_val UID_DOMAIN > uid_domain.txt") + with open("uid_domain.txt", 'r') as f: + uid_domain = f.readline().strip() + requirements.append(' UidDomain =?= "{}"'.format(uid_domain)) + except: + True + + # Write requirements + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + return ile_job, ile_sub_name + + +def write_hyperpost_sub(tag='HYPER', exe=None, input_net='all.marg_net',output='output-samples',universe="vanilla",out_dir=None,log_dir=None, ncopies=1,arg_str=None,request_memory=8192,arg_vals=None, no_grid=False,request_disk=False, transfer_files=None,transfer_output_files=None,use_singularity=False,use_osg=False,use_simple_osg_requirements=False,singularity_image=None,max_runtime_minutes=None,condor_commands=None,**kwargs): + """ + Write a submit file for launching jobs to marginalize the likelihood over hyperparameters. + Almost identical to CIP + + Inputs: + Outputs: + - An instance of the CondorDAGJob that was generated for ILE + """ + + if use_singularity and (singularity_image == None) : + print(" FAIL : Need to specify singularity_image to use singularity ") + sys.exit(0) + if use_singularity and (transfer_files == None) : + print(" FAIL : Need to specify transfer_files to use singularity at present! (we will append the prescript; you should transfer any PSDs as well as the grid file ") + sys.exit(0) + + + exe = exe or which("util_ConstructEOSPosterior.py") + if use_singularity: + path_split = exe.split("/") + print((" Executable: name breakdown ", path_split, " from ", exe)) + singularity_base_exe_path = "/usr/bin/" # should not hardcode this ...! + if 'SINGULARITY_BASE_EXE_DIR' in list(os.environ.keys()) : + singularity_base_exe_path = os.environ['SINGULARITY_BASE_EXE_DIR'] + exe=singularity_base_exe_path + path_split[-1] + if path_split[-1] == 'true': # special universal path for /bin/true, don't override it! + exe = "/usr/bin/true" + ile_job = CondorDAGJob(universe=universe, executable=exe) + # This is a hack since CondorDAGJob hides the queue property + ile_job._CondorJob__queue = ncopies + + + # no grid + if no_grid: + ile_job.add_condor_cmd("MY.DESIRED_SITES",'"nogrid"') + ile_job.add_condor_cmd("MY.flock_local",'true') + try: + os.system("condor_config_val UID_DOMAIN > uid_domain.txt") + with open("uid_domain.txt", 'r') as f: + uid_domain = f.readline().strip() + requirements.append(' UidDomain =?= "{}"'.format(uid_domain)) + except: + True + + requirements=[] + if universe=='local': + requirements.append("IS_GLIDEIN=?=undefined") + + ile_sub_name = tag + '.sub' + ile_job.set_sub_file(ile_sub_name) + + # + # Add options en mass, by brute force + # + arg_str = arg_str.lstrip() # remove leading whitespace and minus signs + arg_str = arg_str.lstrip('-') + ile_job.add_opt(arg_str,'') + + ile_job.add_opt("fname", input_net) + ile_job.add_opt("fname-output-samples", out_dir+"/"+output) + ile_job.add_opt("fname-output-integral", out_dir+"/"+output) + + # + # Macro based options. + # - select EOS from list (done via macro) + # - pass spectral parameters + # + + # + # Logging options + # + uniq_str = "$(macroevent)-$(cluster)-$(process)" + ile_job.set_log_file("%s%s-%s.log" % (log_dir, tag, uniq_str)) + ile_job.set_stderr_file("%s%s-%s.err" % (log_dir, tag, uniq_str)) + ile_job.set_stdout_file("%s%s-%s.out" % (log_dir, tag, uniq_str)) + + if "fname_output_samples" in kwargs and kwargs["fname_output_samples"] is not None: + # + # Need to modify the output file so it's unique + # + ofname = kwargs["fname_output_samples"].split(".") + ofname, ext = ofname[0], ".".join(ofname[1:]) + ile_job.add_file_opt("fname-output-samples", "%s-%s.%s" % (ofname, uniq_str, ext)) + + # + # Add normal arguments + # FIXME: Get valid options from a module + # + for opt, param in list(kwargs.items()): + if isinstance(param, list) or isinstance(param, tuple): + # NOTE: Hack to get around multiple instances of the same option + for p in param: + ile_job.add_arg("--%s %s" % (opt.replace("_", "-"), str(p))) + elif param is True: + ile_job.add_opt(opt.replace("_", "-"), None) + elif param is None or param is False: + continue + else: + ile_job.add_opt(opt.replace("_", "-"), str(param)) + + if not use_osg: + ile_job.add_condor_cmd('getenv', default_getenv_value) + ile_job.add_condor_cmd('request_memory', str(request_memory)+"M") + if not(request_disk is False): + ile_job.add_condor_cmd('request_disk', str(request_disk)) + # To change interactively: + # condor_qedit + # for example: + # for i in `condor_q -hold | grep oshaughn | awk '{print $1}'`; do condor_qedit $i RequestMemory 30000; done; condor_release -all + + requirements = [] + if use_singularity: + # Compare to https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py + ile_job.add_condor_cmd('request_CPUs', str(1)) + ile_job.add_condor_cmd('transfer_executable', 'False') + ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') + ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image + '"') + requirements.append("HAS_SINGULARITY=?=TRUE") + + if use_osg: + # avoid black-holing jobs to specific machines that consistently fail. Uses history attribute for ad + ile_job.add_condor_cmd('periodic_release','(HoldReasonCode == 45) && (HoldReasonSubCode == 0)') + ile_job.add_condor_cmd('job_machine_attrs','Machine') + ile_job.add_condor_cmd('job_machine_attrs_history_length','4') +# for indx in [1,2,3,4]: +# requirements.append("TARGET.GLIDEIN_ResourceName=!=MY.MachineAttrGLIDEIN_ResourceName{}".format(indx)) + if "OSG_DESIRED_SITES" in os.environ: + ile_job.add_condor_cmd('+DESIRED_SITES',os.environ["OSG_DESIRED_SITES"]) + if "OSG_UNDESIRED_SITES" in os.environ: + ile_job.add_condor_cmd('+UNDESIRED_SITES',os.environ["OSG_UNDESIRED_SITES"]) + # Some options to automate restarts, acts on top of RETRY in dag + if use_singularity or use_osg: + # Set up file transfer options + ile_job.add_condor_cmd("when_to_transfer_output",'ON_EXIT') + + # Stream log info + if not ('RIFT_NOSTREAM_LOG' in os.environ): + ile_job.add_condor_cmd("stream_error",'True') + ile_job.add_condor_cmd("stream_output",'True') + + + ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) + + # Stream log info: always stream CIP error, it is a critical bottleneck + if True: # not ('RIFT_NOSTREAM_LOG' in os.environ): + ile_job.add_condor_cmd("stream_error",'True') + ile_job.add_condor_cmd("stream_output",'True') + + try: + ile_job.add_condor_cmd('accounting_group',os.environ['LIGO_ACCOUNTING']) + ile_job.add_condor_cmd('accounting_group_user',os.environ['LIGO_USER_NAME']) + except: + print(" LIGO accounting information not available. You must add this manually to integrate.sub !") + + + if not transfer_files is None: + if not isinstance(transfer_files, list): + fname_str=transfer_files + else: + fname_str = ','.join(transfer_files) + fname_str=fname_str.strip() + ile_job.add_condor_cmd('transfer_input_files', fname_str) + ile_job.add_condor_cmd('should_transfer_files','YES') + + # Periodic remove: kill jobs running longer than max runtime + # https://stackoverflow.com/questions/5900400/maximum-run-time-in-condor + if not(max_runtime_minutes is None): + remove_str = 'JobStatus =?= 2 && (CurrentTime - JobStartDate) > ( {})'.format(60*max_runtime_minutes) + ile_job.add_condor_cmd('periodic_remove', remove_str) + + + ### + ### SUGGESTION FROM STUART (for later) + # request_memory = ifthenelse( (LastHoldReasonCode=!=34 && LastHoldReasonCode=!=26), InitialRequestMemory, int(1.5 * NumJobStarts * MemoryUsage) ) + # periodic_release = ((HoldReasonCode =?= 34) || (HoldReasonCode =?= 26)) + # This will automatically release a job that is put on hold for using too much memory with a 50% increased memory request each tim.e + if condor_commands is not None: + for cmd, value in condor_commands.items(): + ile_job.add_condor_cmd(cmd, value) + + + return ile_job, ile_sub_name + diff --git a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py new file mode 100644 index 000000000..dc8acae9f --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py @@ -0,0 +1,259 @@ +""" +Tests for container family manifest parsing and the expression-valued +SingularityImage / selective-transfer / require_gpus wiring. + +These run without a real HTCondor pool: the parser + expression builders are +pure, and the integration test inspects the generated ``condor_cmds`` on the +job object returned by ``write_ILE_sub_simple`` (no .sub file or condor needed). + +Run directly: python test/test_container_manifest.py +Or via pytest: pytest test/test_container_manifest.py +""" + +import os +import sys +import textwrap + +import pytest + +yaml = pytest.importorskip("yaml") # manifest parsing requires PyYAML + +import RIFT.misc.container_manifest as cm + + +# --------------------------------------------------------------------------- +# helpers +# --------------------------------------------------------------------------- + +MIXED_MANIFEST = textwrap.dedent( + """ + version: 1 + fallback: ancient + containers: + - label: ancient + image: /cvmfs/sw/rift_ancient_cuda11.sif + cuda_capability_min: 3.0 + cuda_capability_max: 7.0 + - label: modern + image: osdf:///igwn/rift_modern_cuda12.sif + cuda_capability_min: 7.0 + """ +) + +ALL_CVMFS_MANIFEST = textwrap.dedent( + """ + version: 1 + fallback: ancient + containers: + - label: ancient + image: /cvmfs/sw/rift_ancient.sif + cuda_capability_min: 3.0 + - label: modern + image: /cvmfs/sw/rift_modern.sif + cuda_capability_min: 7.0 + """ +) + + +def _write(tmp_path, text, name="fam.yaml"): + p = tmp_path / name + p.write_text(text) + return str(p) + + +# --------------------------------------------------------------------------- +# 1. parser +# --------------------------------------------------------------------------- + +def test_parser_sorts_and_resolves_fallback(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + # sorted by capability descending + assert [c["label"] for c in m["containers"]] == ["modern", "ancient"] + assert m["fallback"] == "ancient" + assert m["capability_attr"] == cm.DEFAULT_CAPABILITY_ATTR + + +def test_parser_default_fallback_is_lowest(tmp_path): + # no explicit fallback -> most-compatible (lowest-min) container + text = MIXED_MANIFEST.replace("fallback: ancient\n", "") + m = cm.load_container_manifest(_write(tmp_path, text)) + assert m["fallback"] == "ancient" + + +def test_parser_rejects_unknown_fallback(tmp_path): + text = MIXED_MANIFEST.replace("fallback: ancient", "fallback: nope") + with pytest.raises(cm.ContainerManifestError): + cm.load_container_manifest(_write(tmp_path, text)) + + +def test_parser_rejects_empty(tmp_path): + with pytest.raises(cm.ContainerManifestError): + cm.load_container_manifest(_write(tmp_path, "version: 1\ncontainers: []\n")) + + +def test_parser_rejects_missing_image(tmp_path): + text = "containers:\n - label: x\n cuda_capability_min: 5.0\n" + with pytest.raises(cm.ContainerManifestError): + cm.load_container_manifest(_write(tmp_path, text)) + + +# --------------------------------------------------------------------------- +# 2. expressions +# --------------------------------------------------------------------------- + +def test_image_expression(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + expr = cm.build_singularity_image_expr(m) + assert expr == ( + 'ifThenElse(TARGET.GPUs_Capability >= 7.0, ' + '"./rift_modern_cuda12.sif", "/cvmfs/sw/rift_ancient_cuda11.sif")' + ) + # an expression must NOT be a quoted string literal + assert not expr.startswith('"') + + +def test_transfer_expression_is_comma_free_ternary(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + expr = cm.build_transfer_input_expr(m) + assert expr == ( + '$$([ (TARGET.GPUs_Capability >= 7.0 ? ' + '"osdf:///igwn/rift_modern_cuda12.sif" : "") ])' + ) + # the token sits inside a comma-separated transfer_input_files list, so it + # must contain no commas of its own + assert "," not in expr + + +def test_transfer_expression_none_when_all_in_place(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, ALL_CVMFS_MANIFEST)) + assert cm.build_transfer_input_expr(m) is None + + +def test_require_gpus_floor(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + assert cm.build_require_gpus_floor(m) == "Capability >= 3.0" + + +def test_capability_attr_env_override(tmp_path, monkeypatch): + monkeypatch.setenv("RIFT_GPU_CAPABILITY_ATTR", "CUDACapability") + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + assert "TARGET.CUDACapability >=" in cm.build_singularity_image_expr(m) + + +# --------------------------------------------------------------------------- +# 3-5. integration with write_ILE_sub_simple (inspect generated condor_cmds) +# --------------------------------------------------------------------------- + +def _make_ile_job(tmp_path, monkeypatch, singularity_image): + """Call write_ILE_sub_simple in an isolated cwd; return its condor_cmds dict. + + Skips if the dag_utils_generic backend cannot be imported in this env. + """ + dag = pytest.importorskip("RIFT.misc.dag_utils_generic") + monkeypatch.chdir(tmp_path) + job, _ = dag.write_ILE_sub_simple( + tag="ILE", + log_dir=str(tmp_path) + "/", + exe="/usr/bin/true", + arg_str="--foo bar", + transfer_files=["../all.net"], + use_singularity=True, + singularity_image=singularity_image, + request_gpu=True, + cache_file="local.cache", + ) + return dict(job.condor_cmds) + + +def test_integration_family_mixed(tmp_path, monkeypatch): + monkeypatch.setenv( + "RIFT_REQUIRE_GPUS", '(DeviceName=!="Tesla K10.G1.8GB")' + ) + cmds = _make_ile_job(tmp_path, monkeypatch, _write(tmp_path, MIXED_MANIFEST)) + + img = cmds["MY.SingularityImage"] + assert img.startswith("ifThenElse(") # expression, not a literal + assert not img.startswith('"') + + # selective transfer: exactly one $$() token, whole family NOT dumped + tif = cmds["transfer_input_files"] + assert tif.count("$$([") == 1 + assert "/cvmfs/sw/rift_ancient_cuda11.sif" not in tif # cvmfs image not transferred + assert tif.count("osdf:///igwn/rift_modern_cuda12.sif") == 1 + + # floor composed with (not replacing) the user's RIFT_REQUIRE_GPUS + rg = cmds["require_gpus"] + assert "Capability >= 3.0" in rg + assert 'DeviceName=!="Tesla K10.G1.8GB"' in rg + assert "&&" in rg + + +def test_integration_all_cvmfs_no_transfer_token(tmp_path, monkeypatch): + cmds = _make_ile_job(tmp_path, monkeypatch, _write(tmp_path, ALL_CVMFS_MANIFEST)) + assert "$$([" not in cmds.get("transfer_input_files", "") + # still an expression-valued image + a capability floor + assert cmds["MY.SingularityImage"].startswith("ifThenElse(") + assert "Capability >= 3.0" in cmds["require_gpus"] + + +def test_backward_compat_single_sif(tmp_path, monkeypatch): + monkeypatch.delenv("RIFT_REQUIRE_GPUS", raising=False) + cmds = _make_ile_job(tmp_path, monkeypatch, "./foo.sif") + # byte-identical legacy behavior: quoted literal, no $$() token, no floor + assert cmds["MY.SingularityImage"] == '"./foo.sif"' + assert "$$([" not in cmds.get("transfer_input_files", "") + assert "require_gpus" not in cmds + + +# --------------------------------------------------------------------------- +# 6. container universe: $$()-substituted container_image selection (OSG-safe) +# --------------------------------------------------------------------------- + +def test_container_image_select_expression(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + expr = cm.build_container_image_select(m) + # a $$() match-time substitution token, undefined-safe, with VERBATIM image + # values (osdf URL fetched by container universe; cvmfs path used in place) -- + # NOT a ./basename rewrite + assert expr.startswith("$$([ ") and expr.endswith(" ])") + assert "TARGET.GPUs_Capability =?= undefined" in expr # undefined -> fallback + assert "ifThenElse(TARGET.GPUs_Capability >= 7.0," in expr + assert '"osdf:///igwn/rift_modern_cuda12.sif"' in expr # raw osdf URL + assert '"/cvmfs/sw/rift_ancient_cuda11.sif"' in expr # fallback verbatim + assert "./rift_modern_cuda12.sif" not in expr # no basename rewrite + + +def test_integration_container_universe(tmp_path, monkeypatch): + # Opt-in container-universe mode: per-machine image via $$()-substituted + # container_image; no MY.SingularityImage / BindCVMFS / $$() transfer token; + # universe=container; require_gpus floor still applied. + monkeypatch.setenv("RIFT_CONTAINER_UNIVERSE", "1") + monkeypatch.delenv("RIFT_REQUIRE_GPUS", raising=False) + monkeypatch.chdir(tmp_path) + dag = pytest.importorskip("RIFT.misc.dag_utils_generic") + job, _ = dag.write_ILE_sub_simple( + tag="ILE", + log_dir=str(tmp_path) + "/", + exe="/usr/bin/true", + arg_str="--foo bar", + transfer_files=["../all.net"], + use_singularity=True, + singularity_image=_write(tmp_path, MIXED_MANIFEST), + request_gpu=True, + cache_file="local.cache", + ) + cmds = dict(job.condor_cmds) + + ci = cmds["container_image"] + assert ci.startswith("$$([") # match-time substitution, unquoted + assert not ci.startswith('"') + assert "MY.SingularityImage" not in cmds # the OSG-breaking attr is gone + assert "MY.SingularityBindCVMFS" not in cmds + assert "$$([" not in cmds.get("transfer_input_files", "") # image via container_image, not transfer + assert "Capability >= 3.0" in cmds["require_gpus"] # floor still steers GPUs + + assert job.universe == "container" # HTCondor container universe + + +if __name__ == "__main__": + sys.exit(pytest.main([os.path.abspath(__file__), "-v"])) From 88637d8ae694b848582251e693d860f0e8455197 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Thu, 11 Jun 2026 09:32:56 -0700 Subject: [PATCH 02/63] container family: extend CALPILOT to per-machine image (legacy + container universe) write_calpilot_sub still handed the raw SINGULARITY_RIFT_IMAGE value to MY.SingularityImage, so a .yaml/.yml family MANIFEST reached condor as the image path and the job failed (a manifest is not a .sif). The container-universe work fixed write_ILE_sub_simple but never touched the CALPILOT writer, even though the CALPILOT job runs ILE internally (GPU) and needs the same per-machine selection. Mirror write_ILE_sub_simple exactly: * detect a container manifest (is_container_manifest) and expand it; * legacy (default): universe=vanilla, MY.SingularityImage = ifThenElse(...), plus the selective $$() osdf transfer token and a require_gpus floor; * container universe (opt-in RIFT_CONTAINER_UNIVERSE): universe=container, container_image = $$([...]) (match-time, OSG-safe), no MY.SingularityImage / SingularityBindCVMFS, image delivered via container_image (no transfer token). A plain .sif / osdf:// value keeps the legacy single-image behavior unchanged. Validated offline (pilot DAG build, OSG=1, family manifest) in both modes: the generated CALPILOT.sub container_image is byte-identical to ILE.sub, and the require_gpus floor is applied. test_container_manifest.py: 15/15 pass. Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/dag_utils_generic.py | 63 ++++++++++++++++++- 1 file changed, 60 insertions(+), 3 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py index d602cc722..7c2329d37 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py @@ -2804,6 +2804,37 @@ def write_calpilot_sub(tag='calpilot', exe=None, log_dir=None, universe="vanilla transfer_files = [] transfer_files = list(transfer_files) + # Container family manifest support -- mirror write_ILE_sub_simple. The CALPILOT + # job runs ILE internally (GPU path), so a .yaml/.yml manifest must be expanded + # to a per-machine image exactly like a wide ILE job; otherwise the raw manifest + # path is handed to condor as the image and the job fails. Two modes: + # * legacy (default): execute-side MY.SingularityImage = ifThenElse(...) -- works + # on the CIT-local pool but OSPool pilots read it as a literal string and hold. + # * container universe (opt-in via RIFT_CONTAINER_UNIVERSE): universe=container + + # container_image = $$([...]), a match-time machine-ad substitution the schedd + # resolves to a literal image before the job reaches the EP -- OSG-safe. + # A plain .sif / osdf:// value keeps the legacy single-image behavior below. + singularity_is_family = False + singularity_container_universe = False + singularity_container_image_select = None + singularity_image_expr = None + singularity_require_gpus_floor = None + if singularity_image and is_container_manifest(singularity_image): + singularity_is_family = True + _manifest = load_container_manifest(singularity_image) + singularity_image_expr = build_singularity_image_expr(_manifest) + singularity_require_gpus_floor = build_require_gpus_floor(_manifest) + singularity_container_universe = bool(use_singularity and os.environ.get('RIFT_CONTAINER_UNIVERSE')) + if singularity_container_universe: + singularity_container_image_select = build_container_image_select(_manifest) + else: + # Selective ($$()) transfer of only the matched osdf image (comma-free so + # it survives transfer_input_files comma-splitting). In container-universe + # mode the image is delivered via container_image itself, so skip this. + _transfer_expr = build_transfer_input_expr(_manifest) + if on_osg and _transfer_expr: + transfer_files += [_transfer_expr] + if use_singularity: base = os.environ.get('SINGULARITY_BASE_EXE_DIR', '/usr/bin/') exe = base.rstrip('/') + '/' + exe_base @@ -2839,7 +2870,9 @@ def write_calpilot_sub(tag='calpilot', exe=None, log_dir=None, universe="vanilla transfer_files += [os.path.abspath(pre), frames_dir] exe = pre - job = pipeline.CondorDAGJob(universe="vanilla", executable=exe) + # Container universe (opt-in) runs the executable inside container_image directly; + # otherwise stay vanilla + (optional) condor singularity. + job = pipeline.CondorDAGJob(universe=("container" if singularity_container_universe else "vanilla"), executable=exe) sub_name = tag + '.sub' job.set_sub_file(sub_name) @@ -2870,12 +2903,36 @@ def write_calpilot_sub(tag='calpilot', exe=None, log_dir=None, universe="vanilla requirements = [] if use_singularity and singularity_image: job.add_condor_cmd('transfer_executable', 'False') - job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') - job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image + '"') + if singularity_container_universe: + # Container universe: the per-machine image is delivered via container_image, + # a $$()-substituted (match-time) literal -- emit it raw/unquoted (a $$() + # value must not be wrapped in quotes), with NO MY.SingularityImage / + # MY.SingularityBindCVMFS. GPU access is automatic under request_gpus. + job.add_condor_cmd("container_image", singularity_container_image_select) + else: + job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') + if singularity_is_family: + # Expression-valued: emit the ifThenElse raw, with NO surrounding double + # quotes (a classad expression must not be quoted). + job.add_condor_cmd("MY.SingularityImage", singularity_image_expr) + else: + job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image + '"') job.add_condor_cmd("MY.flock_local", 'true') requirements.append("HAS_SINGULARITY=?=TRUE") elif singularity_image: job.add_condor_cmd("+SingularityImage", '"' + singularity_image + '"') + + # require_gpus: compose the user's RIFT_REQUIRE_GPUS with the container family's + # capability floor (mirrors write_ILE_sub_simple) so a manifest job never matches a + # GPU less capable than anything we ship. CALPILOT requests a GPU (runs ILE). + if request_gpu: + require_gpus_terms = [] + if 'RIFT_REQUIRE_GPUS' in os.environ: + require_gpus_terms.append('({})'.format(os.environ['RIFT_REQUIRE_GPUS'])) + if singularity_is_family and singularity_require_gpus_floor: + require_gpus_terms.append('({})'.format(singularity_require_gpus_floor)) + if require_gpus_terms: + job.add_condor_cmd('require_gpus', ' && '.join(require_gpus_terms)) if use_oauth_files: job.add_condor_cmd('use_oauth_services', use_oauth_files) From 8c3dbdee5c48e55ebc901f003006b7fd86fed4a6 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Thu, 11 Jun 2026 10:00:24 -0700 Subject: [PATCH 03/63] container family: enable osdf scitokens credential for manifest images When SINGULARITY_RIFT_IMAGE is a container-family MANIFEST (.yaml/.yml), the osdf:// image URLs live INSIDE the manifest, so the existing `'osdf:' in singularity_image` auto-detect (which force-sets use_oauth_files='scitokens' for single-image osdf runs) misses it. Result: no `use_oauth_services = scitokens` in the subs -> the execute point has no credential to fetch the selected container -> every ILE/CIP/CALPILOT job is held with "credential is required for osdf://...sif but was not discovered". Add a manifest-aware branch: if singularity_image is a container manifest, inspect its image URLs and pick the same credential the single-image path would (igwn+osdf -> 'igwn', osdf -> 'scitokens'). Pipeline-writer only (bin/), no container rebuild. Validated offline: a family-manifest pilot build now emits `use_oauth_services = scitokens` on ILE/ILE_extr/ILE_puff/CALPILOT/CIP/ CIP_0/CIP_worker0, matching the old working single-image subs. Co-Authored-By: Claude Opus 4.8 --- ...eate_event_parameter_pipeline_BasicIteration | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration b/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration index 47f8043f6..a5f2366f3 100755 --- a/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration +++ b/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration @@ -356,6 +356,23 @@ if opts.use_singularity: opts.use_oauth_files = 'igwn' # force-set this variable, to save time for end user. This will always be the case elif 'osdf:' in singularity_image and not opts.use_oauth_files: opts.use_oauth_files = 'scitokens' # force-set this variable, to save time for end user. This will always be the case + elif not opts.use_oauth_files: + # Container FAMILY manifest (.yaml/.yml): the per-machine image osdf:// URLs live + # INSIDE the manifest, so the 'osdf:' substring checks above miss them and the + # transfer credential is never enabled -> the execute point cannot fetch the + # selected container ("credential is required for osdf://... but was not + # discovered" -> all ILE/CIP/CALPILOT jobs held). Inspect the manifest's image + # URLs and pick the same credential the single-image path would have. + try: + from RIFT.misc.container_manifest import is_container_manifest, load_container_manifest + if is_container_manifest(singularity_image): + _imgs = ' '.join(c.get('image', '') for c in load_container_manifest(singularity_image).get('containers', [])) + if 'igwn+osdf:' in _imgs: + opts.use_oauth_files = 'igwn' + elif 'osdf:' in _imgs: + opts.use_oauth_files = 'scitokens' + except Exception: + pass if (opts.cip_args is None) and (opts.cip_args_list is None): print(" No arguments provided for low-level job") From 9c057b21d76c445cffa8a2f5f439b4e15884a578 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Thu, 11 Jun 2026 15:53:46 -0700 Subject: [PATCH 04/63] container universe: collapse to a single container for non-GPU jobs (CIP fix) A CPU-only job (CIP) requests no GPU, so it matches a slot that advertises NO GPU capability attribute. The per-machine container_image = $$([ ... capability ... ]) then has nothing to resolve against: the $$() substitution fails to expand and HTCondor HOLDS the job -> all CIPs lock up. Fix: when a job requests no GPU, do not emit a $$() capability selection at all; use a SINGLE fixed container (the manifest fallback, i.e. the CPU-safe image). - build_container_image_select(manifest, request_gpu=True): with request_gpu= False it returns the plain fallback image literal (no $$(), no ifThenElse). - write_ILE_sub_simple passes request_gpu through (GPU jobs keep the $$ selector; a no-GPU ILE would also collapse). - write_CIP_sub: wire container universe for CIP too (universe=container, container_image = fallback literal, no MY.SingularityImage / BindCVMFS / $$() transfer token). CIP is CPU-only so it always collapses to the single image; no require_gpus floor (unchanged). Also corrects the stale CIP comment that claimed an undefined capability "collapses to the fallback image" -- true-ish for the native ifThenElse, but false for $$(), which holds the job. Tests: build_container_image_select(request_gpu=False) -> bare fallback image; CIP integration (universe=container, container_image = single fallback literal, no MY.SingularityImage / no $$() token / no require_gpus). 17/17 pass. Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/container_manifest.py | 55 +++++++++++-------- .../Code/RIFT/misc/dag_utils_generic.py | 47 ++++++++++++---- .../Code/test/test_container_manifest.py | 37 +++++++++++++ 3 files changed, 106 insertions(+), 33 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py index f3cda0f00..adb3dff5b 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py @@ -286,32 +286,43 @@ def value_fn(c): return "$$([ {} ])".format(_build_selector(manifest, value_fn, ternary=True)) -def build_container_image_select(manifest): - """Return an unquoted ``$$([ ... ])`` value for the HTCondor *container - universe* ``container_image`` submit command, selecting the per-machine image. - - Unlike :func:`build_singularity_image_expr` (an execute-side ClassAd - expression that OSPool glidein pilots read as a literal string and hold the - job on), this uses HTCondor ``$$()`` *match-time machine-ad substitution*: the - schedd evaluates the bracketed expression against the matched machine ad and - substitutes a literal image string into ``container_image`` before the job - reaches the execution point. ``$$`` in ``container_image`` is HTCondor's +def build_container_image_select(manifest, request_gpu=True): + """Return the value for the HTCondor *container universe* ``container_image`` + submit command for this family. + + GPU jobs (``request_gpu=True``, the default) get a per-machine selection: an + unquoted ``$$([ ... ])`` token. ``$$()`` is HTCondor's *match-time machine-ad + substitution* -- the schedd evaluates the bracketed expression against the + matched machine ad and substitutes a literal image string into + ``container_image`` before the job reaches the execution point. Unlike + :func:`build_singularity_image_expr` (an execute-side ClassAd expression that + OSPool glidein pilots read as a literal string and hold on), the pilot only + ever sees a literal URL. ``$$`` in ``container_image`` is HTCondor's documented mechanism for per-GPU-capability image selection, and it works on - both the CIT-local pool and OSPool glideins. - - The branch value is the manifest image *verbatim* -- an ``osdf://`` URL that - container universe's file-transfer plugin fetches, or a CVMFS/local path used - in place -- NOT a ``./basename`` rewrite (container universe handles the image - itself). ``container_image`` is a single submit command (not a comma list), - so the comma-bearing ``ifThenElse`` form is fine here. - - The expression is undefined-safe: if the matched machine does not advertise - the capability attribute (e.g. a CPU-only slot), it yields the ``fallback`` - image instead of an undefined ``$$()`` that would hold the job. + both the CIT-local pool and OSPool glideins. The branch value is the manifest + image *verbatim* (an ``osdf://`` URL the container-universe file-transfer + plugin fetches, or a CVMFS/local path used in place) -- NOT a ``./basename`` + rewrite. ``container_image`` is a single submit command (not a comma list), + so the comma-bearing ``ifThenElse`` form is fine. + + **Non-GPU jobs (``request_gpu=False``) collapse to a SINGLE fixed container**: + the plain ``fallback`` image (a literal ``container_image``, no ``$$()``). + A CPU-only job (e.g. CIP) matches a slot that advertises **no** GPU capability + attribute, so a ``$$()`` capability expression has nothing to resolve against + -- it fails to expand and HTCondor *holds the job*. There is also nothing to + select between, so the CPU-safe fallback image is the right (and only) choice. + + (The GPU-path expression is also written undefined-safe -- ``=?= undefined`` + yields the fallback -- but a GPU job that requested a GPU will match a slot + that advertises the capability, so that guard is belt-and-suspenders; the + non-GPU case must not use ``$$()`` at all.) """ - attr = _capability_attr(manifest) by_label = {c["label"]: c for c in manifest["containers"]} fb_image = by_label[manifest["fallback"]]["image"] + if not request_gpu: + # Single fixed container: no capability, no $$() -- a plain literal. + return fb_image + attr = _capability_attr(manifest) selector = _build_selector(manifest, lambda c: '"{}"'.format(c["image"])) guarded = 'ifThenElse(TARGET.{attr} =?= undefined, "{fb}", {sel})'.format( attr=attr, fb=fb_image, sel=selector diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py index 7c2329d37..f20b1db71 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py @@ -1924,10 +1924,16 @@ def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output- singularity_image_used = "{}".format(singularity_image) # make copy extra_files = [] # Container family manifest support (see write_ILE_sub_simple). CIP jobs do - # not request GPUs, so no require_gpus floor is added here; on a CPU-only - # slot TARGET.GPUs_Capability is undefined and the selection expression - # collapses to the fallback image, which must be the CPU-safe one. + # NOT request a GPU, so no require_gpus floor is added here. For the + # container-universe path that means the per-capability $$() selection MUST + # NOT be used: a CPU-only slot advertises no GPU capability attribute, so a + # $$() capability expression cannot resolve and HTCondor holds the job. We + # collapse to the single (CPU-safe) fallback image instead. (For the legacy + # MY.SingularityImage=ifThenElse path the same family expression is emitted; + # on CIT-local native singularity it degrades to the fallback branch.) singularity_is_family = False + singularity_container_universe = False + singularity_container_image_select = None singularity_image_expr = None singularity_transfer_expr = None if singularity_image and is_container_manifest(singularity_image): @@ -1935,7 +1941,15 @@ def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output- _manifest = load_container_manifest(singularity_image) singularity_image_expr = build_singularity_image_expr(_manifest) singularity_transfer_expr = build_transfer_input_expr(_manifest) - if singularity_transfer_expr: + # Container universe (opt-in: RIFT_CONTAINER_UNIVERSE). CIP is CPU-only, + # so request_gpu=False -> a SINGLE fixed container (the fallback image), + # NOT a $$() capability selection that a CPU slot can't resolve. + singularity_container_universe = bool(use_singularity and os.environ.get('RIFT_CONTAINER_UNIVERSE')) + if singularity_container_universe: + singularity_container_image_select = build_container_image_select(_manifest, request_gpu=False) + # Container universe delivers the image via container_image itself, so + # skip the $$() transfer token in that mode. + if singularity_transfer_expr and not singularity_container_universe: extra_files += [singularity_transfer_expr] elif singularity_image: if 'osdf:' in singularity_image: @@ -1955,7 +1969,8 @@ def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output- exe=singularity_base_exe_path + exe_base if exe_base == 'true': # special universal path for /bin/true, don't override it! exe = "/usr/bin/true" - ile_job = CondorDAGJob(universe=universe, executable=exe) + # Container universe (opt-in) runs the job inside container_image directly. + ile_job = CondorDAGJob(universe=("container" if singularity_container_universe else universe), executable=exe) # This is a hack since CondorDAGJob hides the queue property ile_job._CondorJob__queue = ncopies @@ -2056,12 +2071,18 @@ def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output- # Compare to https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py ile_job.add_condor_cmd('request_CPUs', str(1)) ile_job.add_condor_cmd('transfer_executable', 'False') - ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') - if singularity_is_family: - # Expression-valued: emit raw, NO surrounding double quotes. - ile_job.add_condor_cmd("MY.SingularityImage", singularity_image_expr) + if singularity_container_universe: + # CPU-only CIP: a SINGLE fixed container (the fallback image) -- no + # capability $$() selection (a CPU slot can't resolve it). No + # MY.SingularityImage / MY.SingularityBindCVMFS. + ile_job.add_condor_cmd("container_image", singularity_container_image_select) else: - ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') + ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') + if singularity_is_family: + # Expression-valued: emit raw, NO surrounding double quotes. + ile_job.add_condor_cmd("MY.SingularityImage", singularity_image_expr) + else: + ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') requirements.append("HAS_SINGULARITY=?=TRUE") if use_oauth_files: @@ -2284,7 +2305,11 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, # CIT-local and OSPool. Requires use_singularity. singularity_container_universe = bool(use_singularity and os.environ.get('RIFT_CONTAINER_UNIVERSE')) if singularity_container_universe: - singularity_container_image_select = build_container_image_select(_manifest) + # request_gpu gates per-capability selection vs a single fixed + # container: a job that requests no GPU matches a slot with no GPU + # capability, so a $$() capability expression cannot resolve (it + # holds the job) -- collapse to the single fallback image instead. + singularity_container_image_select = build_container_image_select(_manifest, request_gpu=request_gpu) # Selective transfer: only the matched osdf image is fetched (via the # $$() token, which is comma-free so it survives transfer_input_files # comma-splitting). CVMFS/local images are referenced in place and diff --git a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py index dc8acae9f..d14a3abbd 100644 --- a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py +++ b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py @@ -255,5 +255,42 @@ def test_integration_container_universe(tmp_path, monkeypatch): assert job.universe == "container" # HTCondor container universe +def test_container_image_select_no_gpu_collapses_to_single(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + # A job that requests no GPU has no capability to key on -- a $$() expression + # would not resolve on a CPU-only slot and would HOLD the job. Collapse to + # the single (CPU-safe) fallback image: a plain literal, no $$(), no ifThenElse. + val = cm.build_container_image_select(m, request_gpu=False) + assert val == "/cvmfs/sw/rift_ancient_cuda11.sif" # the fallback image, verbatim + assert "$$(" not in val + assert "ifThenElse" not in val + + +def test_integration_cip_container_universe_single_image(tmp_path, monkeypatch): + # CIP is CPU-only: under container universe it must use a SINGLE fixed + # container (the fallback image), never the $$() capability selection. + monkeypatch.setenv("RIFT_CONTAINER_UNIVERSE", "1") + monkeypatch.chdir(tmp_path) + dag = pytest.importorskip("RIFT.misc.dag_utils_generic") + job, _ = dag.write_CIP_sub( + tag="CIP", + out_dir=str(tmp_path), + log_dir=str(tmp_path) + "/", + exe="/usr/bin/true", + arg_str="--foo bar", + transfer_files=["../all.net"], + use_singularity=True, + singularity_image=_write(tmp_path, MIXED_MANIFEST), + ) + cmds = dict(job.condor_cmds) + ci = cmds["container_image"] + assert ci == "/cvmfs/sw/rift_ancient_cuda11.sif" # single fixed fallback image + assert "$$(" not in ci # NOT a capability $$() selection + assert "MY.SingularityImage" not in cmds + assert "$$([" not in cmds.get("transfer_input_files", "") + assert "require_gpus" not in cmds # CPU job: no GPU floor + assert job.universe == "container" + + if __name__ == "__main__": sys.exit(pytest.main([os.path.abspath(__file__), "-v"])) From 2c018352a434f8c582662a8756618f446aa252d4 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Thu, 11 Jun 2026 16:10:01 -0700 Subject: [PATCH 05/63] container_manifest: undefined-safe capability selector (legacy path no-hold) Folds in the undefined-safe guard (orig 8b9a0c5d, fix/manifest-cpu-fallback) and unifies it with the container-universe collapse already in this branch. build_singularity_image_expr and build_transfer_input_expr emitted a bare ifThenElse/ternary over TARGET.GPUs_Capability with no guard for that attr being undefined. A job that matches a slot with no capability attribute -- a CPU-only CIP slot, OR an OSPool GPU site that doesn't advertise it -- makes every `TARGET.attr >= N` undefined, so the whole $$([...]) token "cannot expand" and HTCondor HOLDS the job ("Cannot expand $$ expression"). Add an `undefined_safe` option to _build_selector that wraps the selector in `TARGET.attr =?= undefined ? fallback : ` (ternary for the comma-free transfer token; ifThenElse otherwise). Apply it to both legacy builders, and refactor build_container_image_select to reuse it (DRY) instead of its own inline guard. An undefined-capability match now yields the fallback (smallest, CPU-safe) image on every path instead of an unresolvable $$(). This is the central no-hold guard for the LEGACY (non-container-universe) path, complementing the deterministic build-time collapse this branch already does for CPU-only jobs under container universe (CIP -> single fallback container). NB: the osdf scitokens credential for manifest images is a separate fix already on dev (3e187939; re-proposed in PR #11) -- not duplicated here. Tests: legacy builders are undefined-safe; updated the two exact-string expression tests to the guarded form. 18/18 pass. Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/container_manifest.py | 39 +++++++++++++++---- .../Code/test/test_container_manifest.py | 23 +++++++++-- 2 files changed, 51 insertions(+), 11 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py index adb3dff5b..b895880f9 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py @@ -217,18 +217,26 @@ def _capability_attr(manifest): return os.environ.get("RIFT_GPU_CAPABILITY_ATTR") or manifest["capability_attr"] -def _build_selector(manifest, value_fn, ternary=False): +def _build_selector(manifest, value_fn, ternary=False, undefined_safe=False): """Build a nested capability selector over the family. ``value_fn(container)`` returns the ClassAd literal for a container branch (already quoted as appropriate). The highest-min container is the outermost test; the ``fallback`` container is the innermost else (catch-all, also used - when the capability attribute is ``undefined``). + when the capability attribute is below every threshold). With ``ternary=False`` the selector uses ``ifThenElse(cond, a, b)`` (commas). With ``ternary=True`` it uses the comma-free ClassAd ternary ``cond ? a : b`` -- required when the result is embedded as one element of a comma-separated ``transfer_input_files`` list, where internal commas would be mis-split. + + With ``undefined_safe=True`` the whole selector is wrapped in a + ``TARGET.attr =?= undefined ? fallback : `` guard. Without it, a + job that matches a slot with NO capability attribute (a CPU-only slot, or an + OSPool GPU site that does not advertise it) makes every ``TARGET.attr >= N`` + test ``undefined`` -> the whole ifThenElse/ternary is ``undefined`` -> a + ``$$([...])`` token "cannot expand" and HTCondor HOLDS the job. The guard + collapses that case to the (CPU-safe) fallback image instead. """ attr = _capability_attr(manifest) containers = manifest["containers"] # sorted desc by min @@ -254,6 +262,12 @@ def _build_selector(manifest, value_fn, ternary=False): expr = "ifThenElse({cond}, {val}, {inner})".format( cond=cond, val=value_fn(c), inner=expr ) + if undefined_safe: + guard = "TARGET.{attr} =?= undefined".format(attr=attr) + if ternary: + expr = "({g} ? {fb} : {sel})".format(g=guard, fb=value_fn(fb), sel=expr) + else: + expr = "ifThenElse({g}, {fb}, {sel})".format(g=guard, fb=value_fn(fb), sel=expr) return expr @@ -262,9 +276,14 @@ def build_singularity_image_expr(manifest): Each branch literal is the container's *runtime* path (CVMFS/local verbatim, ``./`` for transferred images). + + Undefined-safe: a job that matches a slot advertising no capability attribute + (a CPU-only CIP slot, or an OSPool GPU site that does not advertise it) gets + the fallback image rather than an ``undefined`` expression. """ return _build_selector( - manifest, lambda c: '"{}"'.format(_image_runtime_path(c["image"])) + manifest, lambda c: '"{}"'.format(_image_runtime_path(c["image"])), + undefined_safe=True, ) @@ -283,7 +302,11 @@ def build_transfer_input_expr(manifest): def value_fn(c): return '"{}"'.format(c["image"]) if _image_needs_transfer(c["image"]) else '""' - return "$$([ {} ])".format(_build_selector(manifest, value_fn, ternary=True)) + # undefined_safe: a CPU-only / non-advertising slot collapses to the fallback + # image's transfer value instead of an unresolvable $$() that holds the job. + return "$$([ {} ])".format( + _build_selector(manifest, value_fn, ternary=True, undefined_safe=True) + ) def build_container_image_select(manifest, request_gpu=True): @@ -322,10 +345,10 @@ def build_container_image_select(manifest, request_gpu=True): if not request_gpu: # Single fixed container: no capability, no $$() -- a plain literal. return fb_image - attr = _capability_attr(manifest) - selector = _build_selector(manifest, lambda c: '"{}"'.format(c["image"])) - guarded = 'ifThenElse(TARGET.{attr} =?= undefined, "{fb}", {sel})'.format( - attr=attr, fb=fb_image, sel=selector + # undefined_safe: a GPU job that lands on a GPU slot which does not advertise + # the capability attribute still resolves (to the fallback) instead of holding. + guarded = _build_selector( + manifest, lambda c: '"{}"'.format(c["image"]), undefined_safe=True ) return "$$([ {} ])".format(guarded) diff --git a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py index d14a3abbd..5fac4e4da 100644 --- a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py +++ b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py @@ -104,9 +104,13 @@ def test_parser_rejects_missing_image(tmp_path): def test_image_expression(tmp_path): m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) expr = cm.build_singularity_image_expr(m) + # undefined-safe: an outer `=?= undefined -> fallback` guard wraps the + # capability selector, so a slot that does not advertise the attribute (CPU + # slot, or non-advertising GPU site) resolves to the fallback, not undefined. assert expr == ( + 'ifThenElse(TARGET.GPUs_Capability =?= undefined, "/cvmfs/sw/rift_ancient_cuda11.sif", ' 'ifThenElse(TARGET.GPUs_Capability >= 7.0, ' - '"./rift_modern_cuda12.sif", "/cvmfs/sw/rift_ancient_cuda11.sif")' + '"./rift_modern_cuda12.sif", "/cvmfs/sw/rift_ancient_cuda11.sif"))' ) # an expression must NOT be a quoted string literal assert not expr.startswith('"') @@ -115,9 +119,11 @@ def test_image_expression(tmp_path): def test_transfer_expression_is_comma_free_ternary(tmp_path): m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) expr = cm.build_transfer_input_expr(m) + # undefined-safe outer guard; the fallback (a CVMFS image) has transfer value + # "" (no transfer), so an undefined-capability slot transfers nothing. assert expr == ( - '$$([ (TARGET.GPUs_Capability >= 7.0 ? ' - '"osdf:///igwn/rift_modern_cuda12.sif" : "") ])' + '$$([ (TARGET.GPUs_Capability =?= undefined ? "" : ' + '(TARGET.GPUs_Capability >= 7.0 ? "osdf:///igwn/rift_modern_cuda12.sif" : "")) ])' ) # the token sits inside a comma-separated transfer_input_files list, so it # must contain no commas of its own @@ -134,6 +140,17 @@ def test_require_gpus_floor(tmp_path): assert cm.build_require_gpus_floor(m) == "Capability >= 3.0" +def test_legacy_builders_are_undefined_safe(tmp_path): + # Both the MY.SingularityImage expression and the $$() transfer token guard + # against an undefined capability: a job that matches a slot which does not + # advertise it (a CPU-only CIP slot, or an OSPool GPU site that doesn't + # advertise it) resolves to the fallback instead of an unresolvable $$() + # that "cannot expand" -> HOLD. + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + assert "TARGET.GPUs_Capability =?= undefined" in cm.build_singularity_image_expr(m) + assert "=?= undefined" in cm.build_transfer_input_expr(m) + + def test_capability_attr_env_override(tmp_path, monkeypatch): monkeypatch.setenv("RIFT_GPU_CAPABILITY_ATTR", "CUDACapability") m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) From 6e390e2511b8ed63fe8599b48962f7cf12b05a7d Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Fri, 12 Jun 2026 09:25:52 -0700 Subject: [PATCH 06/63] container family: exclude undefined-capability slots (CIT-local holds) + CIP single image Fixes the CIT-LOCAL hold wave (CITLOCAL_BREADCRUMB_gpus_capability_undefined_holds.md): ~45% of CIT GPU slots satisfy the per-GPU require_gpus floor (per-GPU `Capability` inside AvailableGPUs) yet do NOT advertise the machine-level rollup attr `GPUs_Capability` that the family `$$()`/`ifThenElse` selection reads. On those slots the selection "cannot expand" and the job HOLDS (presents as stuck / MachineAttrMachine0=undefined). Measured 621 undefined / 741 defined, spanning node*/aframe/mly (not one bad host). Correct fix = do NOT match undefined-capability slots (don't guess their image): - container_manifest.build_capability_defined_requirement(manifest) -> "TARGET. =!= undefined" (generic on capability_attr; no-op where every GPU slot advertises it). GPU family jobs (ILE, CALPILOT) append it to Requirements. The defined set still includes the cc12.0 Blackwell nodes, so the family's purpose (Blackwell vs older) is preserved. - REVERT the undefined-safe `=?= undefined -> fallback` guard added in the prior PR (now on dev). It is UNSAFE for GPU jobs: an undefined-capability slot could be a Blackwell that hard-fails on the cuda-11.8 fallback -- the exact failure the family exists to avoid. We must not match it, not guess an image. _build_selector / build_singularity_image_expr / build_transfer_input_expr / build_container_image_select are back to a bare selector (fail-loud: an unexcluded undefined slot HOLDS rather than silently running the wrong image). - CIP (CPU, no GPU) holds the same way -- there is no GPU capability at all. CIP needs no GPU/arch-specific image, so it now uses a SINGLE fixed container = the manifest fallback on BOTH paths: legacy MY.SingularityImage = "./" (QUOTED; a bare path is a ClassAd parse error) + transfer just that image; container universe container_image = the fallback URL. New helper build_fallback_single_image(manifest) -> (runtime_path, transfer_url). NOTE: the osdf scitokens credential for manifest images (3e187939) is already on dev. The getenv True->* default (dag_utils_generic vs dag_utils) is a separate, related item the breadcrumb flags -- not addressed here. Tests: capability-defined requirement (+ attr override); fallback single image (cvmfs in place vs osdf transferred); selectors are NOT undefined-guarded; ILE (legacy + container universe) emit the Requirements exclusion; CIP legacy emits a single quoted fallback with no exclusion. 22/22 pass. Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/container_manifest.py | 106 ++++++++++++------ .../Code/RIFT/misc/dag_utils_generic.py | 56 +++++---- .../Code/test/test_container_manifest.py | 98 ++++++++++++---- 3 files changed, 182 insertions(+), 78 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py index b895880f9..64a6850af 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/container_manifest.py @@ -57,6 +57,8 @@ "build_transfer_input_expr", "build_require_gpus_floor", "build_container_image_select", + "build_capability_defined_requirement", + "build_fallback_single_image", ] # Default machine ClassAd attribute advertising GPU compute capability. The @@ -217,26 +219,25 @@ def _capability_attr(manifest): return os.environ.get("RIFT_GPU_CAPABILITY_ATTR") or manifest["capability_attr"] -def _build_selector(manifest, value_fn, ternary=False, undefined_safe=False): +def _build_selector(manifest, value_fn, ternary=False): """Build a nested capability selector over the family. ``value_fn(container)`` returns the ClassAd literal for a container branch (already quoted as appropriate). The highest-min container is the outermost - test; the ``fallback`` container is the innermost else (catch-all, also used - when the capability attribute is below every threshold). + test; the ``fallback`` container is the innermost else (catch-all, used when + the capability is below every threshold). With ``ternary=False`` the selector uses ``ifThenElse(cond, a, b)`` (commas). With ``ternary=True`` it uses the comma-free ClassAd ternary ``cond ? a : b`` -- required when the result is embedded as one element of a comma-separated ``transfer_input_files`` list, where internal commas would be mis-split. - With ``undefined_safe=True`` the whole selector is wrapped in a - ``TARGET.attr =?= undefined ? fallback : `` guard. Without it, a - job that matches a slot with NO capability attribute (a CPU-only slot, or an - OSPool GPU site that does not advertise it) makes every ``TARGET.attr >= N`` - test ``undefined`` -> the whole ifThenElse/ternary is ``undefined`` -> a - ``$$([...])`` token "cannot expand" and HTCondor HOLDS the job. The guard - collapses that case to the (CPU-safe) fallback image instead. + NOTE: the selector is intentionally NOT undefined-guarded. A GPU job must add + a ``Requirements`` clause excluding slots that do not advertise the capability + attribute (:func:`build_capability_defined_requirement`); guessing an image + for an undefined slot is unsafe (it could be a Blackwell that hard-fails on the + older fallback image), so the correct action is to NOT match such a slot. A + non-GPU job must not use this selector at all -- it has no capability to read. """ attr = _capability_attr(manifest) containers = manifest["containers"] # sorted desc by min @@ -262,12 +263,6 @@ def _build_selector(manifest, value_fn, ternary=False, undefined_safe=False): expr = "ifThenElse({cond}, {val}, {inner})".format( cond=cond, val=value_fn(c), inner=expr ) - if undefined_safe: - guard = "TARGET.{attr} =?= undefined".format(attr=attr) - if ternary: - expr = "({g} ? {fb} : {sel})".format(g=guard, fb=value_fn(fb), sel=expr) - else: - expr = "ifThenElse({g}, {fb}, {sel})".format(g=guard, fb=value_fn(fb), sel=expr) return expr @@ -277,13 +272,13 @@ def build_singularity_image_expr(manifest): Each branch literal is the container's *runtime* path (CVMFS/local verbatim, ``./`` for transferred images). - Undefined-safe: a job that matches a slot advertising no capability attribute - (a CPU-only CIP slot, or an OSPool GPU site that does not advertise it) gets - the fallback image rather than an ``undefined`` expression. + GPU jobs that emit this MUST also add + :func:`build_capability_defined_requirement` so they never match a slot that + does not advertise the capability attribute (where this expression would be + ``undefined``). """ return _build_selector( - manifest, lambda c: '"{}"'.format(_image_runtime_path(c["image"])), - undefined_safe=True, + manifest, lambda c: '"{}"'.format(_image_runtime_path(c["image"])) ) @@ -302,11 +297,10 @@ def build_transfer_input_expr(manifest): def value_fn(c): return '"{}"'.format(c["image"]) if _image_needs_transfer(c["image"]) else '""' - # undefined_safe: a CPU-only / non-advertising slot collapses to the fallback - # image's transfer value instead of an unresolvable $$() that holds the job. - return "$$([ {} ])".format( - _build_selector(manifest, value_fn, ternary=True, undefined_safe=True) - ) + # GPU jobs that emit this MUST also add build_capability_defined_requirement so + # they never match a slot where TARGET. is undefined (this $$ token would + # then "cannot expand" and HOLD the job). + return "$$([ {} ])".format(_build_selector(manifest, value_fn, ternary=True)) def build_container_image_select(manifest, request_gpu=True): @@ -335,22 +329,62 @@ def build_container_image_select(manifest, request_gpu=True): -- it fails to expand and HTCondor *holds the job*. There is also nothing to select between, so the CPU-safe fallback image is the right (and only) choice. - (The GPU-path expression is also written undefined-safe -- ``=?= undefined`` - yields the fallback -- but a GPU job that requested a GPU will match a slot - that advertises the capability, so that guard is belt-and-suspenders; the - non-GPU case must not use ``$$()`` at all.) + The GPU-path ``$$()`` is NOT undefined-guarded: the GPU job that emits it MUST + also add :func:`build_capability_defined_requirement` so it never matches a + slot where the capability attr is undefined (guessing an image for such a slot + is unsafe -- it could be a Blackwell that hard-fails on the older fallback). + The non-GPU case never reaches the ``$$()`` (it returns the literal fallback). """ by_label = {c["label"]: c for c in manifest["containers"]} fb_image = by_label[manifest["fallback"]]["image"] if not request_gpu: # Single fixed container: no capability, no $$() -- a plain literal. return fb_image - # undefined_safe: a GPU job that lands on a GPU slot which does not advertise - # the capability attribute still resolves (to the fallback) instead of holding. - guarded = _build_selector( - manifest, lambda c: '"{}"'.format(c["image"]), undefined_safe=True - ) - return "$$([ {} ])".format(guarded) + selector = _build_selector(manifest, lambda c: '"{}"'.format(c["image"])) + return "$$([ {} ])".format(selector) + + +def build_capability_defined_requirement(manifest): + """Return a ``Requirements`` clause that excludes machines which do not + advertise the capability attribute the family selection reads. + + A GPU family job MUST add this. Measured on the CIT pool (2026-06-12), ~45% + of GPU slots satisfy the per-GPU ``require_gpus`` floor (which matches the + per-GPU ``Capability`` inside ``AvailableGPUs``) yet do NOT advertise the + machine-level rollup attribute (default ``GPUs_Capability``) that the + ``$$()``/``ifThenElse`` selection reads. On such a slot the selection cannot + expand and the job HOLDS ("Cannot expand $$ expression"). Excluding these + slots is the safe fix: an undefined-capability slot could be a Blackwell that + hard-fails on the older fallback image, so we must NOT match it (rather than + guess its image). The defined set still includes the high-capability nodes, + so the family's purpose is preserved. + + Generic on ``capability_attr``; a no-op on pools where every GPU slot + advertises it, hence merge-safe. + """ + return "TARGET.{attr} =!= undefined".format(attr=_capability_attr(manifest)) + + +def build_fallback_single_image(manifest): + """For jobs that must use a SINGLE fixed container (no capability selection) -- + e.g. CPU-only CIP, which requests no GPU and so cannot resolve a + ``$$()``/``ifThenElse`` capability selection (its matched slot advertises no + capability attribute -> the selection holds the job). + + Returns ``(runtime_path, transfer_url)`` for the manifest ``fallback`` (the + CPU-safe image): + + * ``runtime_path`` -- what ``MY.SingularityImage`` / ``container_image`` + references: ``./`` for a transferred ``osdf://`` image, the path + verbatim for a CVMFS/local image. (``MY.SingularityImage`` callers must + quote it; ``container_image`` takes it unquoted.) + * ``transfer_url`` -- the ``osdf://`` URL to add to ``transfer_input_files``, + or ``None`` if the image is referenced in place (CVMFS/local). + """ + fb_image = {c["label"]: c for c in manifest["containers"]}[manifest["fallback"]]["image"] + runtime_path = _image_runtime_path(fb_image) + transfer_url = fb_image if _image_needs_transfer(fb_image) else None + return runtime_path, transfer_url def build_require_gpus_floor(manifest): diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py index f20b1db71..1baf07cad 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py @@ -136,6 +136,8 @@ def emit_dag(self, dag, path): build_transfer_input_expr, build_require_gpus_floor, build_container_image_select, + build_capability_defined_requirement, + build_fallback_single_image, ContainerManifestError, ) _HAVE_CONTAINER_MANIFEST = True @@ -1923,34 +1925,29 @@ def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output- singularity_image_used = "{}".format(singularity_image) # make copy extra_files = [] - # Container family manifest support (see write_ILE_sub_simple). CIP jobs do - # NOT request a GPU, so no require_gpus floor is added here. For the - # container-universe path that means the per-capability $$() selection MUST - # NOT be used: a CPU-only slot advertises no GPU capability attribute, so a - # $$() capability expression cannot resolve and HTCondor holds the job. We - # collapse to the single (CPU-safe) fallback image instead. (For the legacy - # MY.SingularityImage=ifThenElse path the same family expression is emitted; - # on CIT-local native singularity it degrades to the fallback branch.) + # Container family manifest support. CIP is CPU-only: it requests NO GPU, so a + # per-capability $$()/ifThenElse selection cannot resolve on its matched slot + # (no capability attribute -> the $$ "cannot expand" -> HOLD). CIP needs no GPU + # and no arch-specific image, so it uses a SINGLE fixed container = the manifest + # fallback (CPU-safe) image on BOTH the legacy and container-universe paths. singularity_is_family = False singularity_container_universe = False singularity_container_image_select = None - singularity_image_expr = None - singularity_transfer_expr = None + singularity_fallback_runtime = None if singularity_image and is_container_manifest(singularity_image): singularity_is_family = True _manifest = load_container_manifest(singularity_image) - singularity_image_expr = build_singularity_image_expr(_manifest) - singularity_transfer_expr = build_transfer_input_expr(_manifest) - # Container universe (opt-in: RIFT_CONTAINER_UNIVERSE). CIP is CPU-only, - # so request_gpu=False -> a SINGLE fixed container (the fallback image), - # NOT a $$() capability selection that a CPU slot can't resolve. singularity_container_universe = bool(use_singularity and os.environ.get('RIFT_CONTAINER_UNIVERSE')) if singularity_container_universe: + # container_image = the fallback image URL verbatim (container universe + # fetches it); request_gpu=False -> single image, no $$() selection. singularity_container_image_select = build_container_image_select(_manifest, request_gpu=False) - # Container universe delivers the image via container_image itself, so - # skip the $$() transfer token in that mode. - if singularity_transfer_expr and not singularity_container_universe: - extra_files += [singularity_transfer_expr] + else: + # legacy: MY.SingularityImage = the single fallback (quoted in the image + # block below); transfer just that one image if it is an osdf URL. + singularity_fallback_runtime, _fb_transfer = build_fallback_single_image(_manifest) + if _fb_transfer: + extra_files += [_fb_transfer] elif singularity_image: if 'osdf:' in singularity_image: singularity_image_used = "./{}".format(singularity_image.split('/')[-1]) @@ -2079,8 +2076,10 @@ def write_CIP_sub(tag='integrate', exe=None, input_net='all.net',output='output- else: ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') if singularity_is_family: - # Expression-valued: emit raw, NO surrounding double quotes. - ile_job.add_condor_cmd("MY.SingularityImage", singularity_image_expr) + # CPU-only CIP -> single fixed (quoted) fallback image, NOT the + # family capability selection (a CPU slot can't resolve it). A + # bare/unquoted path is a ClassAd parse error, so quote it. + ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_fallback_runtime + '"') else: ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') requirements.append("HAS_SINGULARITY=?=TRUE") @@ -2610,6 +2609,15 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, for name in name_list: requirements.append('TARGET.Machine =!= "{}" '.format(name)) + # Container-family GPU jobs: exclude slots that don't advertise the machine-level + # capability attribute the per-machine image selection reads. ~45% of CIT GPU + # slots satisfy the per-GPU require_gpus floor but don't advertise the rollup + # attr, so the $$()/ifThenElse "cannot expand" and the job HOLDS. Excluding + # them is safe; guessing an image is not (an undefined slot could be a Blackwell + # that hard-fails on the older fallback). Only when a GPU is actually requested. + if singularity_is_family and request_gpu: + requirements.append(build_capability_defined_requirement(_manifest)) + # Write requirements # From https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py ile_job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) @@ -2969,6 +2977,12 @@ def write_calpilot_sub(tag='calpilot', exe=None, log_dir=None, universe="vanilla job.add_condor_cmd('should_transfer_files', 'YES') job.add_condor_cmd('when_to_transfer_output', 'ON_EXIT') job.add_condor_cmd('transfer_output_files', 'cal_consolidated_$(macroiteration).npz') + # Container-family GPU jobs (CALPILOT runs ILE on a GPU): exclude slots that + # don't advertise the capability attr the per-machine image selection reads, + # else the $$()/ifThenElse "cannot expand" and the job HOLDS (see ILE). + if singularity_is_family and request_gpu: + requirements.append(build_capability_defined_requirement(_manifest)) + if requirements: job.add_condor_cmd('requirements', '&&'.join('({0})'.format(r) for r in requirements)) diff --git a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py index 5fac4e4da..df9c06a8f 100644 --- a/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py +++ b/MonteCarloMarginalizeCode/Code/test/test_container_manifest.py @@ -104,13 +104,12 @@ def test_parser_rejects_missing_image(tmp_path): def test_image_expression(tmp_path): m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) expr = cm.build_singularity_image_expr(m) - # undefined-safe: an outer `=?= undefined -> fallback` guard wraps the - # capability selector, so a slot that does not advertise the attribute (CPU - # slot, or non-advertising GPU site) resolves to the fallback, not undefined. + # Bare capability selector (NOT undefined-guarded): a GPU job must instead add + # the build_capability_defined_requirement Requirements clause so it never + # matches a slot where this would be undefined. assert expr == ( - 'ifThenElse(TARGET.GPUs_Capability =?= undefined, "/cvmfs/sw/rift_ancient_cuda11.sif", ' 'ifThenElse(TARGET.GPUs_Capability >= 7.0, ' - '"./rift_modern_cuda12.sif", "/cvmfs/sw/rift_ancient_cuda11.sif"))' + '"./rift_modern_cuda12.sif", "/cvmfs/sw/rift_ancient_cuda11.sif")' ) # an expression must NOT be a quoted string literal assert not expr.startswith('"') @@ -119,11 +118,9 @@ def test_image_expression(tmp_path): def test_transfer_expression_is_comma_free_ternary(tmp_path): m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) expr = cm.build_transfer_input_expr(m) - # undefined-safe outer guard; the fallback (a CVMFS image) has transfer value - # "" (no transfer), so an undefined-capability slot transfers nothing. assert expr == ( - '$$([ (TARGET.GPUs_Capability =?= undefined ? "" : ' - '(TARGET.GPUs_Capability >= 7.0 ? "osdf:///igwn/rift_modern_cuda12.sif" : "")) ])' + '$$([ (TARGET.GPUs_Capability >= 7.0 ? ' + '"osdf:///igwn/rift_modern_cuda12.sif" : "") ])' ) # the token sits inside a comma-separated transfer_input_files list, so it # must contain no commas of its own @@ -140,15 +137,41 @@ def test_require_gpus_floor(tmp_path): assert cm.build_require_gpus_floor(m) == "Capability >= 3.0" -def test_legacy_builders_are_undefined_safe(tmp_path): - # Both the MY.SingularityImage expression and the $$() transfer token guard - # against an undefined capability: a job that matches a slot which does not - # advertise it (a CPU-only CIP slot, or an OSPool GPU site that doesn't - # advertise it) resolves to the fallback instead of an unresolvable $$() - # that "cannot expand" -> HOLD. +def test_selectors_are_not_undefined_guarded(tmp_path): + # The capability selectors must NOT default an undefined-capability slot to the + # fallback image: that slot could be a Blackwell that hard-fails on the older + # fallback. The safe fix is the Requirements exclusion below, not a guess. m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) - assert "TARGET.GPUs_Capability =?= undefined" in cm.build_singularity_image_expr(m) - assert "=?= undefined" in cm.build_transfer_input_expr(m) + assert "=?= undefined" not in cm.build_singularity_image_expr(m) + assert "=?= undefined" not in cm.build_transfer_input_expr(m) + assert "=?= undefined" not in cm.build_container_image_select(m) + + +def test_capability_defined_requirement(tmp_path): + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + # excludes slots that don't advertise the (machine-level) capability attr + assert cm.build_capability_defined_requirement(m) == "TARGET.GPUs_Capability =!= undefined" + + +def test_capability_defined_requirement_respects_attr_override(tmp_path, monkeypatch): + monkeypatch.setenv("RIFT_GPU_CAPABILITY_ATTR", "CUDACapability") + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + assert cm.build_capability_defined_requirement(m) == "TARGET.CUDACapability =!= undefined" + + +def test_fallback_single_image(tmp_path): + # MIXED fallback (ancient) is a CVMFS image -> referenced in place, no transfer + m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) + runtime, transfer = cm.build_fallback_single_image(m) + assert runtime == "/cvmfs/sw/rift_ancient_cuda11.sif" + assert transfer is None + # an osdf fallback -> runtime is ./basename and it IS transferred + osdf_text = MIXED_MANIFEST.replace("/cvmfs/sw/rift_ancient_cuda11.sif", + "osdf:///igwn/rift_ancient_cuda11.sif") + m2 = cm.load_container_manifest(_write(tmp_path, osdf_text, name="fam2.yaml")) + runtime2, transfer2 = cm.build_fallback_single_image(m2) + assert runtime2 == "./rift_ancient_cuda11.sif" + assert transfer2 == "osdf:///igwn/rift_ancient_cuda11.sif" def test_capability_attr_env_override(tmp_path, monkeypatch): @@ -204,6 +227,10 @@ def test_integration_family_mixed(tmp_path, monkeypatch): assert 'DeviceName=!="Tesla K10.G1.8GB"' in rg assert "&&" in rg + # GPU family job: Requirements exclude slots that don't advertise the + # machine-level capability attr (else the selection $$/ifThenElse holds). + assert "TARGET.GPUs_Capability =!= undefined" in cmds["requirements"] + def test_integration_all_cvmfs_no_transfer_token(tmp_path, monkeypatch): cmds = _make_ile_job(tmp_path, monkeypatch, _write(tmp_path, ALL_CVMFS_MANIFEST)) @@ -229,11 +256,11 @@ def test_backward_compat_single_sif(tmp_path, monkeypatch): def test_container_image_select_expression(tmp_path): m = cm.load_container_manifest(_write(tmp_path, MIXED_MANIFEST)) expr = cm.build_container_image_select(m) - # a $$() match-time substitution token, undefined-safe, with VERBATIM image - # values (osdf URL fetched by container universe; cvmfs path used in place) -- - # NOT a ./basename rewrite + # a $$() match-time substitution token with VERBATIM image values (osdf URL + # fetched by container universe; cvmfs path used in place) -- NOT a ./basename + # rewrite, and NOT undefined-guarded (Requirements exclusion is used instead) assert expr.startswith("$$([ ") and expr.endswith(" ])") - assert "TARGET.GPUs_Capability =?= undefined" in expr # undefined -> fallback + assert "=?= undefined" not in expr # not a guess-guard assert "ifThenElse(TARGET.GPUs_Capability >= 7.0," in expr assert '"osdf:///igwn/rift_modern_cuda12.sif"' in expr # raw osdf URL assert '"/cvmfs/sw/rift_ancient_cuda11.sif"' in expr # fallback verbatim @@ -268,6 +295,8 @@ def test_integration_container_universe(tmp_path, monkeypatch): assert "MY.SingularityBindCVMFS" not in cmds assert "$$([" not in cmds.get("transfer_input_files", "") # image via container_image, not transfer assert "Capability >= 3.0" in cmds["require_gpus"] # floor still steers GPUs + # GPU family job: still excludes slots that don't advertise the capability attr + assert "TARGET.GPUs_Capability =!= undefined" in cmds["requirements"] assert job.universe == "container" # HTCondor container universe @@ -309,5 +338,32 @@ def test_integration_cip_container_universe_single_image(tmp_path, monkeypatch): assert job.universe == "container" +def test_integration_cip_legacy_single_image(tmp_path, monkeypatch): + # CIP (CPU-only) on the LEGACY path: a single QUOTED fallback MY.SingularityImage + # (a bare path is a ClassAd parse error), NOT the family $$()/ifThenElse selection + # (a CPU slot can't resolve it), no $$() transfer token, and NO capability + # Requirements exclusion (CIP requests no GPU, so it must not be GPU-constrained). + monkeypatch.delenv("RIFT_CONTAINER_UNIVERSE", raising=False) + monkeypatch.chdir(tmp_path) + dag = pytest.importorskip("RIFT.misc.dag_utils_generic") + job, _ = dag.write_CIP_sub( + tag="CIP", + out_dir=str(tmp_path), + log_dir=str(tmp_path) + "/", + exe="/usr/bin/true", + arg_str="--foo bar", + transfer_files=["../all.net"], + use_singularity=True, + singularity_image=_write(tmp_path, MIXED_MANIFEST), + ) + cmds = dict(job.condor_cmds) + assert cmds["MY.SingularityImage"] == '"/cvmfs/sw/rift_ancient_cuda11.sif"' # single, quoted + assert "ifThenElse" not in cmds["MY.SingularityImage"] + assert "container_image" not in cmds + assert "$$([" not in cmds.get("transfer_input_files", "") + assert "=!= undefined" not in cmds.get("requirements", "") # CPU job: no GPU exclusion + assert "require_gpus" not in cmds + + if __name__ == "__main__": sys.exit(pytest.main([os.path.abspath(__file__), "-v"])) From a78b4cb60cfa611baaaee7901bdc6532879ed5b9 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Fri, 12 Jun 2026 09:33:09 -0700 Subject: [PATCH 07/63] dag_utils_generic: default getenv '*' not 'True' (CIT SUBMIT_ALLOW_GETENV=false) dag_utils_generic.py defaulted default_getenv_value / default_getenv_osg_value to 'True', emitting `getenv = True`, which schedds with SUBMIT_ALLOW_GETENV=false (e.g. CIT) reject -> the DAG aborts. The newer dag_utils.py already defaults '*' (all-env, the modern form); bring generic in line (value-only change, file's own formatting preserved to minimize a later oshaughn/rift_O4d->rift merge conflict). Still overridable via RIFT_GETENV / RIFT_GETENV_OSG. Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/dag_utils_generic.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py index 1baf07cad..215db4249 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py @@ -158,9 +158,12 @@ def is_container_manifest(value): # Utility helpers # =========================================================================== -# getenv=True deprecated, will need workaround to explicitly pull extra environment variables -default_getenv_value = 'True' -default_getenv_osg_value = 'True' +# getenv=True deprecated, will need workaround to explicitly pull extra environment variables. +# Default to '*' (all env, the modern condor form) to match dag_utils.py: a literal +# `getenv = True` is rejected by schedds with SUBMIT_ALLOW_GETENV=false (e.g. CIT) and +# aborts the DAG. Override with RIFT_GETENV / RIFT_GETENV_OSG. +default_getenv_value = '*' +default_getenv_osg_value = '*' if 'RIFT_GETENV' in os.environ: default_getenv_value = os.environ['RIFT_GETENV'] if 'RIFT_GETENV_OSG' in os.environ: From bc949f63bc55e3cba70207452f786db9e74ef0ed Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Sat, 27 Jun 2026 02:05:39 -0700 Subject: [PATCH 08/63] ILE: optional multi-GPU fan-out per batch job (RIFT_ILE_GPU_FANOUT) A single integrate_likelihood_extrinsic_batchmode invocation evaluates the contiguous intrinsic-grid range [--event, --event+--n-events-to-analyze) serially on ONE GPU. On clusters where whole multi-GPU nodes are reserved but ILE only requests request_GPUs=1, the remaining GPUs sit idle (e.g. macrongroup ~100 points per job on a 4-GPU node uses 1/4 of the hardware). ile_pre.sh now wraps the ILE executable in a small launcher that, when opted in, splits that point range into N disjoint shards run concurrently -- one per GPU. Each shard is pinned with CUDA_VISIBLE_DEVICES and given a distinct --output-file prefix (.gpu), so the per-point output files (__.dat / .xml.gz) never collide. Downstream collection is unaffected: util_ILEdagPostprocess.sh globs CME*.dat and util_CleanILE.py de-duplicates by parameter value, not filename. The shards partition the range exactly (sizes differ by <=1), so coverage is identical to the serial run; the launcher's exit code is the first non-zero shard code, preserving condor retry/hold behaviour (e.g. CUDA hard-fail 62). Controlled by env var RIFT_ILE_GPU_FANOUT (propagated to jobs via getenv=*RIFT*): unset / "0" / "1" -> no fan-out; the launcher exec()s the binary unchanged, so default behaviour is byte-for-byte identical. "auto" -> one shard per visible GPU (CUDA_VISIBLE_DEVICES, else nvidia-smi); for a whole node held with request_GPUs=1. -> up to N shards (capped by #GPUs and #points); the DAG also requests request_GPUs=N and request_CPUs=N so HTCondor assigns the devices. Changes are mirrored in dag_utils.py and dag_utils_generic.py (each carries its own copy of write_ILE_sub_simple). request_CPUs is threaded through the singularity branch so the fan-out CPU count is not clobbered back to 1. Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/dag_utils.py | 137 +++++++++++++++++- .../Code/RIFT/misc/dag_utils_generic.py | 132 ++++++++++++++++- 2 files changed, 260 insertions(+), 9 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py index ab2b63ec1..4132535f5 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py @@ -55,6 +55,124 @@ def which(program): return None + +# ---------------------------------------------------------------------------- +# Multi-GPU ILE fan-out +# +# A single ILE "batchmode" invocation processes the contiguous intrinsic-grid +# range [--event, --event+--n-events-to-analyze) serially on ONE GPU. On nodes +# where several GPUs are reserved (request_GPUs>1, or a whole node is held), the +# other GPUs sit idle. The launcher below splits that range into N disjoint +# shards run concurrently -- one per GPU -- each pinned with CUDA_VISIBLE_DEVICES +# and given a distinct --output-file prefix (downstream collection globs +# CME*.dat / EXTR*, so distinct shard names are harmless and de-duplication is +# by parameter value, not filename). +# +# Behaviour is controlled at job runtime by the env var RIFT_ILE_GPU_FANOUT, +# propagated to the job through the existing getenv=*RIFT* mechanism: +# unset / "0" / "1" -> no fan-out, exec the binary unchanged (no overhead) +# "auto" -> one shard per visible GPU (CUDA_VISIBLE_DEVICES, else +# nvidia-smi). Use when a whole node is reserved but +# only request_GPUs=1 is requested. +# -> up to N shards (capped by #GPUs and #points). Pair +# with request_GPUs=N so HTCondor assigns N devices. +# ---------------------------------------------------------------------------- +ILE_MULTIGPU_LAUNCHER_PY = r''' +import os, sys, subprocess +def _val(argv, names): + v=None + for i,a in enumerate(argv): + for nm in names: + if a==nm and i+10: out.append((s,c)); s+=c + return out +exe=sys.argv[1]; ile=sys.argv[2:] +fan=os.environ.get("RIFT_ILE_GPU_FANOUT","1").strip().lower() +event=_val(ile,["--event","-E"]); ngroup=_val(ile,["--n-events-to-analyze"]) +outfile=_val(ile,["--output-file","-o"]) +event=int(event) if event is not None else 0 +ngroup=int(ngroup) if ngroup is not None else 1 +devs=_devices() +if fan in ("","0","1"): n=1 +elif fan=="auto": n=len(devs) +else: + try: n=int(fan) + except ValueError: n=1 +n=max(1,min(n,len(devs),ngroup)) +if n<=1 or outfile is None: + os.execvp(exe,[exe]+ile) +procs=[] +for i,(cs,cc) in enumerate(_partition(event,ngroup,n)): + dev=devs[i%len(devs)] + a=_setopt(ile,["--event","-E"],cs) + a=_setopt(a,["--n-events-to-analyze"],cc) + a=_setopt(a,["--output-file","-o"],"{}.gpu{}".format(outfile,dev)) + env=dict(os.environ); env["CUDA_VISIBLE_DEVICES"]=str(dev) + sys.stderr.write("[rift_ile_multigpu] shard {} GPU {} events [{},{}) -> {}.gpu{}\n".format(i,dev,cs,cs+cc,outfile,dev)) + procs.append(subprocess.Popen([exe]+a,env=env)) +rc=0 +for p in procs: + r=p.wait() + if r!=0 and rc==0: rc=r +sys.exit(rc) +''' + + +def ile_invocation_shell(exe): + """Return the shell snippet that invokes the ILE executable `exe` with + multi-GPU fan-out (see RIFT_ILE_GPU_FANOUT). The launcher reads its source + from stdin via a quoted here-doc, so the ILE arguments in "$@" (which include + nested-quoted values) are passed through untouched. With fan-out disabled + the launcher simply exec()s the binary, so default behaviour is unchanged.""" + return ( + 'PY=python3; command -v python3 >/dev/null 2>&1 || PY=python\n' + 'exec "$PY" - "{exe}" "$@" <<\'RIFT_ILE_MULTIGPU_EOF\'\n'.format(exe=exe) + + ILE_MULTIGPU_LAUNCHER_PY + + '\nRIFT_ILE_MULTIGPU_EOF\n' + ) + + +def ile_gpu_fanout_count(): + """Concrete number of GPUs an ILE job should *request* for fan-out, parsed + from RIFT_ILE_GPU_FANOUT. Returns an int >=1. 'auto' (split across whatever + is visible at runtime) cannot size a request ahead of time, so it returns 1 + and relies on the user reserving the node.""" + fan = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip().lower() + if fan in ('', '0', '1', 'auto'): + return 1 + try: + return max(1, int(fan)) + except ValueError: + return 1 + + def mkdir(dir_name): try : os.mkdir(dir_name) @@ -982,18 +1100,29 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, ile_job.add_condor_cmd('request_disk', str(request_disk)) nGPUs =0 requirements = [] + ile_gpu_cpus = 1 # CPUs to drive the ILE GPU job(s); >1 for multi-GPU fan-out if request_gpu: nGPUs=1 if request_cross_platform: # recipe from https://opensciencegrid.atlassian.net/browse/HTCONDOR-2200 nGPUs = 'countMatches(RequireGPUs, AvailableGPUs) >= 1 ? 1 : 0' ile_job.add_condor_cmd('rank', 'RequestGPUs') - ile_job.add_condor_cmd('request_GPUs', str(nGPUs)) + else: + # Multi-GPU fan-out (RIFT_ILE_GPU_FANOUT=N): reserve N GPUs (and N + # CPUs to drive them) so HTCondor hands this job the whole-node GPUs + # that ile_pre.sh then splits the intrinsic-grid range across. + fanout = ile_gpu_fanout_count() + if fanout > 1: + nGPUs = fanout + ile_gpu_cpus = fanout + if not use_singularity: + ile_job.add_condor_cmd('request_CPUs', str(fanout)) + ile_job.add_condor_cmd('request_GPUs', str(nGPUs)) # Claim we don't need to make this request anymore to avoid out-of-memory errors. Also, no longer in 'requirements' -# requirements.append("CUDAGlobalMemoryMb >= 2048") +# requirements.append("CUDAGlobalMemoryMb >= 2048") if use_singularity: # Compare to https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py - ile_job.add_condor_cmd('request_CPUs', str(1)) + ile_job.add_condor_cmd('request_CPUs', str(ile_gpu_cpus)) ile_job.add_condor_cmd('transfer_executable', 'False') ile_job.add_condor_cmd("MY.SingularityBindCVMFS", 'True') ile_job.add_condor_cmd("MY.SingularityImage", '"' + singularity_image_used + '"') @@ -1071,7 +1200,7 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, f.write("for i in `ls " + frames_local + "`; do echo "+ frames_local + "/$i; done > base_paths.dat \n") f.write("paste local_stripped.cache base_paths.dat > local_relative.cache \n") f.write("cp local_relative.cache local.cache \n") - f.write('{exe} "$@" '.format(exe=exe)) + f.write(ile_invocation_shell(exe)) os.system("chmod a+x ile_pre.sh") ile_job.set_executable("ile_pre.sh") # transferred, used as executable # ile_job.add_condor_cmd('+PreCmd', '"ile_pre.sh"') diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py index 215db4249..8756f1d2a 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py @@ -187,6 +187,118 @@ def which(program): return None +# ---------------------------------------------------------------------------- +# Multi-GPU ILE fan-out (see the matching block in dag_utils.py for details) +# +# A single ILE "batchmode" invocation processes the contiguous intrinsic-grid +# range [--event, --event+--n-events-to-analyze) serially on ONE GPU. On nodes +# where several GPUs are reserved, the others sit idle. The launcher below +# splits that range into N disjoint shards run concurrently -- one per GPU -- +# each pinned with CUDA_VISIBLE_DEVICES and given a distinct --output-file +# prefix (downstream collection globs CME*.dat / EXTR*, so distinct shard names +# are harmless; de-duplication is by parameter value, not filename). +# +# Controlled at job runtime by RIFT_ILE_GPU_FANOUT (propagated via getenv=*RIFT*): +# unset / "0" / "1" -> no fan-out, exec the binary unchanged (no overhead) +# "auto" -> one shard per visible GPU (CUDA_VISIBLE_DEVICES, else +# nvidia-smi); for a whole node reserved with request_GPUs=1 +# -> up to N shards (capped by #GPUs and #points); pair +# with request_GPUs=N so HTCondor assigns N devices. +# ---------------------------------------------------------------------------- +ILE_MULTIGPU_LAUNCHER_PY = r''' +import os, sys, subprocess +def _val(argv, names): + v=None + for i,a in enumerate(argv): + for nm in names: + if a==nm and i+10: out.append((s,c)); s+=c + return out +exe=sys.argv[1]; ile=sys.argv[2:] +fan=os.environ.get("RIFT_ILE_GPU_FANOUT","1").strip().lower() +event=_val(ile,["--event","-E"]); ngroup=_val(ile,["--n-events-to-analyze"]) +outfile=_val(ile,["--output-file","-o"]) +event=int(event) if event is not None else 0 +ngroup=int(ngroup) if ngroup is not None else 1 +devs=_devices() +if fan in ("","0","1"): n=1 +elif fan=="auto": n=len(devs) +else: + try: n=int(fan) + except ValueError: n=1 +n=max(1,min(n,len(devs),ngroup)) +if n<=1 or outfile is None: + os.execvp(exe,[exe]+ile) +procs=[] +for i,(cs,cc) in enumerate(_partition(event,ngroup,n)): + dev=devs[i%len(devs)] + a=_setopt(ile,["--event","-E"],cs) + a=_setopt(a,["--n-events-to-analyze"],cc) + a=_setopt(a,["--output-file","-o"],"{}.gpu{}".format(outfile,dev)) + env=dict(os.environ); env["CUDA_VISIBLE_DEVICES"]=str(dev) + sys.stderr.write("[rift_ile_multigpu] shard {} GPU {} events [{},{}) -> {}.gpu{}\n".format(i,dev,cs,cs+cc,outfile,dev)) + procs.append(subprocess.Popen([exe]+a,env=env)) +rc=0 +for p in procs: + r=p.wait() + if r!=0 and rc==0: rc=r +sys.exit(rc) +''' + + +def ile_invocation_shell(exe): + """Shell snippet invoking the ILE executable `exe` with multi-GPU fan-out + (RIFT_ILE_GPU_FANOUT). The launcher reads its source from stdin via a quoted + here-doc, so the ILE arguments in "$@" (including nested-quoted values) pass + through untouched; with fan-out disabled it just exec()s the binary.""" + return ( + 'PY=python3; command -v python3 >/dev/null 2>&1 || PY=python\n' + 'exec "$PY" - "{exe}" "$@" <<\'RIFT_ILE_MULTIGPU_EOF\'\n'.format(exe=exe) + + ILE_MULTIGPU_LAUNCHER_PY + + '\nRIFT_ILE_MULTIGPU_EOF\n' + ) + + +def ile_gpu_fanout_count(): + """Concrete number of GPUs an ILE job should *request*, parsed from + RIFT_ILE_GPU_FANOUT. Returns int >=1; 'auto' returns 1 (it cannot size a + request ahead of time and relies on the user reserving the node).""" + fan = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip().lower() + if fan in ('', '0', '1', 'auto'): + return 1 + try: + return max(1, int(fan)) + except ValueError: + return 1 + + def mkdir(dir_name): try: os.mkdir(dir_name) @@ -2470,18 +2582,29 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, ile_job.add_condor_cmd('request_disk', str(request_disk)) nGPUs =0 requirements = [] + ile_gpu_cpus = 1 # CPUs to drive the ILE GPU job(s); >1 for multi-GPU fan-out if request_gpu: nGPUs=1 if request_cross_platform: # recipe from https://opensciencegrid.atlassian.net/browse/HTCONDOR-2200 nGPUs = 'countMatches(RequireGPUs, AvailableGPUs) >= 1 ? 1 : 0' ile_job.add_condor_cmd('rank', 'RequestGPUs') - ile_job.add_condor_cmd('request_GPUs', str(nGPUs)) + else: + # Multi-GPU fan-out (RIFT_ILE_GPU_FANOUT=N): reserve N GPUs (+N CPUs) + # so HTCondor hands this job the whole-node GPUs that ile_pre.sh then + # splits the intrinsic-grid range across. + fanout = ile_gpu_fanout_count() + if fanout > 1: + nGPUs = fanout + ile_gpu_cpus = fanout + if not use_singularity: + ile_job.add_condor_cmd('request_CPUs', str(fanout)) + ile_job.add_condor_cmd('request_GPUs', str(nGPUs)) # Claim we don't need to make this request anymore to avoid out-of-memory errors. Also, no longer in 'requirements' -# requirements.append("CUDAGlobalMemoryMb >= 2048") +# requirements.append("CUDAGlobalMemoryMb >= 2048") if use_singularity: # Compare to https://github.com/lscsoft/lalsuite/blob/master/lalinference/python/lalinference/lalinference_pipe_utils.py - ile_job.add_condor_cmd('request_CPUs', str(1)) + ile_job.add_condor_cmd('request_CPUs', str(ile_gpu_cpus)) ile_job.add_condor_cmd('transfer_executable', 'False') if singularity_container_universe: # Container universe: the per-machine image is delivered via @@ -2588,7 +2711,7 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, f.write("for i in `ls " + frames_local + "`; do echo "+ frames_local + "/$i; done > base_paths.dat \n") f.write("paste local_stripped.cache base_paths.dat > local_relative.cache \n") f.write("cp local_relative.cache local.cache \n") - f.write('{exe} "$@" '.format(exe=exe)) + f.write(ile_invocation_shell(exe)) os.system("chmod a+x ile_pre.sh") ile_job.set_executable("ile_pre.sh") # transferred, used as executable # ile_job.add_condor_cmd('+PreCmd', '"ile_pre.sh"') @@ -4886,4 +5009,3 @@ def write_hyperpost_sub(tag='HYPER', exe=None, input_net='all.marg_net',output=' return ile_job, ile_sub_name - From 4cafc5dd2210ffc826cb4d56b366990bf6d96858 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Sat, 27 Jun 2026 03:40:30 -0700 Subject: [PATCH 09/63] ILE multi-GPU fan-out: CLI flag + asimov-safe baking + demo Make the fan-out usable through every front-end, including asimov (which builds the DAG in a clean environment and cannot rely on RIFT_ILE_GPU_FANOUT being exported in the submit shell): - Bake the resolved fan-out value into the generated ile_pre.sh as export RIFT_ILE_GPU_FANOUT="${RIFT_ILE_GPU_FANOUT:-N}", so the job needs NO runtime environment (a runtime value still overrides). ile_invocation_shell() now takes the value; ile_gpu_fanout_value() resolves it at build time. Mirrored in dag_utils.py and dag_utils_generic.py. - Add --ile-gpu-fanout to util_RIFT_pseudo_pipe.py and create_event_parameter_pipeline_BasicIteration; both funnel it through RIFT_ILE_GPU_FANOUT (pseudo_pipe runs BasicIteration via os.system, inheriting the env), so request_GPUs/CPUs sizing and ile_pre.sh baking happen on one path. Asimov needs no code change: a blueprint sets the value via scheduler.environment variables: {RIFT_ILE_GPU_FANOUT: N} (rift.py copies it into os.environ before running the pipeline) or scheduler.pipeline: {ile-gpu-fanout: N} (-> CLI flag). Demo: demo/rift/infra/multi_gpu/ (README, Makefile, CI ini, asimov blueprint + frozen container-family pin, fake_ile stub). - make smoke-local: builds a REAL ile_pre.sh from the shipped helper around a stub ILE and runs it across this node's GPUs; asserts exact coverage, GPU spread, distinct per-shard output prefixes. Runs anywhere (no cupy/condor). - make build / make verify: builds a real pipeline run dir on the CI synthetic data (singularity/OSG so ile_pre.sh is emitted) with --ile-gpu-fanout and asserts ILE.sub gets request_GPUs=N/request_CPUs=N and ile_pre.sh bakes N. Verified end-to-end: ILE.sub -> request_GPUs=4/request_CPUs=4, ile_pre.sh -> RIFT_ILE_GPU_FANOUT:-4 wrapping the container ILE binary; default stays 1 (no-op). Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/dag_utils.py | 20 ++- .../Code/RIFT/misc/dag_utils_generic.py | 19 ++- ...te_event_parameter_pipeline_BasicIteration | 7 + .../Code/bin/util_RIFT_pseudo_pipe.py | 8 + .../Code/demo/rift/infra/multi_gpu/.gitignore | 5 + .../Code/demo/rift/infra/multi_gpu/Makefile | 142 ++++++++++++++++++ .../Code/demo/rift/infra/multi_gpu/README.md | 127 ++++++++++++++++ .../multi_gpu/blueprints/rift-multigpu.yaml | 82 ++++++++++ .../blueprints/rift_container_family.cit.yaml | 28 ++++ .../demo/rift/infra/multi_gpu/fake_ile.py | 50 ++++++ .../demo/rift/infra/multi_gpu/multigpu_ci.ini | 65 ++++++++ 11 files changed, 547 insertions(+), 6 deletions(-) create mode 100644 MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/.gitignore create mode 100644 MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile create mode 100644 MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md create mode 100644 MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml create mode 100644 MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift_container_family.cit.yaml create mode 100755 MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/fake_ile.py create mode 100644 MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/multigpu_ci.ini diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py index 4132535f5..290f36cce 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py @@ -145,15 +145,29 @@ def _partition(start,count,n): ''' -def ile_invocation_shell(exe): +def ile_gpu_fanout_value(): + """Raw RIFT_ILE_GPU_FANOUT string resolved at DAG-build time (default '1').""" + return os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() or '1' + + +def ile_invocation_shell(exe, fanout=None): """Return the shell snippet that invokes the ILE executable `exe` with multi-GPU fan-out (see RIFT_ILE_GPU_FANOUT). The launcher reads its source from stdin via a quoted here-doc, so the ILE arguments in "$@" (which include nested-quoted values) are passed through untouched. With fan-out disabled - the launcher simply exec()s the binary, so default behaviour is unchanged.""" + the launcher simply exec()s the binary, so default behaviour is unchanged. + + The build-time fan-out value is BAKED into ile_pre.sh as the runtime default + (still overridable by a runtime RIFT_ILE_GPU_FANOUT), so the job does NOT + depend on the submit/execute environment propagating the variable -- this is + what makes the feature work under asimov, whose blueprints can only set the + value at DAG-build time.""" + if fanout is None: + fanout = ile_gpu_fanout_value() return ( 'PY=python3; command -v python3 >/dev/null 2>&1 || PY=python\n' - 'exec "$PY" - "{exe}" "$@" <<\'RIFT_ILE_MULTIGPU_EOF\'\n'.format(exe=exe) + + 'export RIFT_ILE_GPU_FANOUT="${RIFT_ILE_GPU_FANOUT:-' + str(fanout) + '}"\n' + + 'exec "$PY" - "' + exe + '" "$@" <<\'RIFT_ILE_MULTIGPU_EOF\'\n' + ILE_MULTIGPU_LAUNCHER_PY + '\nRIFT_ILE_MULTIGPU_EOF\n' ) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py index 8756f1d2a..c28128199 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py @@ -273,14 +273,27 @@ def _partition(start,count,n): ''' -def ile_invocation_shell(exe): +def ile_gpu_fanout_value(): + """Raw RIFT_ILE_GPU_FANOUT string resolved at DAG-build time (default '1').""" + return os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() or '1' + + +def ile_invocation_shell(exe, fanout=None): """Shell snippet invoking the ILE executable `exe` with multi-GPU fan-out (RIFT_ILE_GPU_FANOUT). The launcher reads its source from stdin via a quoted here-doc, so the ILE arguments in "$@" (including nested-quoted values) pass - through untouched; with fan-out disabled it just exec()s the binary.""" + through untouched; with fan-out disabled it just exec()s the binary. + + The build-time fan-out value is BAKED into ile_pre.sh as the runtime default + (still overridable at runtime), so the job does not depend on the submit/ + execute environment propagating RIFT_ILE_GPU_FANOUT -- needed for asimov, + which can only set the value at DAG-build time.""" + if fanout is None: + fanout = ile_gpu_fanout_value() return ( 'PY=python3; command -v python3 >/dev/null 2>&1 || PY=python\n' - 'exec "$PY" - "{exe}" "$@" <<\'RIFT_ILE_MULTIGPU_EOF\'\n'.format(exe=exe) + + 'export RIFT_ILE_GPU_FANOUT="${RIFT_ILE_GPU_FANOUT:-' + str(fanout) + '}"\n' + + 'exec "$PY" - "' + exe + '" "$@" <<\'RIFT_ILE_MULTIGPU_EOF\'\n' + ILE_MULTIGPU_LAUNCHER_PY + '\nRIFT_ILE_MULTIGPU_EOF\n' ) diff --git a/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration b/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration index a5f2366f3..c5d99fcc7 100755 --- a/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration +++ b/MonteCarloMarginalizeCode/Code/bin/create_event_parameter_pipeline_BasicIteration @@ -279,6 +279,7 @@ parser.add_argument("--ile-batch",action='store_true',help="Different workflow: parser.add_argument("--transfer-file-list",default=None,help="File containing list of *input* filenames to transfer, one name per file. Copied into transfer_files for condor directly. If provided, also enables attempts to deduce files that need to be transferred for the pipeline to operate, as needed for OSG, etc") parser.add_argument("--request-memory-ILE",default=4096,type=int,help="Memory request for condor (in Mb) for ILE jobs.") parser.add_argument("--request-gpu-ILE",action='store_true',help="ILE will request a GPU. [Note: if you do this, your code will only run on GPU-enabled slots]") +parser.add_argument("--ile-gpu-fanout",default=None,help="Multi-GPU ILE fan-out: split each ILE batch's intrinsic-grid range across N GPUs (one shard per GPU). Integer N (also requests N GPUs+CPUs) or 'auto'. Baked into ile_pre.sh; equivalent to RIFT_ILE_GPU_FANOUT. Requires --request-gpu-ILE.") parser.add_argument("--request-xpu-ILE",action='store_true',help="ILE will *prefer* a gpu, but not require it") parser.add_argument("--request-memory-CIP",default=16384,type=int,help="Memory request for condor (in Mb) for fitting jobs.") parser.add_argument("--request-memory-CIP-flex",action='store_true',help="If true,memory request is flexible and will increment. Beware of possible runaway requests") @@ -307,6 +308,12 @@ parser.add_argument("--use-bw-psd",action='store_true',help="Use BW PSD, attempt parser.add_argument("--use-full-submit-paths",action='store_true',help="DAG created has full paths to submit files generated. Note this is implemented on a per-file/as-needed basis, mainly to facilitate using this dag as an external subdag") opts= parser.parse_args() +# Multi-GPU ILE fan-out: funnel --ile-gpu-fanout through RIFT_ILE_GPU_FANOUT, which +# dag_utils reads at write_ILE_sub_simple time to size request_GPUs/CPUs and bake the +# value into ile_pre.sh. CLI value wins over any inherited environment value. +if opts.ile_gpu_fanout is not None: + os.environ['RIFT_ILE_GPU_FANOUT'] = str(opts.ile_gpu_fanout) + local_worker_universe="vanilla" local_worker_universe_cip = "vanilla" diff --git a/MonteCarloMarginalizeCode/Code/bin/util_RIFT_pseudo_pipe.py b/MonteCarloMarginalizeCode/Code/bin/util_RIFT_pseudo_pipe.py index c196de4f3..003b42a2f 100755 --- a/MonteCarloMarginalizeCode/Code/bin/util_RIFT_pseudo_pipe.py +++ b/MonteCarloMarginalizeCode/Code/bin/util_RIFT_pseudo_pipe.py @@ -234,6 +234,7 @@ def unsafe_parse_arg_string_dict(my_argstr): parser.add_argument("--ile-no-gpu",action='store_true') parser.add_argument("--ile-xpu",action='store_true',help='Request ILE run on both GPU and CPU. Disables ile_force_gpu, if provided!') parser.add_argument("--ile-force-gpu",action='store_true') +parser.add_argument("--ile-gpu-fanout",default=None,help="Multi-GPU ILE fan-out: split each ILE batch's intrinsic-grid range across N GPUs on the node (one shard per GPU). Integer N (also requests N GPUs+CPUs) or 'auto' (split across whatever GPUs are visible at runtime). Baked into the generated ile_pre.sh, so it needs no runtime environment. Equivalent to setting RIFT_ILE_GPU_FANOUT. Requires --ile-force-gpu.") parser.add_argument("--fake-data-cache",type=str) parser.add_argument("--spin-magnitude-prior",default='default',type=str,help="options are default [uniform mag for precessing, zprior for aligned], volumetric, uniform_mag_prec, uniform_mag_aligned, zprior_aligned") parser.add_argument("--force-lambda-max",default=None,type=float,help="Provide this value to override the value of lambda-max provided") @@ -322,6 +323,13 @@ def unsafe_parse_arg_string_dict(my_argstr): parser.add_argument("--internal-mitigate-fd-J-frame",default="L_frame",help="L_frame|rotate, choose method to deal with ChooseFDWaveform being in wrong frame. Default is to request L frame for inputs") opts= parser.parse_args() +# Multi-GPU ILE fan-out: --ile-gpu-fanout funnels through RIFT_ILE_GPU_FANOUT, which +# create_event_parameter_pipeline_BasicIteration (run via os.system, inheriting this +# environment) and dag_utils read at DAG-build time to size request_GPUs/CPUs and bake +# the value into ile_pre.sh. A CLI value wins over any inherited environment value. +if opts.ile_gpu_fanout is not None: + os.environ['RIFT_ILE_GPU_FANOUT'] = str(opts.ile_gpu_fanout) + config_stored=None; config_dict=None ile_condor_commands = None if (opts.use_ini): diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/.gitignore b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/.gitignore new file mode 100644 index 000000000..398582b84 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/.gitignore @@ -0,0 +1,5 @@ +# Generated by `make build` / `make smoke-local` -- regenerable, not committed. +rundir_multigpu/ +_smoke/ +ci_coinc.xml +proposed-grid* diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile new file mode 100644 index 000000000..64eabbd00 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile @@ -0,0 +1,142 @@ +# ============================================================================ +# Multi-GPU ILE fan-out demo +# +# RIFT ILE "batchmode" evaluates a contiguous block of intrinsic grid points +# (--event .. --event+--n-events-to-analyze) SERIALLY on ONE GPU. On a node +# with several GPUs reserved, the rest sit idle. The fan-out splits that block +# into N shards run concurrently -- one per GPU -- via the generated ile_pre.sh. +# +# Turn it on with RIFT_ILE_GPU_FANOUT=N (env) or --ile-gpu-fanout N (CLI) or, for +# asimov, a blueprint key (see README.md + blueprints/). The value is BAKED into +# ile_pre.sh and sets request_GPUs=N/request_CPUs=N, so nothing depends on the +# submit/execute environment -- which is what makes it work under asimov. +# +# Targets: +# make smoke-local # PROVE the split+pin logic on THIS node's GPUs (no cupy, +# # no condor, no data). Runs the real shipped launcher +# # around a stub ILE. This is the runnable-anywhere proof. +# make build # build a real pipeline run dir on the CI data with +# # --ile-gpu-fanout (needs SINGULARITY_RIFT_IMAGE). +# make verify # assert the generated ILE.sub + ile_pre.sh carry the fan-out. +# make inspect # show the generated launcher + sub requests. +# make clean +# +# See README.md for the asimov path and host/GPU matching. +# ============================================================================ + +RIFT_CODE_ROOT := $(abspath ../../../..) +REPO_ROOT := $(abspath ../../../../../..) +CI_DEMO := $(REPO_ROOT)/.travis/ILE-GPU-Paper/demos + +ENV := PYTHONPATH=$(RIFT_CODE_ROOT):$${PYTHONPATH:-} PATH=$(RIFT_CODE_ROOT)/bin:$${PATH} + +CACHE ?= $(CI_DEMO)/zero_noise.cache +PSD ?= $(CI_DEMO)/HLV-ILIGO_PSD.xml.gz +SIM_XML ?= $(CI_DEMO)/mdc.xml.gz +FRAMES ?= $(CI_DEMO)/zero_noise_mdc + +# How many GPUs to fan out across. The smoke test caps this at the number of +# GPUs actually visible; the pipeline build bakes exactly this into ile_pre.sh +# and requests this many GPUs+CPUs. +FANOUT ?= 4 + +BUILD_DIR := $(CURDIR)/rundir_multigpu +DAG := marginalize_intrinsic_parameters_BasicIterationWorkflow.dag + +.PHONY: all smoke-local build verify inspect clean coinc help + +help: + @sed -n '2,40p' Makefile + +# --------------------------------------------------------------------------- +# smoke-local: the genuinely-runnable-anywhere proof. +# +# Builds a REAL ile_pre.sh from the shipped helper (RIFT.misc.dag_utils_generic. +# ile_invocation_shell), wrapping the fake_ile.py stub, then runs it with the +# fan-out enabled. Asserts: every grid point covered exactly once, spread across +# the GPUs, distinct per-shard output prefixes (no clobber). No cupy, no GPU +# compute -- this isolates the only new logic (partition + CUDA_VISIBLE_DEVICES +# pinning). Set DEVICES=0,1,2,3 to force a device list on a shared node. +# --------------------------------------------------------------------------- +DEVICES ?= +smoke-local: + @rm -rf _smoke && mkdir _smoke + @# Generate ile_pre.sh exactly as the pipeline would, baking FANOUT in. + @cd _smoke && $(ENV) RIFT_ILE_GPU_FANOUT=$(FANOUT) python3 -c \ + "import os; from RIFT.misc.dag_utils_generic import ile_invocation_shell; \ + open('ile_pre.sh','w').write('#! /bin/bash -xe\n'+ile_invocation_shell(os.path.abspath('../fake_ile.py'))); \ + os.chmod('ile_pre.sh',0o755)" + @echo "=== generated ile_pre.sh (launcher head) ===" + @grep -nE 'RIFT_ILE_GPU_FANOUT|exec ' _smoke/ile_pre.sh | head -3 + @echo "=== running: 100-point block, fan-out=$(FANOUT)$(if $(DEVICES), on devices $(DEVICES)) ===" + @cd _smoke && $(if $(DEVICES),CUDA_VISIBLE_DEVICES=$(DEVICES) ,)./ile_pre.sh \ + --n-events-to-analyze 100 --output-file=CME_out-0-1-0.xml --event=0 \ + --some-other-arg foo --internal-waveform-extra-kwargs '{a:1,b:2}' 2>shards.log; echo " launcher exit=$$?" + @grep rift_ile_multigpu _smoke/shards.log | sed 's/^/ /' + @cd _smoke && python3 -c "import glob; \ + rows=[l.split() for f in glob.glob('CME*.dat') for l in open(f)]; \ + g=sorted(int(r[0]) for r in rows); gpus=sorted(set(r[1] for r in rows)); \ + prefixes=sorted(set(f.rsplit('_',2)[0] for f in glob.glob('CME*.dat'))); \ + assert g==list(range(100)), 'COVERAGE FAIL: '+str(g); \ + assert len(g)==len(set(g)), 'DUPLICATE points'; \ + print(' PASS: 100/100 grid points, no dups, range %d..%d'%(g[0],g[-1])); \ + print(' PASS: spread across GPUs', gpus); \ + print(' PASS: %d distinct shard output prefixes (no clobber):'%len(prefixes)); \ + [print(' '+p) for p in prefixes]" + +# --------------------------------------------------------------------------- +# build: a real pipeline run directory on the CI synthetic data, with fan-out. +# +# Uses singularity/OSG mode so ile_pre.sh is generated (the fan-out launcher +# lives there). Needs the container-family manifest: +# export SINGULARITY_RIFT_IMAGE=~/rift_cit_build_container_family/built_containers/rift_container_family.cit.yaml +# export SINGULARITY_BASE_EXE_DIR=/usr/local/bin/ +# (or pass SINGULARITY_RIFT_IMAGE=... on the make line). This only BUILDS the +# DAG; it does not submit. +# --------------------------------------------------------------------------- +coinc: ci_coinc.xml +ci_coinc.xml: + $(ENV) util_SimInspiralToCoinc.py --sim-xml $(SIM_XML) --event 0 \ + --ifo H1 --ifo L1 --ifo V1 --output $(CURDIR)/ci_coinc.xml --injected-snr 17.5 + +build: ci_coinc.xml + @test -n "$(SINGULARITY_RIFT_IMAGE)" || (echo "ERROR: export SINGULARITY_RIFT_IMAGE= first (see README.md)"; false) + rm -rf $(BUILD_DIR) + $(ENV) util_RIFT_pseudo_pipe.py \ + --use-ini $(CURDIR)/multigpu_ci.ini --use-coinc $(CURDIR)/ci_coinc.xml --use-rundir $(BUILD_DIR) \ + --fake-data-cache $(CACHE) \ + --assume-nospin --approx IMRPhenomD --ile-sampler-method AV \ + --internal-force-iterations 2 --ile-n-eff 10 \ + --ile-force-gpu --ile-gpu-fanout $(FANOUT) \ + --use-osg --use-osg-file-transfer --internal-truncate-files-for-osg-file-transfer \ + --internal-ile-request-disk 4G + @# OSG file-transfer mode wants frames + per-IFO PSDs physically present in the rundir. + @# (In OSG mode the PSD is supplied as run-dir -psd.xml.gz, copied below, not via a flag.) + rm -rf $(BUILD_DIR)/frames_dir; cp -r $(FRAMES) $(BUILD_DIR)/frames_dir + for ifo in H1 L1 V1; do cp $(PSD) $(BUILD_DIR)/$$ifo-psd.xml.gz; done + @echo + @echo "Built $(BUILD_DIR). Now: make verify" + +# --------------------------------------------------------------------------- +# verify: assert the fan-out actually landed in the generated submit files. +# --------------------------------------------------------------------------- +verify: + @test -f $(BUILD_DIR)/ILE.sub || (echo "no $(BUILD_DIR)/ILE.sub -- run 'make build' first"; false) + @echo "=== ILE.sub GPU/CPU requests (expect $(FANOUT)) ===" + @grep -iE 'request_GPUs|request_CPUs' $(BUILD_DIR)/ILE.sub + @grep -qiE 'request_GPUs *= *$(FANOUT)' $(BUILD_DIR)/ILE.sub || (echo "FAIL: request_GPUs != $(FANOUT)"; false) + @grep -qiE 'request_CPUs *= *$(FANOUT)' $(BUILD_DIR)/ILE.sub || (echo "FAIL: request_CPUs != $(FANOUT)"; false) + @echo "=== ile_pre.sh baked fan-out + launcher ===" + @grep -nE 'RIFT_ILE_GPU_FANOUT|rift_ile_multigpu' $(BUILD_DIR)/ile_pre.sh | head -3 + @grep -qE 'RIFT_ILE_GPU_FANOUT:-$(FANOUT)' $(BUILD_DIR)/ile_pre.sh || (echo "FAIL: ile_pre.sh did not bake fan-out=$(FANOUT)"; false) + @echo + @echo "OK: ILE.sub requests $(FANOUT) GPUs + $(FANOUT) CPUs; ile_pre.sh bakes RIFT_ILE_GPU_FANOUT=$(FANOUT)" + @echo " and will split each 100-point ILE block into $(FANOUT) shards, one per GPU." + +inspect: + @echo "######## $(BUILD_DIR)/ile_pre.sh ########"; cat $(BUILD_DIR)/ile_pre.sh + @echo; echo "######## ILE.sub (resource + container lines) ########" + @grep -iE 'request_|require_gpus|SingularityImage|executable|when_to_transfer' $(BUILD_DIR)/ILE.sub + +clean: + rm -rf _smoke $(BUILD_DIR) ci_coinc.xml diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md new file mode 100644 index 000000000..fa6642142 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md @@ -0,0 +1,127 @@ +# Multi-GPU ILE fan-out + +**What it does.** A single RIFT ILE "batchmode" job evaluates a contiguous block +of intrinsic grid points — `[--event, --event + --n-events-to-analyze)` — one at +a time on **one** GPU. (In production that block is large: the distexport runs use +`jobs per worker: 100`, i.e. 100 points per ILE job.) On a node where several GPUs +are reserved, the other GPUs sit idle. + +The fan-out splits that block into **N disjoint shards run concurrently, one per +GPU**. Each shard is pinned with `CUDA_VISIBLE_DEVICES`, gets a disjoint +sub-range, and writes to a distinct `--output-file` prefix (`.gpu`) so +the per-point output files never collide. Downstream collection is unchanged: +`util_ILEdagPostprocess.sh` globs `CME*.dat` and `util_CleanILE.py` de-duplicates +by *parameter value*, not filename. Coverage is identical to the serial run (the +shards partition the range exactly), and the launcher returns the first non-zero +shard exit code, so condor retry/hold behaviour is preserved. + +It is **off by default** and a no-op unless you ask for it. With it disabled the +generated `ile_pre.sh` just `exec`s the ILE binary exactly as before. + +--- + +## How it is wired (and why it works under asimov) + +The fan-out lives entirely in **generated submit files**, decided at **DAG-build +time**: + +- `ile_pre.sh` wraps the ILE binary in a tiny launcher (`RIFT.misc.dag_utils*. + ile_invocation_shell`). The chosen fan-out value is **baked into `ile_pre.sh`** + as `export RIFT_ILE_GPU_FANOUT="${RIFT_ILE_GPU_FANOUT:-N}"`, so the running job + needs **no environment variable** — critical because asimov submits in a clean + shell. (A runtime `RIFT_ILE_GPU_FANOUT` still overrides the baked default.) +- The ILE `.sub` gets `request_GPUs=N` and `request_CPUs=N`, so HTCondor only + matches a slot/node that actually has N GPUs, and gives you N CPUs to drive them. + +> **Requires the `ile_pre.sh` path.** The launcher is the `ile_pre.sh` the pipeline +> generates for frames-based / singularity / OSG-file-transfer runs (the standard +> production configuration). A bare `--fake-data-cache` run with no frames does not +> emit `ile_pre.sh`, so there is nothing to fan out there. + +--- + +## Three ways to turn it on + +All three resolve to the same baked `RIFT_ILE_GPU_FANOUT`. + +| Context | How | +|---|---| +| **Env var** (interactive) | `export RIFT_ILE_GPU_FANOUT=4` before `util_RIFT_pseudo_pipe.py` | +| **CLI flag** (direct) | `util_RIFT_pseudo_pipe.py ... --ile-force-gpu --ile-gpu-fanout 4` (also on `create_event_parameter_pipeline_BasicIteration`) | +| **asimov blueprint** | `scheduler.environment variables: {RIFT_ILE_GPU_FANOUT: 4}` **or** `scheduler.pipeline: {ile-gpu-fanout: 4}` | + +Values: an integer `N` (also sizes `request_GPUs`/`request_CPUs`), or `auto` (split +across whatever GPUs are visible at runtime — for a whole node held with +`request_GPUs=1`; `auto` cannot size the request, so reserve the node yourself). + +--- + +## Demonstrations in this directory + +### 1. `make smoke-local` — proof the split + pin logic works (runs anywhere) + +Builds a **real** `ile_pre.sh` from the shipped helper, wrapping a stub ILE +(`fake_ile.py`), and runs it across this node's GPUs. No cupy, no condor, no data — +it isolates the only new logic. Asserts every grid point is covered exactly once, +spread across the GPUs, with distinct per-shard output prefixes: + +``` +make smoke-local # uses nvidia-smi to find GPUs +make smoke-local FANOUT=2 # split across 2 +make smoke-local DEVICES=0,1,2,3 # force a device list (shared node) +``` + +### 2. `make build` + `make verify` — a real pipeline run dir + +Builds a pipeline on the CI synthetic data (same data as `demo/rift/calmarg`), +in singularity/OSG mode so `ile_pre.sh` is generated, with the fan-out baked in. +Needs the container-family manifest: + +``` +export SINGULARITY_RIFT_IMAGE=$(pwd)/blueprints/rift_container_family.cit.yaml +export SINGULARITY_BASE_EXE_DIR=/usr/local/bin/ +make build FANOUT=4 +make verify # asserts ILE.sub has request_GPUs=4/request_CPUs=4 + # and ile_pre.sh bakes RIFT_ILE_GPU_FANOUT=4 +make inspect # show the generated launcher + sub resource lines +``` + +`make build` only builds the DAG (it does not submit). To actually run it you need +a pool with ≥ N-GPU nodes; submit with `condor_submit_dag` from the run dir. + +### 3. `blueprints/` — the asimov path + +- `rift-multigpu.yaml` — analysis blueprint showing **both** blueprint encodings + (environment-variable and pipeline-CLI) plus host/GPU matching. +- `rift_container_family.cit.yaml` — frozen container-family pin (per-machine image + selection + GPU capability floor). + +Apply with `asimov apply -f blueprints/rift-multigpu.yaml` (after the matching +event blueprint). The RIFT asimov pipeline (`RIFT/asimov/rift.py`) copies +`environment variables` into the build environment / turns `pipeline` keys into +`util_RIFT_pseudo_pipe.py` flags, so the value reaches the DAG build with **no +assumption about the submit shell**. + +--- + +## Host / GPU matching + +`request_GPUs=N` is the primary matcher: HTCondor will only place the job where N +GPUs are available. Layer on a capability floor and host pins as needed: + +- `scheduler.gpu architectures:` → `RIFT_REQUIRE_GPUS` device exclusions (drop slow + cards), and `RIFT_REQUIRE_GPUS='(Capability >= 8.0)'`-style floors via the env. +- `scheduler.avoid hosts:` → `RIFT_AVOID_HOSTS` (blacklist single-GPU or bad nodes). + +**Fan-out targets local / dedicated multi-GPU pools.** OSG glide-in slots typically +expose one GPU each, so `request_GPUs=4` will not match there — use your local pool +(`osg: False`, or a site requirement) for fan-out runs. + +--- + +## Sizing + +- `jobs per worker` (= `--ile-n-events-to-analyze`, the per-job block) should be a + comfortable multiple of the fan-out, e.g. 100 points / 4 GPUs = 25 points/GPU. +- GPU memory is per shard on its own device, so N shards on N distinct GPUs do not + contend; an A100 (80 GB) runs one ILE shard with room to spare. diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml new file mode 100644 index 000000000..5cea6e325 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml @@ -0,0 +1,82 @@ +# =========================================================================== +# Asimov RIFT analysis blueprint -- multi-GPU ILE fan-out +# +# asimov apply -f blueprints/rift-multigpu.yaml +# +# THE POINT: asimov runs the pipeline in a clean environment, so we CANNOT rely +# on RIFT_ILE_GPU_FANOUT being exported in the submit shell. The value must be +# ENCODED in the blueprint. There are two equivalent ways; this file shows the +# primary one and documents the alternative. Either way, the RIFT pipeline +# (RIFT/asimov/rift.py) turns it into the build-time RIFT_ILE_GPU_FANOUT, which +# dag_utils bakes into ile_pre.sh and uses to set request_GPUs=N/request_CPUs=N. +# Nothing depends on the runtime environment after the DAG is built. +# +# This is a TEMPLATE; the event/data keys mirror the real distexport blueprints. +# =========================================================================== +kind: analysis +name: rift-multigpu +event: S-DEMO # replace with your event; pair with an event blueprint +pipeline: rift +status: ready +comment: Multi-GPU ILE fan-out demo - split each ILE batch across N GPUs per node. +bootstrap: manual + +waveform: + approximant: IMRPhenomD + reference frequency: 20 + maximum mode: 2 + +scheduler: + accounting group: ligo.dev.o4.cbc.pe.rift + + # --- Container family (per-machine image selection + GPU capability floor). --- + singularity image: 'blueprints/rift_container_family.cit.yaml' + singularity base exe directory: '/usr/local/bin/' + + # ======================================================================= + # (1) PRIMARY: encode the fan-out as an environment variable. rift.py copies + # every key here into os.environ BEFORE invoking util_RIFT_pseudo_pipe.py + # (rift.py: "adding environment variables before starting pipeline"), so it + # reaches the DAG build with NO assumption about the submit shell. + # ======================================================================= + environment variables: + RIFT_ILE_GPU_FANOUT: 4 # split each ILE block across 4 GPUs (one shard/GPU) + + # ======================================================================= + # (2) ALTERNATIVE (equivalent): pass it as a pseudo_pipe CLI flag. Every key + # under `pipeline:` becomes --key=value on the util_RIFT_pseudo_pipe.py line + # (rift.py builds `--{key}={value}`). Uncomment INSTEAD of (1) if you + # prefer the explicit flag. A CLI value wins over the environment one. + # + # pipeline: + # ile-gpu-fanout: 4 + # ======================================================================= + + # --- Host / GPU matching: land on nodes that actually HAVE >= N GPUs. --- + # request_GPUs=N (set automatically from the fan-out) already forces HTCondor to + # match only slots/nodes offering N GPUs. Add a capability floor so the shards + # land on fast cards, and (for a dedicated local multi-GPU pool) pin/avoid hosts. + # NOTE: multi-GPU fan-out targets LOCAL / dedicated multi-GPU nodes. OSG glide-in + # slots typically expose ONE GPU each, so request_GPUs=4 will not match there -- + # keep `osg: False` (or target your local pool) for fan-out runs. + osg: False + gpu architectures: # -> RIFT_REQUIRE_GPUS device exclusions, e.g. drop slow cards + - 'Tesla T4' + # avoid hosts: + # - some-single-gpu-host.example.edu + +sampler: + force iterations: 2 + cip: + fitting method: rf + sampling method: AV + ile: + n eff: 10 + sampling method: AV + # macrongroup: points evaluated per ILE job == what the fan-out splits. + # Keep a comfortable multiple of the fan-out (here 100 / 4 = 25 points/GPU). + jobs per worker: 100 + runtime max minutes: 120 + # NOTE: the RIFT asimov pipeline ALWAYS passes --ile-force-gpu, so the GPU + # code path the fan-out needs is on by default -- no extra key required here. + n output samples: 2000 diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift_container_family.cit.yaml b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift_container_family.cit.yaml new file mode 100644 index 000000000..db38e94f7 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift_container_family.cit.yaml @@ -0,0 +1,28 @@ +# RIFT CIT container family manifest -- AUTO-GENERATED by build_cit_family.sh +# Build date: 20260615b Branch: rift_O4d_fix_rvs_clear_fairdraw_batch +# +# Point SINGULARITY_RIFT_IMAGE at this file to deploy the family. The +# pipeline turns it into a per-machine MY.SingularityImage selection, a +# selective osdf $$() transfer, and a require_gpus capability floor. +# +# Images are referenced at their OSDF staging location; COPY the built +# .sif files there first (this script does NOT copy them): +# /osdf/igwn/cit/staging/richard.oshaughnessy/rift_containers/ +# +# A single SINGULARITY_BASE_EXE_DIR applies to EVERY image; both are built +# from one template so their internal layout is identical -- do not mix in +# foreign images. +version: 1 +capability_attr: GPUs_Capability +fallback: cc60-90 +containers: + - label: cc60-90 + image: osdf:///igwn/cit/staging/richard.oshaughnessy/rift_containers/rift_o4d-calmarg_in_loop_cc60-90_cuda118_20260615b.sif + cuda_capability_min: 6.0 + cuda_capability_max: 9.0 + note: "base=nvidia/cuda:11.8.0-runtime-ubuntu22.04, cupy-cuda11x, branch=rift_O4d_fix_rvs_clear_fairdraw_batch, built=20260615b" + - label: cc90-120 + image: osdf:///igwn/cit/staging/richard.oshaughnessy/rift_containers/rift_o4d-calmarg_in_loop_cc90-120_cuda128_20260615b.sif + cuda_capability_min: 9.0 + cuda_capability_max: 12.0 + note: "base=nvidia/cuda:12.8.0-devel-ubuntu22.04, cupy-cuda12x, branch=rift_O4d_fix_rvs_clear_fairdraw_batch, built=20260615b" diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/fake_ile.py b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/fake_ile.py new file mode 100755 index 000000000..2a4a07167 --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/fake_ile.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +"""Stand-in for integrate_likelihood_extrinsic_batchmode, for the multi-GPU +smoke test (`make smoke-local`). It mimics ONLY the two behaviours the fan-out +launcher relies on: + + 1. grid slicing: evaluates grid indices [--event, --event+--n-events-to-analyze) + 2. output naming: writes <--output-file>__.dat (local 0..n-1), + exactly like the real ILE (fname = output_file+"_"+str(indx)+"_.dat"). + +Each row records the GLOBAL grid index and the GPU the shard ran on +(CUDA_VISIBLE_DEVICES), so the test can prove the whole grid was covered exactly +once and spread across the GPUs. No cupy / no real GPU work -- this validates +the launcher's partition + pinning, which is the only new logic. +""" +import os +import sys + +GRID_N = int(os.environ.get("FAKE_ILE_GRID_N", "100")) # pretend overlap-grid size + + +def getopt(argv, names, default=None): + for i, a in enumerate(argv): + for nm in names: + if a == nm and i + 1 < len(argv): + return argv[i + 1] + if a.startswith(nm + "="): + return a.split("=", 1)[1] + return default + + +def main(): + argv = sys.argv[1:] + event = int(getopt(argv, ["--event", "-E"], "0")) + ngroup = int(getopt(argv, ["--n-events-to-analyze"], "1")) + outfile = getopt(argv, ["--output-file", "-o"], None) + gpu = os.environ.get("CUDA_VISIBLE_DEVICES", "") + if outfile is None: + sys.stderr.write("fake_ile: no --output-file\n") + return 2 + n_event_max = min(GRID_N, event + ngroup) + for local, gidx in enumerate(range(event, n_event_max)): + with open("{}_{}_.dat".format(outfile, local), "w") as f: + f.write("{} {} {}\n".format(gidx, gpu, -1.0)) # global_idx gpu (pretend lnL) + sys.stderr.write("[fake_ile] event={} ngroup={} gpu={} wrote {} pts prefix={}\n".format( + event, ngroup, gpu, n_event_max - event, outfile)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/multigpu_ci.ini b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/multigpu_ci.ini new file mode 100644 index 000000000..74e9ba0fb --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/multigpu_ci.ini @@ -0,0 +1,65 @@ +# CI-matched RIFT ini for the multi-GPU fan-out demo (`make build`). +# Matches the zero-noise synthetic CI data in .travis/ILE-GPU-Paper/demos: 3 IFOs +# (H1,L1,V1) with FAKE-STRAIN channels, srate 4096, seglen 8, fmin 10, zero spin +# (IMRPhenomD-compatible), mc in [23,35], event 1000000014.236. This is the SAME +# data the calmarg demo uses; here we strip calmarg and instead exercise the +# multi-GPU ILE fan-out. CLI args on the util_RIFT_pseudo_pipe.py line win, so +# keep [rift-pseudo-pipe] minimal. + +[analysis] +ifos=['H1','L1','V1'] +singularity=False +osg=False + +[paths] + +[input] +max-psd-length=10000 + +[condor] +accounting_group=ligo.sim.o4.cbc.pe.rift +accounting_group_user=richard.oshaughnessy + +[datafind] +url-type=file +types = {'H1': 'fake_strain', 'L1': 'fake_strain', 'V1': 'fake_strain'} + +[data] +channels = {'H1': 'H1:FAKE-STRAIN','L1': 'L1:FAKE-STRAIN', 'V1': 'V1:FAKE-STRAIN'} + +[lalinference] +flow = {'H1': 10, 'L1': 10, 'V1': 10} +fhigh = { 'H1': 1700, 'L1': 1700, 'V1': 1700 } + +[engine] +fref=20 +amporder = -1 +seglen = 8 +srate = 4096 +a_spin1-max = 0.0 +a_spin2-max = 0.0 +chirpmass-min = 23.0 +chirpmass-max = 35.0 +comp-min = 1 +comp-max = 1000 +distance-max = 1000 +aligned-spin = +alignedspin-zprior = + +[rift-pseudo-pipe] +cip-fit-method="rf" +ile-n-eff=10 +l-max=2 +internal-distance-max=1000 +ile-runtime-max-minutes=120 +# Many intrinsic points per ILE job (= macrongroup). This is what the fan-out +# splits across the GPUs: with 100 points/job and --ile-gpu-fanout 4, each GPU +# evaluates ~25 points concurrently. Keep it a comfortable multiple of the +# fan-out so the split is even. (Production distexport runs use 100.) +ile-jobs-per-worker=100 +internal-propose-converge-last-stage=True +force-eta-range="[0.20,0.24999]" +fmin-template=10 +event-time=1000000014.236547946 +n-output-samples=2000 +use-online-psd=False From ff54575a0299b3bb486cb1523bcb013239df94e2 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Sat, 27 Jun 2026 04:08:18 -0700 Subject: [PATCH 10/63] ILE multi-GPU fan-out: adaptive 1..N requests + use-all-physical mode Add the two ways to request a VARIABLE number of GPUs (HTCondor's plain request_GPUs is a single fixed count, so it cannot natively ask for "1 to N"): - RIFT_ILE_GPU_FANOUT=auto-max-N (shared / partitionable-slot pools) request_GPUs/request_CPUs become a ClassAd expression that asks for up to N of the capability-matching GPUs available on the matched slot: ifThenElse(countMatches(RequireGPUs,AvailableGPUs) >= N, N, ifThenElse(... >= 1, ..., 1)) (same countMatches idiom RIFT already uses for cross-platform GPU matching). ile_pre.sh bakes 'auto', so the launcher splits across exactly the 1..N GPUs condor grants. Override the expression with RIFT_ILE_GPU_REQUEST_EXPR if your pool exposes GPU counts under a different attribute. - RIFT_ILE_GPU_FANOUT=all (dedicated / whole nodes you reserve) keep request_GPUs=1 (matches a node with ANY GPU count) and have the launcher enumerate EVERY physical GPU via nvidia-smi, ignoring CUDA_VISIBLE_DEVICES. Launcher _devices(physical=True) drives this. The runtime split was already adaptive (the launcher splits the point block across however many GPUs it is handed); these add the matching request side. Implementation: ile_gpu_fanout_count() -> ile_gpu_request() returning (request_gpus, request_cpus) as an int OR a ClassAd expression; ile_gpu_fanout_value() maps auto-max-N -> baked 'auto'. Mirrored in dag_utils.py and dag_utils_generic.py. Verified: launcher splits across 1/2/3/4 granted GPUs (full coverage each); 'all' uses all 4 physical even with CUDA_VISIBLE_DEVICES=0; generated ILE.sub carries the adaptive expression for auto-max-4, request 1 for 'all', fixed N for N. Demo: new `make requests` shows request_GPUs/CPUs + baked launcher for each mode; README "Values -- fixed vs. adaptive" documents the hot-swap options, the partitionable-slot requirement, and the cgroup/reservation caveats; blueprint shows the adaptive variants. Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/dag_utils.py | 103 ++++++++++++------ .../Code/RIFT/misc/dag_utils_generic.py | 102 +++++++++++------ .../Code/demo/rift/infra/multi_gpu/Makefile | 44 ++++++-- .../Code/demo/rift/infra/multi_gpu/README.md | 44 +++++++- .../multi_gpu/blueprints/rift-multigpu.yaml | 6 + 5 files changed, 224 insertions(+), 75 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py index 290f36cce..0d04c5af4 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py @@ -97,14 +97,15 @@ def _setopt(argv, names, new): out[i]="{}={}".format(nm,new); hit=True; i+=1; break if not hit: i+=1 return out -def _devices(): - cvd=os.environ.get("CUDA_VISIBLE_DEVICES") - if cvd and cvd.strip(): - return [d.strip() for d in cvd.split(",") if d.strip()] +def _devices(physical=False): + if not physical: + cvd=os.environ.get("CUDA_VISIBLE_DEVICES") + if cvd and cvd.strip(): + return [d.strip() for d in cvd.split(",") if d.strip()] try: out=subprocess.check_output(["nvidia-smi","-L"]).decode() - n=len([l for l in out.splitlines() if l.strip().startswith("GPU ")]) - return [str(i) for i in range(n)] or ["0"] + m=len([l for l in out.splitlines() if l.strip().startswith("GPU ")]) + return [str(i) for i in range(m)] or ["0"] except Exception: return ["0"] def _partition(start,count,n): @@ -119,10 +120,14 @@ def _partition(start,count,n): outfile=_val(ile,["--output-file","-o"]) event=int(event) if event is not None else 0 ngroup=int(ngroup) if ngroup is not None else 1 -devs=_devices() -if fan in ("","0","1"): n=1 -elif fan=="auto": n=len(devs) +if fan=="all": + devs=_devices(physical=True); n=len(devs) # ignore CVD: use every physical GPU (reserved node) +elif fan=="auto": + devs=_devices(); n=len(devs) # split across exactly what condor granted +elif fan in ("","0","1"): + devs=_devices(); n=1 else: + devs=_devices() try: n=int(fan) except ValueError: n=1 n=max(1,min(n,len(devs),ngroup)) @@ -146,8 +151,52 @@ def _partition(start,count,n): def ile_gpu_fanout_value(): - """Raw RIFT_ILE_GPU_FANOUT string resolved at DAG-build time (default '1').""" - return os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() or '1' + """Launcher directive baked into ile_pre.sh, resolved from RIFT_ILE_GPU_FANOUT + at DAG-build time. Recognised values (see ile_gpu_request for the matching + condor request): + '' / '0' / '1' -> no fan-out + N (int) -> split the granted GPUs into <=N shards + 'auto' -> split across exactly the GPUs condor granted (CUDA_VISIBLE_DEVICES) + 'all' -> split across EVERY physical GPU (ignore CVD; reserved/whole node) + 'auto-max-N' -> adaptive: condor grants 1..N GPUs; launcher splits across the + granted set, so the baked launcher directive is just 'auto'.""" + raw = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() or '1' + if raw.lower().startswith('auto-max-'): + return 'auto' + return raw + + +def ile_gpu_request(): + """Resolve RIFT_ILE_GPU_FANOUT into the condor (request_gpus, request_cpus) + values for an ILE job. Each is an int (fixed count) OR a string ClassAd + expression (adaptive). Returns (1, 1) when fan-out is off, or for 'auto'/'all' + (those keep request_GPUs=1 and obtain their GPUs at runtime from a node you + have reserved -- the request cannot size them ahead of time). + + 'auto-max-N' emits an expression that requests up to N of the capability-matching + GPUs actually available on the matched (partitionable) slot, so ONE job flavour + lands on a 1/2/3/.../N-GPU node and grabs them all. Override the expression with + RIFT_ILE_GPU_REQUEST_EXPR if your pool exposes GPU counts differently + (verify the attribute with `condor_status -long `).""" + raw = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() + low = raw.lower() + if low in ('', '0', '1', 'auto', 'all'): + return (1, 1) + if low.startswith('auto-max-'): + try: + cap = max(1, int(low.rsplit('-', 1)[1])) + except ValueError: + return (1, 1) + expr = os.environ.get('RIFT_ILE_GPU_REQUEST_EXPR') + if not expr: + cm = "countMatches(RequireGPUs, AvailableGPUs)" + expr = "ifThenElse({cm} >= {N}, {N}, ifThenElse({cm} >= 1, {cm}, 1))".format(cm=cm, N=cap) + return (expr, expr) + try: + n = max(1, int(low)) + return (n, n) + except ValueError: + return (1, 1) def ile_invocation_shell(exe, fanout=None): @@ -173,20 +222,6 @@ def ile_invocation_shell(exe, fanout=None): ) -def ile_gpu_fanout_count(): - """Concrete number of GPUs an ILE job should *request* for fan-out, parsed - from RIFT_ILE_GPU_FANOUT. Returns an int >=1. 'auto' (split across whatever - is visible at runtime) cannot size a request ahead of time, so it returns 1 - and relies on the user reserving the node.""" - fan = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip().lower() - if fan in ('', '0', '1', 'auto'): - return 1 - try: - return max(1, int(fan)) - except ValueError: - return 1 - - def mkdir(dir_name): try : os.mkdir(dir_name) @@ -1122,15 +1157,17 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, nGPUs = 'countMatches(RequireGPUs, AvailableGPUs) >= 1 ? 1 : 0' ile_job.add_condor_cmd('rank', 'RequestGPUs') else: - # Multi-GPU fan-out (RIFT_ILE_GPU_FANOUT=N): reserve N GPUs (and N - # CPUs to drive them) so HTCondor hands this job the whole-node GPUs - # that ile_pre.sh then splits the intrinsic-grid range across. - fanout = ile_gpu_fanout_count() - if fanout > 1: - nGPUs = fanout - ile_gpu_cpus = fanout + # Multi-GPU fan-out (RIFT_ILE_GPU_FANOUT): request the GPUs (+matching CPUs) + # that ile_pre.sh then splits the intrinsic-grid range across. req_g/req_c + # are an int (fixed N) or a ClassAd expression ('auto-max-N' -> request up to + # N of the GPUs available on the matched slot). 'auto'/'all' keep 1 here and + # grab their GPUs at runtime from a node you have reserved. + req_g, req_c = ile_gpu_request() + if req_g != 1: + nGPUs = req_g + ile_gpu_cpus = req_c if not use_singularity: - ile_job.add_condor_cmd('request_CPUs', str(fanout)) + ile_job.add_condor_cmd('request_CPUs', str(req_c)) ile_job.add_condor_cmd('request_GPUs', str(nGPUs)) # Claim we don't need to make this request anymore to avoid out-of-memory errors. Also, no longer in 'requirements' # requirements.append("CUDAGlobalMemoryMb >= 2048") diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py index c28128199..d2ec44ad8 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py @@ -225,14 +225,15 @@ def _setopt(argv, names, new): out[i]="{}={}".format(nm,new); hit=True; i+=1; break if not hit: i+=1 return out -def _devices(): - cvd=os.environ.get("CUDA_VISIBLE_DEVICES") - if cvd and cvd.strip(): - return [d.strip() for d in cvd.split(",") if d.strip()] +def _devices(physical=False): + if not physical: + cvd=os.environ.get("CUDA_VISIBLE_DEVICES") + if cvd and cvd.strip(): + return [d.strip() for d in cvd.split(",") if d.strip()] try: out=subprocess.check_output(["nvidia-smi","-L"]).decode() - n=len([l for l in out.splitlines() if l.strip().startswith("GPU ")]) - return [str(i) for i in range(n)] or ["0"] + m=len([l for l in out.splitlines() if l.strip().startswith("GPU ")]) + return [str(i) for i in range(m)] or ["0"] except Exception: return ["0"] def _partition(start,count,n): @@ -247,10 +248,14 @@ def _partition(start,count,n): outfile=_val(ile,["--output-file","-o"]) event=int(event) if event is not None else 0 ngroup=int(ngroup) if ngroup is not None else 1 -devs=_devices() -if fan in ("","0","1"): n=1 -elif fan=="auto": n=len(devs) +if fan=="all": + devs=_devices(physical=True); n=len(devs) # ignore CVD: use every physical GPU (reserved node) +elif fan=="auto": + devs=_devices(); n=len(devs) # split across exactly what condor granted +elif fan in ("","0","1"): + devs=_devices(); n=1 else: + devs=_devices() try: n=int(fan) except ValueError: n=1 n=max(1,min(n,len(devs),ngroup)) @@ -274,8 +279,52 @@ def _partition(start,count,n): def ile_gpu_fanout_value(): - """Raw RIFT_ILE_GPU_FANOUT string resolved at DAG-build time (default '1').""" - return os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() or '1' + """Launcher directive baked into ile_pre.sh, resolved from RIFT_ILE_GPU_FANOUT + at DAG-build time. Recognised values (see ile_gpu_request for the matching + condor request): + '' / '0' / '1' -> no fan-out + N (int) -> split the granted GPUs into <=N shards + 'auto' -> split across exactly the GPUs condor granted (CUDA_VISIBLE_DEVICES) + 'all' -> split across EVERY physical GPU (ignore CVD; reserved/whole node) + 'auto-max-N' -> adaptive: condor grants 1..N GPUs; launcher splits across the + granted set, so the baked launcher directive is just 'auto'.""" + raw = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() or '1' + if raw.lower().startswith('auto-max-'): + return 'auto' + return raw + + +def ile_gpu_request(): + """Resolve RIFT_ILE_GPU_FANOUT into the condor (request_gpus, request_cpus) + values for an ILE job. Each is an int (fixed count) OR a string ClassAd + expression (adaptive). Returns (1, 1) when fan-out is off, or for 'auto'/'all' + (those keep request_GPUs=1 and obtain their GPUs at runtime from a node you + have reserved -- the request cannot size them ahead of time). + + 'auto-max-N' emits an expression that requests up to N of the capability-matching + GPUs actually available on the matched (partitionable) slot, so ONE job flavour + lands on a 1/2/3/.../N-GPU node and grabs them all. Override the expression with + RIFT_ILE_GPU_REQUEST_EXPR if your pool exposes GPU counts differently + (verify the attribute with `condor_status -long `).""" + raw = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() + low = raw.lower() + if low in ('', '0', '1', 'auto', 'all'): + return (1, 1) + if low.startswith('auto-max-'): + try: + cap = max(1, int(low.rsplit('-', 1)[1])) + except ValueError: + return (1, 1) + expr = os.environ.get('RIFT_ILE_GPU_REQUEST_EXPR') + if not expr: + cm = "countMatches(RequireGPUs, AvailableGPUs)" + expr = "ifThenElse({cm} >= {N}, {N}, ifThenElse({cm} >= 1, {cm}, 1))".format(cm=cm, N=cap) + return (expr, expr) + try: + n = max(1, int(low)) + return (n, n) + except ValueError: + return (1, 1) def ile_invocation_shell(exe, fanout=None): @@ -299,19 +348,6 @@ def ile_invocation_shell(exe, fanout=None): ) -def ile_gpu_fanout_count(): - """Concrete number of GPUs an ILE job should *request*, parsed from - RIFT_ILE_GPU_FANOUT. Returns int >=1; 'auto' returns 1 (it cannot size a - request ahead of time and relies on the user reserving the node).""" - fan = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip().lower() - if fan in ('', '0', '1', 'auto'): - return 1 - try: - return max(1, int(fan)) - except ValueError: - return 1 - - def mkdir(dir_name): try: os.mkdir(dir_name) @@ -2603,15 +2639,17 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, nGPUs = 'countMatches(RequireGPUs, AvailableGPUs) >= 1 ? 1 : 0' ile_job.add_condor_cmd('rank', 'RequestGPUs') else: - # Multi-GPU fan-out (RIFT_ILE_GPU_FANOUT=N): reserve N GPUs (+N CPUs) - # so HTCondor hands this job the whole-node GPUs that ile_pre.sh then - # splits the intrinsic-grid range across. - fanout = ile_gpu_fanout_count() - if fanout > 1: - nGPUs = fanout - ile_gpu_cpus = fanout + # Multi-GPU fan-out (RIFT_ILE_GPU_FANOUT): request the GPUs (+matching CPUs) + # that ile_pre.sh then splits the intrinsic-grid range across. req_g/req_c + # are an int (fixed N) or a ClassAd expression ('auto-max-N' -> request up to + # N of the GPUs available on the matched slot). 'auto'/'all' keep 1 here and + # grab their GPUs at runtime from a node you have reserved. + req_g, req_c = ile_gpu_request() + if req_g != 1: + nGPUs = req_g + ile_gpu_cpus = req_c if not use_singularity: - ile_job.add_condor_cmd('request_CPUs', str(fanout)) + ile_job.add_condor_cmd('request_CPUs', str(req_c)) ile_job.add_condor_cmd('request_GPUs', str(nGPUs)) # Claim we don't need to make this request anymore to avoid out-of-memory errors. Also, no longer in 'requirements' # requirements.append("CUDAGlobalMemoryMb >= 2048") diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile index 64eabbd00..1384e060e 100644 --- a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile @@ -6,15 +6,24 @@ # with several GPUs reserved, the rest sit idle. The fan-out splits that block # into N shards run concurrently -- one per GPU -- via the generated ile_pre.sh. # -# Turn it on with RIFT_ILE_GPU_FANOUT=N (env) or --ile-gpu-fanout N (CLI) or, for -# asimov, a blueprint key (see README.md + blueprints/). The value is BAKED into -# ile_pre.sh and sets request_GPUs=N/request_CPUs=N, so nothing depends on the +# Turn it on with RIFT_ILE_GPU_FANOUT= (env) or --ile-gpu-fanout (CLI) +# or, for asimov, a blueprint key (see README.md + blueprints/). The value is BAKED +# into ile_pre.sh and sizes request_GPUs/request_CPUs, so nothing depends on the # submit/execute environment -- which is what makes it work under asimov. # +# Values (see README.md "Values -- fixed vs. adaptive"): +# N fixed: request N GPUs+CPUs, split across them +# auto-max-N ADAPTIVE hot-swap (shared pool): request_GPUs = expression for up to N +# of the GPUs available on the matched partitionable slot; split across +# whatever condor grants (1..N) +# all request 1 GPU, but use EVERY physical GPU on the node (reserved/whole node) +# # Targets: # make smoke-local # PROVE the split+pin logic on THIS node's GPUs (no cupy, -# # no condor, no data). Runs the real shipped launcher -# # around a stub ILE. This is the runnable-anywhere proof. +# # no condor, no data). FANOUT=4|all|... This is the +# # runnable-anywhere proof. +# make requests # show request_GPUs/CPUs + baked launcher for each mode +# # (1, 4, all, auto-max-4) -- the "how do I get 1-4 GPUs" ref. # make build # build a real pipeline run dir on the CI data with # # --ile-gpu-fanout (needs SINGULARITY_RIFT_IMAGE). # make verify # assert the generated ILE.sub + ile_pre.sh carry the fan-out. @@ -43,7 +52,7 @@ FANOUT ?= 4 BUILD_DIR := $(CURDIR)/rundir_multigpu DAG := marginalize_intrinsic_parameters_BasicIterationWorkflow.dag -.PHONY: all smoke-local build verify inspect clean coinc help +.PHONY: all smoke-local build verify inspect requests clean coinc help help: @sed -n '2,40p' Makefile @@ -133,10 +142,31 @@ verify: @echo "OK: ILE.sub requests $(FANOUT) GPUs + $(FANOUT) CPUs; ile_pre.sh bakes RIFT_ILE_GPU_FANOUT=$(FANOUT)" @echo " and will split each 100-point ILE block into $(FANOUT) shards, one per GPU." +# --------------------------------------------------------------------------- +# requests: show the condor request_GPUs/request_CPUs + baked launcher directive +# that each fan-out mode produces (fixed N vs. adaptive vs. all-physical). Fast: +# calls write_ILE_sub_simple directly, no full pipeline build. This is the +# reference for "how do I get 1/2/3/4 GPUs?". +# --------------------------------------------------------------------------- +MANIFEST := $(CURDIR)/blueprints/rift_container_family.cit.yaml +requests: + @rm -rf _req && mkdir -p _req/frames_dir && touch _req/frames_dir/H1.gwf + @for MODE in 1 4 all auto-max-4; do \ + cd $(CURDIR)/_req; \ + $(ENV) RIFT_ILE_GPU_FANOUT=$$MODE RIFT_REQUIRE_GPUS='(Capability >= 8.0)' LIGO_ACCOUNTING=x LIGO_USER_NAME=y \ + python3 -c "import os,RIFT.misc.dag_utils_generic as d; \ + j,n=d.write_ILE_sub_simple(tag='ILE',exe='/usr/local/bin/integrate_likelihood_extrinsic_batchmode',request_gpu=True,use_singularity=True,singularity_image='$(MANIFEST)',frames_dir=os.path.abspath('frames_dir'),transfer_files=['../g.xml.gz'],arg_str=' --n-events-to-analyze \$$(macrongroup) --event=\$$(macroevent) ',output_file='CME_out.xml'); j.write_sub_file()" >/dev/null 2>&1; \ + echo "=== RIFT_ILE_GPU_FANOUT=$$MODE ==="; \ + grep -iE 'request_GPUs|request_CPUs' ILE.sub | sed 's/^/ /'; \ + echo " ile_pre.sh launcher directive: $$(grep -o 'RIFT_ILE_GPU_FANOUT:-[A-Za-z0-9]*' ile_pre.sh)"; \ + cd $(CURDIR); \ + done + @rm -rf _req + inspect: @echo "######## $(BUILD_DIR)/ile_pre.sh ########"; cat $(BUILD_DIR)/ile_pre.sh @echo; echo "######## ILE.sub (resource + container lines) ########" @grep -iE 'request_|require_gpus|SingularityImage|executable|when_to_transfer' $(BUILD_DIR)/ILE.sub clean: - rm -rf _smoke $(BUILD_DIR) ci_coinc.xml + rm -rf _smoke _req $(BUILD_DIR) ci_coinc.xml diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md index fa6642142..ea1746921 100644 --- a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md @@ -50,9 +50,47 @@ All three resolve to the same baked `RIFT_ILE_GPU_FANOUT`. | **CLI flag** (direct) | `util_RIFT_pseudo_pipe.py ... --ile-force-gpu --ile-gpu-fanout 4` (also on `create_event_parameter_pipeline_BasicIteration`) | | **asimov blueprint** | `scheduler.environment variables: {RIFT_ILE_GPU_FANOUT: 4}` **or** `scheduler.pipeline: {ile-gpu-fanout: 4}` | -Values: an integer `N` (also sizes `request_GPUs`/`request_CPUs`), or `auto` (split -across whatever GPUs are visible at runtime — for a whole node held with -`request_GPUs=1`; `auto` cannot size the request, so reserve the node yourself). +### Values — fixed vs. adaptive ("hot-swap 1–4") + +`RIFT_ILE_GPU_FANOUT` / `--ile-gpu-fanout` accepts: + +| Value | `request_GPUs` (condor) | What the launcher splits across | Use when | +|---|---|---|---| +| `1` / unset | 1 | — (no fan-out) | default | +| `N` (int) | `N` | the N granted GPUs (adapts down if fewer) | every node has exactly N GPUs | +| `auto-max-N` | **expression** ≤ N | exactly the GPUs condor granted | **shared pool, partitionable GPU slots — the real hot-swap** | +| `all` | 1 | **every physical GPU** (ignores `CUDA_VISIBLE_DEVICES`) | **dedicated / whole node you reserved** | +| `auto` | 1 | the GPUs condor granted | you sized a multi-GPU slot some other way | + +**The runtime split is already fully adaptive** — the launcher splits the point block +across however many GPUs it is handed (1, 2, 3, 4 …), always covering the whole block. +So "adapt to the number found" is solved regardless of value. The only real question is +how to make condor *hand you* a variable number; that is what the two adaptive values do: + +- **`auto-max-N` (shared pool):** HTCondor's plain `request_GPUs` is a single fixed count, + so it cannot natively say "give me 1 to 4". This value instead emits a ClassAd + **expression** for `request_GPUs`/`request_CPUs` that asks for *up to N of the + capability-matching GPUs available on the matched (partitionable) slot* — so ONE job + flavour lands on a 1/2/3/4-GPU slot and grabs them all, and the launcher (`auto`) fans + out across exactly that many. The default expression is + `ifThenElse(countMatches(RequireGPUs,AvailableGPUs) >= N, N, ifThenElse(... >= 1, ..., 1))` + (same `countMatches` idiom RIFT already uses for cross-platform GPU matching). It + **requires partitionable slots and a GPU-aware negotiator** — verify on your pool with + `condor_status -long | grep -i gpu`, and if the attribute differs, override the + whole expression with `RIFT_ILE_GPU_REQUEST_EXPR=''` (no code change). + +- **`all` (dedicated node):** if you already reserve whole nodes, keep `request_GPUs=1` + (so it matches a node with *any* number of GPUs) and let the launcher enumerate **all + physical GPUs** via `nvidia-smi`, ignoring `CUDA_VISIBLE_DEVICES`. Simplest, needs no + partitionable-slot support. **Caveats:** only safe when the node is exclusively yours + (otherwise you would step on co-scheduled jobs), and it assumes condor is *not* cgroup- + isolating the GPU devices (with strict device isolation a shard pinned to a non-granted + GPU cannot use it). It also still requests 1 CPU — bump `request_CPUs` if your scheduler + confines CPUs too. + +Is a variable request "even possible"? Not as a plain `request_GPUs` number — that is one +value. It *is* possible either as the `auto-max-N` expression (condor sizes the dynamic +slot to what's available) or by reserving the node and using `all`. Pick by pool type. --- diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml index 5cea6e325..c882664ad 100644 --- a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml @@ -41,6 +41,12 @@ scheduler: # ======================================================================= environment variables: RIFT_ILE_GPU_FANOUT: 4 # split each ILE block across 4 GPUs (one shard/GPU) + # Hot-swap variants (request a VARIABLE number of GPUs, adapt to what's found): + # RIFT_ILE_GPU_FANOUT: auto-max-4 # shared/partitionable pool: request up to 4 of the + # # GPUs available on the matched slot; split across + # # whatever condor grants (1..4). + # RIFT_ILE_GPU_FANOUT: all # dedicated/whole node: request 1, use every physical GPU. + # (See README.md "Values -- fixed vs. adaptive".) # ======================================================================= # (2) ALTERNATIVE (equivalent): pass it as a pseudo_pipe CLI flag. Every key From 4bda6a23044a40ed05d4e30bf55436a54427dcd9 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Sat, 27 Jun 2026 04:25:20 -0700 Subject: [PATCH 11/63] ILE multi-GPU fan-out: default to grab ALL node GPUs; single-GPU fallback Per deployment policy (whole nodes are reserved; partitionable GPU slots are not a sustainable long-term path), make the GPU multi-GPU policy default to 'all': - DEFAULT_ILE_GPU_FANOUT = 'all'. When RIFT_ILE_GPU_FANOUT is unset, a GPU ILE job keeps request_GPUs=1 (matching unchanged -- lands on any GPU node exactly as before) but the launcher enumerates EVERY physical GPU (nvidia-smi, ignoring CUDA_VISIBLE_DEVICES) and splits the ILE block across all of them. On a 1-GPU node this is a no-op; only multi-GPU nodes change. - Fallback to the old single-GPU run: RIFT_ILE_GPU_FANOUT=1 (or 'single'/'off', or --ile-gpu-fanout 1). Aliases handled in the resolver and the launcher. - Safety: the bake is gated on request_gpu, so a CPU-only ILE job always bakes '1' and never grabs the node's GPUs under the 'all' default. Implementation: shared _raw_ile_gpu_fanout() applies the default + single/off aliases; ile_gpu_fanout_value()/ile_gpu_request() build on it; write_ILE_sub_simple passes fanout=(ile_gpu_fanout_value() if request_gpu else '1'). Mirrored in dag_utils.py and dag_utils_generic.py. Note: HTCondor partitionable GPU slots DO work today (verified on the CIT pool: a 2-GPU partitionable slot carves per-GPU dynamic slots), so auto-max-N remains available, but it is no longer the recommended/default path. Demo updated: README "Default policy" + Values table (all=default, 1/single=fallback); `make requests` shows default vs fallback; blueprint defaults to no override. Verified: default bakes 'all' and runs across all 4 physical GPUs even with CVD=0; RIFT_ILE_GPU_FANOUT=1 runs single; CPU-only job bakes '1'; fixed N and auto-max-N unchanged. Co-Authored-By: Claude Opus 4.8 --- .../Code/RIFT/misc/dag_utils.py | 64 +++++++++++++------ .../Code/RIFT/misc/dag_utils_generic.py | 64 +++++++++++++------ .../Code/demo/rift/infra/multi_gpu/Makefile | 24 ++++--- .../Code/demo/rift/infra/multi_gpu/README.md | 44 +++++++++---- .../multi_gpu/blueprints/rift-multigpu.yaml | 25 ++++---- 5 files changed, 146 insertions(+), 75 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py index 0d04c5af4..7af5f7677 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils.py @@ -124,7 +124,7 @@ def _partition(start,count,n): devs=_devices(physical=True); n=len(devs) # ignore CVD: use every physical GPU (reserved node) elif fan=="auto": devs=_devices(); n=len(devs) # split across exactly what condor granted -elif fan in ("","0","1"): +elif fan in ("","0","1","single","off"): devs=_devices(); n=1 else: devs=_devices() @@ -150,36 +150,58 @@ def _partition(start,count,n): ''' +# Default multi-GPU policy for GPU ILE jobs when RIFT_ILE_GPU_FANOUT is unset. +# 'all' = grab EVERY physical GPU on the node and split the ILE batch across them +# (request_GPUs stays 1, so matching is unchanged -- the job lands on any GPU node, +# then uses all of that node's GPUs). This assumes whole nodes are reserved; on a +# SHARED multi-GPU node it would step on co-scheduled jobs, so set RIFT_ILE_GPU_FANOUT=1 +# (or 'single'/'off', or --ile-gpu-fanout 1) to fall back to the old single-GPU run. +# We deliberately do NOT use 'auto-max-N' (partitionable-GPU-slot) as the default: +# partitionable GPU slots are not considered a sustainable long-term path. +DEFAULT_ILE_GPU_FANOUT = 'all' + + +def _raw_ile_gpu_fanout(): + """RIFT_ILE_GPU_FANOUT with the default applied and aliases normalised. + 'single'/'off' -> '1'. Returns a lowercase string.""" + raw = os.environ.get('RIFT_ILE_GPU_FANOUT') + if raw is None or raw.strip() == '': + raw = DEFAULT_ILE_GPU_FANOUT + low = raw.strip().lower() + if low in ('single', 'off'): + return '1' + return low + + def ile_gpu_fanout_value(): """Launcher directive baked into ile_pre.sh, resolved from RIFT_ILE_GPU_FANOUT at DAG-build time. Recognised values (see ile_gpu_request for the matching condor request): - '' / '0' / '1' -> no fan-out - N (int) -> split the granted GPUs into <=N shards - 'auto' -> split across exactly the GPUs condor granted (CUDA_VISIBLE_DEVICES) - 'all' -> split across EVERY physical GPU (ignore CVD; reserved/whole node) - 'auto-max-N' -> adaptive: condor grants 1..N GPUs; launcher splits across the - granted set, so the baked launcher directive is just 'auto'.""" - raw = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() or '1' - if raw.lower().startswith('auto-max-'): + '1' / 'single' / 'off' -> no fan-out, single-GPU run (the fallback) + 'all' (DEFAULT) -> split across EVERY physical GPU (ignore CVD; whole node) + N (int) -> split the granted GPUs into <=N shards + 'auto' -> split across exactly the GPUs condor granted (CUDA_VISIBLE_DEVICES) + 'auto-max-N' -> adaptive (partitionable slots): condor grants 1..N GPUs; + launcher splits across the granted set, so baked value is 'auto'.""" + low = _raw_ile_gpu_fanout() + if low.startswith('auto-max-'): return 'auto' - return raw + return low def ile_gpu_request(): """Resolve RIFT_ILE_GPU_FANOUT into the condor (request_gpus, request_cpus) values for an ILE job. Each is an int (fixed count) OR a string ClassAd - expression (adaptive). Returns (1, 1) when fan-out is off, or for 'auto'/'all' - (those keep request_GPUs=1 and obtain their GPUs at runtime from a node you - have reserved -- the request cannot size them ahead of time). + expression (adaptive). Returns (1, 1) for the default 'all' and for 'auto'/'off' + (those keep request_GPUs=1 -- 'all'/'auto' obtain their GPUs at runtime from a node + you have reserved; the request cannot size them ahead of time). 'auto-max-N' emits an expression that requests up to N of the capability-matching - GPUs actually available on the matched (partitionable) slot, so ONE job flavour - lands on a 1/2/3/.../N-GPU node and grabs them all. Override the expression with - RIFT_ILE_GPU_REQUEST_EXPR if your pool exposes GPU counts differently - (verify the attribute with `condor_status -long `).""" - raw = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() - low = raw.lower() + GPUs actually available on the matched (partitionable) slot. Override the expression + with RIFT_ILE_GPU_REQUEST_EXPR if your pool exposes GPU counts differently + (verify the attribute with `condor_status -long `). Partitionable GPU + slots are not the recommended path; 'all' is the default instead.""" + low = _raw_ile_gpu_fanout() if low in ('', '0', '1', 'auto', 'all'): return (1, 1) if low.startswith('auto-max-'): @@ -1251,7 +1273,9 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, f.write("for i in `ls " + frames_local + "`; do echo "+ frames_local + "/$i; done > base_paths.dat \n") f.write("paste local_stripped.cache base_paths.dat > local_relative.cache \n") f.write("cp local_relative.cache local.cache \n") - f.write(ile_invocation_shell(exe)) + # Only GPU ILE jobs fan out; a CPU-only ILE job bakes '1' so the default + # 'all' policy never makes a non-GPU job grab the node's GPUs. + f.write(ile_invocation_shell(exe, fanout=(ile_gpu_fanout_value() if request_gpu else '1'))) os.system("chmod a+x ile_pre.sh") ile_job.set_executable("ile_pre.sh") # transferred, used as executable # ile_job.add_condor_cmd('+PreCmd', '"ile_pre.sh"') diff --git a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py index d2ec44ad8..3156869a2 100644 --- a/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py +++ b/MonteCarloMarginalizeCode/Code/RIFT/misc/dag_utils_generic.py @@ -252,7 +252,7 @@ def _partition(start,count,n): devs=_devices(physical=True); n=len(devs) # ignore CVD: use every physical GPU (reserved node) elif fan=="auto": devs=_devices(); n=len(devs) # split across exactly what condor granted -elif fan in ("","0","1"): +elif fan in ("","0","1","single","off"): devs=_devices(); n=1 else: devs=_devices() @@ -278,36 +278,58 @@ def _partition(start,count,n): ''' +# Default multi-GPU policy for GPU ILE jobs when RIFT_ILE_GPU_FANOUT is unset. +# 'all' = grab EVERY physical GPU on the node and split the ILE batch across them +# (request_GPUs stays 1, so matching is unchanged -- the job lands on any GPU node, +# then uses all of that node's GPUs). This assumes whole nodes are reserved; on a +# SHARED multi-GPU node it would step on co-scheduled jobs, so set RIFT_ILE_GPU_FANOUT=1 +# (or 'single'/'off', or --ile-gpu-fanout 1) to fall back to the old single-GPU run. +# We deliberately do NOT use 'auto-max-N' (partitionable-GPU-slot) as the default: +# partitionable GPU slots are not considered a sustainable long-term path. +DEFAULT_ILE_GPU_FANOUT = 'all' + + +def _raw_ile_gpu_fanout(): + """RIFT_ILE_GPU_FANOUT with the default applied and aliases normalised. + 'single'/'off' -> '1'. Returns a lowercase string.""" + raw = os.environ.get('RIFT_ILE_GPU_FANOUT') + if raw is None or raw.strip() == '': + raw = DEFAULT_ILE_GPU_FANOUT + low = raw.strip().lower() + if low in ('single', 'off'): + return '1' + return low + + def ile_gpu_fanout_value(): """Launcher directive baked into ile_pre.sh, resolved from RIFT_ILE_GPU_FANOUT at DAG-build time. Recognised values (see ile_gpu_request for the matching condor request): - '' / '0' / '1' -> no fan-out - N (int) -> split the granted GPUs into <=N shards - 'auto' -> split across exactly the GPUs condor granted (CUDA_VISIBLE_DEVICES) - 'all' -> split across EVERY physical GPU (ignore CVD; reserved/whole node) - 'auto-max-N' -> adaptive: condor grants 1..N GPUs; launcher splits across the - granted set, so the baked launcher directive is just 'auto'.""" - raw = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() or '1' - if raw.lower().startswith('auto-max-'): + '1' / 'single' / 'off' -> no fan-out, single-GPU run (the fallback) + 'all' (DEFAULT) -> split across EVERY physical GPU (ignore CVD; whole node) + N (int) -> split the granted GPUs into <=N shards + 'auto' -> split across exactly the GPUs condor granted (CUDA_VISIBLE_DEVICES) + 'auto-max-N' -> adaptive (partitionable slots): condor grants 1..N GPUs; + launcher splits across the granted set, so baked value is 'auto'.""" + low = _raw_ile_gpu_fanout() + if low.startswith('auto-max-'): return 'auto' - return raw + return low def ile_gpu_request(): """Resolve RIFT_ILE_GPU_FANOUT into the condor (request_gpus, request_cpus) values for an ILE job. Each is an int (fixed count) OR a string ClassAd - expression (adaptive). Returns (1, 1) when fan-out is off, or for 'auto'/'all' - (those keep request_GPUs=1 and obtain their GPUs at runtime from a node you - have reserved -- the request cannot size them ahead of time). + expression (adaptive). Returns (1, 1) for the default 'all' and for 'auto'/'off' + (those keep request_GPUs=1 -- 'all'/'auto' obtain their GPUs at runtime from a node + you have reserved; the request cannot size them ahead of time). 'auto-max-N' emits an expression that requests up to N of the capability-matching - GPUs actually available on the matched (partitionable) slot, so ONE job flavour - lands on a 1/2/3/.../N-GPU node and grabs them all. Override the expression with - RIFT_ILE_GPU_REQUEST_EXPR if your pool exposes GPU counts differently - (verify the attribute with `condor_status -long `).""" - raw = os.environ.get('RIFT_ILE_GPU_FANOUT', '1').strip() - low = raw.lower() + GPUs actually available on the matched (partitionable) slot. Override the expression + with RIFT_ILE_GPU_REQUEST_EXPR if your pool exposes GPU counts differently + (verify the attribute with `condor_status -long `). Partitionable GPU + slots are not the recommended path; 'all' is the default instead.""" + low = _raw_ile_gpu_fanout() if low in ('', '0', '1', 'auto', 'all'): return (1, 1) if low.startswith('auto-max-'): @@ -2762,7 +2784,9 @@ def write_ILE_sub_simple(tag='integrate', exe=None, log_dir=None, use_eos=False, f.write("for i in `ls " + frames_local + "`; do echo "+ frames_local + "/$i; done > base_paths.dat \n") f.write("paste local_stripped.cache base_paths.dat > local_relative.cache \n") f.write("cp local_relative.cache local.cache \n") - f.write(ile_invocation_shell(exe)) + # Only GPU ILE jobs fan out; a CPU-only ILE job bakes '1' so the default + # 'all' policy never makes a non-GPU job grab the node's GPUs. + f.write(ile_invocation_shell(exe, fanout=(ile_gpu_fanout_value() if request_gpu else '1'))) os.system("chmod a+x ile_pre.sh") ile_job.set_executable("ile_pre.sh") # transferred, used as executable # ile_job.add_condor_cmd('+PreCmd', '"ile_pre.sh"') diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile index 1384e060e..ca2ce492d 100644 --- a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/Makefile @@ -6,17 +6,19 @@ # with several GPUs reserved, the rest sit idle. The fan-out splits that block # into N shards run concurrently -- one per GPU -- via the generated ile_pre.sh. # -# Turn it on with RIFT_ILE_GPU_FANOUT= (env) or --ile-gpu-fanout (CLI) -# or, for asimov, a blueprint key (see README.md + blueprints/). The value is BAKED -# into ile_pre.sh and sizes request_GPUs/request_CPUs, so nothing depends on the +# DEFAULT: GPU ILE jobs grab EVERY physical GPU on the node ('all'): request_GPUs=1 +# (matching unchanged), but the launcher splits the ILE block across all of the node's +# GPUs. Assumes whole nodes are reserved. Override with RIFT_ILE_GPU_FANOUT= +# (env) or --ile-gpu-fanout (CLI) or an asimov blueprint key. The value is +# BAKED into ile_pre.sh and sizes request_GPUs/request_CPUs, so nothing depends on the # submit/execute environment -- which is what makes it work under asimov. # -# Values (see README.md "Values -- fixed vs. adaptive"): +# Values (see README.md "Values"): +# all DEFAULT: request 1 GPU, use EVERY physical GPU on the node (reserved node) +# 1 / single FALLBACK: single-GPU run (shared node / old behaviour) # N fixed: request N GPUs+CPUs, split across them -# auto-max-N ADAPTIVE hot-swap (shared pool): request_GPUs = expression for up to N -# of the GPUs available on the matched partitionable slot; split across -# whatever condor grants (1..N) -# all request 1 GPU, but use EVERY physical GPU on the node (reserved/whole node) +# auto-max-N partitionable-slot pool (not recommended): request up to N of the GPUs +# available on the matched slot; split across whatever condor grants (1..N) # # Targets: # make smoke-local # PROVE the split+pin logic on THIS node's GPUs (no cupy, @@ -151,9 +153,11 @@ verify: MANIFEST := $(CURDIR)/blueprints/rift_container_family.cit.yaml requests: @rm -rf _req && mkdir -p _req/frames_dir && touch _req/frames_dir/H1.gwf - @for MODE in 1 4 all auto-max-4; do \ + @echo "(default = unset => 'all'; '1'/single = fallback)" + @for MODE in default 1 all 4 auto-max-4; do \ cd $(CURDIR)/_req; \ - $(ENV) RIFT_ILE_GPU_FANOUT=$$MODE RIFT_REQUIRE_GPUS='(Capability >= 8.0)' LIGO_ACCOUNTING=x LIGO_USER_NAME=y \ + if [ $$MODE = default ]; then unset RIFT_ILE_GPU_FANOUT; else export RIFT_ILE_GPU_FANOUT=$$MODE; fi; \ + $(ENV) RIFT_REQUIRE_GPUS='(Capability >= 8.0)' LIGO_ACCOUNTING=x LIGO_USER_NAME=y \ python3 -c "import os,RIFT.misc.dag_utils_generic as d; \ j,n=d.write_ILE_sub_simple(tag='ILE',exe='/usr/local/bin/integrate_likelihood_extrinsic_batchmode',request_gpu=True,use_singularity=True,singularity_image='$(MANIFEST)',frames_dir=os.path.abspath('frames_dir'),transfer_files=['../g.xml.gz'],arg_str=' --n-events-to-analyze \$$(macrongroup) --event=\$$(macroevent) ',output_file='CME_out.xml'); j.write_sub_file()" >/dev/null 2>&1; \ echo "=== RIFT_ILE_GPU_FANOUT=$$MODE ==="; \ diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md index ea1746921..e4f3762c6 100644 --- a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/README.md @@ -15,8 +15,26 @@ by *parameter value*, not filename. Coverage is identical to the serial run (the shards partition the range exactly), and the launcher returns the first non-zero shard exit code, so condor retry/hold behaviour is preserved. -It is **off by default** and a no-op unless you ask for it. With it disabled the -generated `ile_pre.sh` just `exec`s the ILE binary exactly as before. +### Default policy: grab every GPU on the node + +**GPU ILE jobs now fan out across all of the node's GPUs by default** (`RIFT_ILE_GPU_FANOUT` +unset ⇒ `all`). The job still **requests one GPU** (`request_GPUs=1`), so condor matching +is unchanged — it lands on any GPU node exactly as before — but at runtime the launcher +enumerates **every physical GPU** (`nvidia-smi`, ignoring `CUDA_VISIBLE_DEVICES`) and splits +the point block across all of them. On a 1-GPU node this is a byte-for-byte no-op; only +multi-GPU nodes change. + +> **This assumes you reserve whole nodes.** Requesting 1 GPU but using all of them means +> condor still thinks the other GPUs are free — on a *shared* multi-GPU node it would step +> on co-scheduled jobs. If you are not reserving whole nodes, set the single-GPU fallback. + +**Fallback to the old single-GPU run:** `RIFT_ILE_GPU_FANOUT=1` (or `single` / `off`, or +`--ile-gpu-fanout 1`). Then `ile_pre.sh` just `exec`s the ILE binary exactly as before. + +Why this default and not the partitionable-slot `auto-max-N`? HTCondor *does* support +partitionable GPU slots today (verified on the CIT pool: a 2-GPU partitionable slot carves +per-GPU dynamic slots), but that path is not considered sustainable long-term, so the +default is the reservation-based `all` instead. --- @@ -40,26 +58,26 @@ time**: --- -## Three ways to turn it on +## Setting the policy -All three resolve to the same baked `RIFT_ILE_GPU_FANOUT`. +The default (`all`) needs no action — GPU ILE jobs fan out across the node automatically. +To **override** it (a fixed count, or the single-GPU fallback), set `RIFT_ILE_GPU_FANOUT` +any of these ways (all resolve to the same baked value): | Context | How | |---|---| -| **Env var** (interactive) | `export RIFT_ILE_GPU_FANOUT=4` before `util_RIFT_pseudo_pipe.py` | -| **CLI flag** (direct) | `util_RIFT_pseudo_pipe.py ... --ile-force-gpu --ile-gpu-fanout 4` (also on `create_event_parameter_pipeline_BasicIteration`) | -| **asimov blueprint** | `scheduler.environment variables: {RIFT_ILE_GPU_FANOUT: 4}` **or** `scheduler.pipeline: {ile-gpu-fanout: 4}` | - -### Values — fixed vs. adaptive ("hot-swap 1–4") +| **Env var** (interactive) | `export RIFT_ILE_GPU_FANOUT=1` before `util_RIFT_pseudo_pipe.py` (e.g. fallback) | +| **CLI flag** (direct) | `util_RIFT_pseudo_pipe.py ... --ile-force-gpu --ile-gpu-fanout 1` (also on `create_event_parameter_pipeline_BasicIteration`) | +| **asimov blueprint** | `scheduler.environment variables: {RIFT_ILE_GPU_FANOUT: 1}` **or** `scheduler.pipeline: {ile-gpu-fanout: 1}` | -`RIFT_ILE_GPU_FANOUT` / `--ile-gpu-fanout` accepts: +### Values | Value | `request_GPUs` (condor) | What the launcher splits across | Use when | |---|---|---|---| -| `1` / unset | 1 | — (no fan-out) | default | +| `all` **(unset ⇒ default)** | 1 | **every physical GPU** (ignores `CUDA_VISIBLE_DEVICES`) | **dedicated / whole node you reserved** | +| `1` / `single` / `off` | 1 | — (no fan-out) | **fallback: shared node / single-GPU run** | | `N` (int) | `N` | the N granted GPUs (adapts down if fewer) | every node has exactly N GPUs | -| `auto-max-N` | **expression** ≤ N | exactly the GPUs condor granted | **shared pool, partitionable GPU slots — the real hot-swap** | -| `all` | 1 | **every physical GPU** (ignores `CUDA_VISIBLE_DEVICES`) | **dedicated / whole node you reserved** | +| `auto-max-N` | **expression** ≤ N | exactly the GPUs condor granted | shared pool, partitionable GPU slots (not the recommended path) | | `auto` | 1 | the GPUs condor granted | you sized a multi-GPU slot some other way | **The runtime split is already fully adaptive** — the launcher splits the point block diff --git a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml index c882664ad..2b04d9cd1 100644 --- a/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml +++ b/MonteCarloMarginalizeCode/Code/demo/rift/infra/multi_gpu/blueprints/rift-multigpu.yaml @@ -34,19 +34,20 @@ scheduler: singularity base exe directory: '/usr/local/bin/' # ======================================================================= - # (1) PRIMARY: encode the fan-out as an environment variable. rift.py copies - # every key here into os.environ BEFORE invoking util_RIFT_pseudo_pipe.py - # (rift.py: "adding environment variables before starting pipeline"), so it - # reaches the DAG build with NO assumption about the submit shell. + # GPU fan-out. THE DEFAULT IS ALREADY 'all' (grab every physical GPU on the node, + # request_GPUs=1), so for a whole-node-reservation pool you need set NOTHING here. + # Set RIFT_ILE_GPU_FANOUT only to OVERRIDE the default. rift.py copies every key + # under `environment variables` into os.environ BEFORE invoking + # util_RIFT_pseudo_pipe.py ("adding environment variables before starting pipeline"), + # so it reaches the DAG build with NO assumption about the submit shell. # ======================================================================= - environment variables: - RIFT_ILE_GPU_FANOUT: 4 # split each ILE block across 4 GPUs (one shard/GPU) - # Hot-swap variants (request a VARIABLE number of GPUs, adapt to what's found): - # RIFT_ILE_GPU_FANOUT: auto-max-4 # shared/partitionable pool: request up to 4 of the - # # GPUs available on the matched slot; split across - # # whatever condor grants (1..4). - # RIFT_ILE_GPU_FANOUT: all # dedicated/whole node: request 1, use every physical GPU. - # (See README.md "Values -- fixed vs. adaptive".) + environment variables: {} + # Override examples (uncomment ONE): + # RIFT_ILE_GPU_FANOUT: 1 # FALLBACK: single-GPU run (shared node / old behaviour) + # RIFT_ILE_GPU_FANOUT: 2 # fixed: request exactly 2 GPUs+CPUs, split across them + # RIFT_ILE_GPU_FANOUT: auto-max-4 # partitionable-slot pool (not recommended long-term): + # # request up to 4 of the GPUs available on the matched slot + # (See README.md "Values".) # ======================================================================= # (2) ALTERNATIVE (equivalent): pass it as a pseudo_pipe CLI flag. Every key From 87b0e5520d0fa05a10f18a886a090cb9900ce763 Mon Sep 17 00:00:00 2001 From: Colm Talbot Date: Mon, 6 Jul 2026 14:43:13 -0400 Subject: [PATCH 12/63] BUG: make sure marginal log likelihood calculation uses all available values --- .../Code/bin/util_CIPDirSummarizeEvidence.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MonteCarloMarginalizeCode/Code/bin/util_CIPDirSummarizeEvidence.py b/MonteCarloMarginalizeCode/Code/bin/util_CIPDirSummarizeEvidence.py index bf0a17b59..17f29adb8 100755 --- a/MonteCarloMarginalizeCode/Code/bin/util_CIPDirSummarizeEvidence.py +++ b/MonteCarloMarginalizeCode/Code/bin/util_CIPDirSummarizeEvidence.py @@ -31,8 +31,8 @@ net_dat.append([lnL, sigma_lnL, n_eff]) net_dat =np.array(net_dat) -lnL = np.average(lnL, weights=1./sigma_lnL**2) -sigma_lnL = np.max([np.sqrt(np.mean(sigma_lnL**2)/len(net_dat)),np.std(lnL)]) # not quite right but ok +lnL = np.average(net_dat[:, 0], weights=1./net_dat[:, 1]**2) +sigma_lnL = np.max([np.sqrt(np.mean(net_dat[:, 1]**2)/len(net_dat)),np.std(net_dat[:, 0])]) # not quite right but ok dat_out = [lnL, sigma_lnL] if opts.stream_output: print(*dat_out) From 975810724d9d02f86462a3bc0abc812791c9872a Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Wed, 15 Jul 2026 04:18:03 -0700 Subject: [PATCH 13/63] Fix Virgo calibration correction convention --- .../Code/RIFT/calmarg/calibration.py | 26 +++++++++++++ .../Code/bin/calibration_reweighting.py | 8 +++- .../Code/test/test_calmarg_calibration.py | 37 +++++++++++++++++++ 3 files changed, 70 insertions(+), 1 deletion(-) create mode 100644 MonteCarloMarginalizeCode/Code/RIFT/calmarg/calibration.py create mode 100644 MonteCarloMarginalizeCode/Code/test/test_calmarg_calibration.py diff --git a/MonteCarloMarginalizeCode/Code/RIFT/calmarg/calibration.py b/MonteCarloMarginalizeCode/Code/RIFT/calmarg/calibration.py new file mode 100644 index 000000000..46b74453b --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/RIFT/calmarg/calibration.py @@ -0,0 +1,26 @@ +"""Helpers for matching bilby-pipe calibration conventions.""" + + +def correction_type_for_ifo(setting, ifo_name, parse_dict=None): + """Resolve bilby-pipe's calibration correction type for one detector.""" + if setting is None or setting == "None": + return "template" if ifo_name == "V1" else "data" + + if isinstance(setting, str): + if setting in ("data", "template"): + return setting + if parse_dict is None: + raise ValueError("parse_dict is required for detector-specific settings") + setting = parse_dict(setting) + + try: + correction_type = setting[ifo_name] + except (KeyError, TypeError) as exc: + raise ValueError( + f"No calibration correction type specified for {ifo_name}" + ) from exc + if correction_type not in ("data", "template"): + raise ValueError( + f"Invalid calibration correction type for {ifo_name}: {correction_type}" + ) + return correction_type diff --git a/MonteCarloMarginalizeCode/Code/bin/calibration_reweighting.py b/MonteCarloMarginalizeCode/Code/bin/calibration_reweighting.py index 12b431735..70fca7c27 100755 --- a/MonteCarloMarginalizeCode/Code/bin/calibration_reweighting.py +++ b/MonteCarloMarginalizeCode/Code/bin/calibration_reweighting.py @@ -57,6 +57,7 @@ # TODO this should not be a hardcoded path! import RIFT.calmarg.rift_source as rift_source +from RIFT.calmarg.calibration import correction_type_for_ifo from bilby.core.utils import logger @@ -315,6 +316,8 @@ def alt_reweight(result, label=None, new_likelihood=None, new_prior=None, spline_calibration_envelope_dict = bilby_pipe.utils.convert_string_to_dict( data.meta_data['command_line_args']['spline_calibration_envelope_dict']) +calibration_correction_type = data.meta_data['command_line_args'].get( + 'calibration_correction_type') ifos_for_reweighting = deepcopy(ifos) for ifo in ifos: # removes any model for the calibration that was set up in the file ifo.calibration_model = bilby.gw.calibration.Recalibrate() @@ -392,7 +395,10 @@ def alt_reweight(result, label=None, new_likelihood=None, new_prior=None, if args.use_local_cal_files: calibration_file_path = './cal_envelopes/' + os.path.basename(calibration_file_path) # force local, specific name. Copied in place earlier ifo_calibration_priors = bilby.gw.prior.CalibrationPriorDict.from_envelope_file( - calibration_file_path, ifo.minimum_frequency, ifo.maximum_frequency, 10, ifo.name) + calibration_file_path, ifo.minimum_frequency, ifo.maximum_frequency, 10, + ifo.name, correction_type=correction_type_for_ifo( + calibration_correction_type, ifo.name, + parse_dict=bilby_pipe.utils.convert_string_to_dict)) # TODO FOR DEBUGGING PURPOSES # for key in ifo_calibration_priors.keys(): diff --git a/MonteCarloMarginalizeCode/Code/test/test_calmarg_calibration.py b/MonteCarloMarginalizeCode/Code/test/test_calmarg_calibration.py new file mode 100644 index 000000000..43cdb0a4c --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/test/test_calmarg_calibration.py @@ -0,0 +1,37 @@ +import pytest + +from RIFT.calmarg.calibration import correction_type_for_ifo + + +@pytest.mark.parametrize( + "ifo_name, expected", + [("H1", "data"), ("L1", "data"), ("K1", "data"), ("V1", "template")], +) +def test_bilby_pipe_default_correction_types(ifo_name, expected): + assert correction_type_for_ifo(None, ifo_name) == expected + + +@pytest.mark.parametrize("setting", ["data", "template"]) +def test_global_correction_type(setting): + assert correction_type_for_ifo(setting, "V1") == setting + + +def test_detector_specific_correction_types(): + setting = {"H1": "template", "V1": "data"} + assert correction_type_for_ifo(setting, "H1") == "template" + assert correction_type_for_ifo(setting, "V1") == "data" + + +def test_string_detector_specific_correction_types(): + def parse_dict(value): + assert value == "{H1: data, V1: template}" + return {"H1": "data", "V1": "template"} + + assert correction_type_for_ifo( + "{H1: data, V1: template}", "V1", parse_dict=parse_dict + ) == "template" + + +def test_missing_detector_is_rejected(): + with pytest.raises(ValueError, match="No calibration correction type"): + correction_type_for_ifo({"H1": "data"}, "V1") From eac6636a3f14eeee10be382e02bedd02695d7a0d Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Mon, 20 Jul 2026 14:31:55 -0400 Subject: [PATCH 14/63] 0.0.17.9rc1 --- CHANGES.rst | 4 +++- setup.py | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index 507316f0b..c41581f62 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -4,7 +4,9 @@ development tree is rift_O4c_staging -> rift_O4c; draft MR notes at https://git.ligo.org/rapidpe-rift/rift/-/merge_requests/49 - (rc0) write_bilby_pickle: shutil.copyfile threw error if cache file already existed (copy into same file error) in code that protected against duplicate IFO entries. - + - (rc1) V calibration convention sign (rapidpe-rift/rift!50); multi-container capability; multi-GPU 'fanout' capability +release is rc1 + 0.0.17.8 ------------ diff --git a/setup.py b/setup.py index 71965c22c..dcdaa0c7b 100644 --- a/setup.py +++ b/setup.py @@ -34,7 +34,7 @@ setuptools.setup( name="RIFT", - version="0.0.17.9rc0", # do not build on OSX machine, side effects + version="0.0.17.9rc1", # do not build on OSX machine, side effects author="Richard O'Shaughnessy", author_email="richard.oshaughnessy@ligo.org", description="RIFT parameter estimation pipeline. Note branch used is temp-RIT-Tides-port_python3_restructure_package (which will become master shortly)!", From a1abeb28088c605268093230b783c6ade2a35b4a Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Mon, 20 Jul 2026 14:32:05 -0400 Subject: [PATCH 15/63] 0.0.17.9 --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index dcdaa0c7b..c40443484 100644 --- a/setup.py +++ b/setup.py @@ -34,7 +34,7 @@ setuptools.setup( name="RIFT", - version="0.0.17.9rc1", # do not build on OSX machine, side effects + version="0.0.17.9", # do not build on OSX machine, side effects author="Richard O'Shaughnessy", author_email="richard.oshaughnessy@ligo.org", description="RIFT parameter estimation pipeline. Note branch used is temp-RIT-Tides-port_python3_restructure_package (which will become master shortly)!", From 4363eecaba24fd6aff97f7c209f3a05ce2d4620e Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Wed, 22 Jul 2026 10:43:51 -0700 Subject: [PATCH 16/63] ILE: honour --srate-resample-time-marginalization instead of always doubling The time-marginalisation upsampling block treated --srate-resample-time-marginalization as a boolean: whenever the requested rate exceeded --srate it refined the internal time grid by a hardcoded factor of two and discarded the requested value. With the O4c production settings (--srate 4096, --data-integration-window-half 0.075) the internal grid is tvals = linspace(-0.075, 0.075, int(0.15*4096) = 614) whose spacing is 0.15/613 s (4086.7 Hz - already ~0.2% coarser than 1/4096, because linspace spans the closed interval with N points). Doubling that gives an exported time resolution of 8173 Hz. Every O4c production RIFT run requested 16384 Hz and exported at ~8.2 kHz instead; this was confirmed by measuring the minimum spacing between distinct geocentre times in extrinsic_posterior_samples.dat across all 70 production rundirs. Three changes: * derive the refinement factor from the requested rate; * derive it from the actual grid spacing rather than fSample, so ceil(requested/fSample) cannot land just short of the target; * end the dense grid on tvals[-1] rather than tvals[-1] + deltaT/2, so the cubic spline is no longer asked to extrapolate past its last knot. The dense grid still contains every original node, so lnL at the original times is unchanged - this is a strict refinement, not a re-derivation. Adds test/test_srate_resample_time_marginalization.py, including a guard that fails if the shipped block and the tested reference implementation drift apart. Co-Authored-By: Claude Opus 4.8 --- .../integrate_likelihood_extrinsic_batchmode | 11 +- ...est_srate_resample_time_marginalization.py | 166 ++++++++++++++++++ 2 files changed, 175 insertions(+), 2 deletions(-) create mode 100644 MonteCarloMarginalizeCode/Code/test/test_srate_resample_time_marginalization.py diff --git a/MonteCarloMarginalizeCode/Code/bin/integrate_likelihood_extrinsic_batchmode b/MonteCarloMarginalizeCode/Code/bin/integrate_likelihood_extrinsic_batchmode index 5cb53aa90..bc0c62f96 100755 --- a/MonteCarloMarginalizeCode/Code/bin/integrate_likelihood_extrinsic_batchmode +++ b/MonteCarloMarginalizeCode/Code/bin/integrate_likelihood_extrinsic_batchmode @@ -1359,10 +1359,17 @@ def resample_samples(my_samples, # IF UPSAMPLING, PERFORM NOW. (Currently on if opts.srate_resample_time_marginalization and opts.srate_resample_time_marginalization > fSample: deltaT_orig = tvals[1]-tvals[0] - tvals_denser = tvals[0] + deltaT_orig/2 * np.arange(2*len(tvals)) + # Refinement factor from the REQUESTED rate, measured against the actual grid + # spacing (tvals is a closed-interval linspace, so deltaT_orig is slightly + # larger than 1/fSample and ceil(requested/fSample) would fall just short). + n_upsample = max(2, int(np.ceil(opts.srate_resample_time_marginalization * deltaT_orig))) + n_dense = n_upsample*(len(tvals)-1) + 1 + # Terminate on tvals[-1]: the old grid ran half a sample past the last knot, + # forcing the spline to extrapolate. + tvals_denser = tvals[0] + (deltaT_orig/n_upsample) * np.arange(n_dense) from scipy.interpolate import RegularGridInterpolator, CubicSpline # cubic spline at first, easiest - generally not exporting too many events - lnLt_new = np.zeros( (lnLt.shape[0], lnLt.shape[1]*2) ) + lnLt_new = np.zeros( (lnLt.shape[0], n_dense) ) for indx_here in np.arange(n_samples): cs = CubicSpline(tvals, lnLt[indx_here]) lnLt_new[indx_here] = cs(tvals_denser) diff --git a/MonteCarloMarginalizeCode/Code/test/test_srate_resample_time_marginalization.py b/MonteCarloMarginalizeCode/Code/test/test_srate_resample_time_marginalization.py new file mode 100644 index 000000000..ce3896d1b --- /dev/null +++ b/MonteCarloMarginalizeCode/Code/test/test_srate_resample_time_marginalization.py @@ -0,0 +1,166 @@ +#!/usr/bin/env python3 +""" +Unit tests for the time-marginalisation upsampling used by +``--srate-resample-time-marginalization`` in +``bin/integrate_likelihood_extrinsic_batchmode``. + +Before the fix, the option was effectively a boolean: whenever the requested +rate exceeded --srate, the internal time grid was refined by a hardcoded factor +of two and the requested value was discarded. With the O4c production settings +(--srate 4096, --data-integration-window-half 0.075) asking for 16384 Hz +delivered ~8173 Hz, and the exported geocentre times inherited that resolution. + +These tests exercise a transcription of the shipped block, kept in sync by +``test_source_matches_reference_implementation`` below. +""" + +import os +import re + +import numpy as np +import pytest + +# RIFT defaults exercised by the O4c production configuration. +SRATE = 4096.0 +WINDOW_HALF = 75e-3 # --data-integration-window-half default +REQUESTED = 16384 + +ILE_SCRIPT = os.path.join( + os.path.dirname(os.path.abspath(__file__)), + "..", + "bin", + "integrate_likelihood_extrinsic_batchmode", +) + + +def rift_tvals(srate=SRATE, window_half=WINDOW_HALF): + """The internal time grid built by analyze_event_extrinsic_export.""" + n_points = int(2 * window_half / (1.0 / srate)) + return np.linspace(-window_half, window_half, n_points) + + +def upsample(tvals, lnLt, requested, fsample=SRATE): + """Reference implementation, mirroring the shipped code.""" + from scipy.interpolate import CubicSpline + + if not (requested and requested > fsample): + return tvals, lnLt + deltaT_orig = tvals[1] - tvals[0] + n_upsample = max(2, int(np.ceil(requested * deltaT_orig))) + n_dense = n_upsample * (len(tvals) - 1) + 1 + tvals_denser = tvals[0] + (deltaT_orig / n_upsample) * np.arange(n_dense) + lnLt_new = np.zeros((lnLt.shape[0], n_dense)) + for index in range(lnLt.shape[0]): + lnLt_new[index] = CubicSpline(tvals, lnLt[index])(tvals_denser) + return tvals_denser, lnLt_new + + +def effective_rate(tvals): + return 1.0 / np.diff(tvals).min() + + +@pytest.fixture +def toy_lnl(): + """A smooth, sharply peaked lnL(t): Gaussians of ~1 ms width.""" + tvals = rift_tvals() + peak = np.array([[-0.7e-3], [0.0], [1.3e-3]]) + return tvals, -0.5 * ((tvals[None, :] - peak) / 1.0e-3) ** 2 + + +def test_internal_grid_is_slightly_coarser_than_srate(): + """ + linspace(-W, W, N) with N = int(2*W*fS) spans the closed interval with N + points, so the spacing is deltaT*N/(N-1) - about 0.2% coarser than 1/fS. + The refinement factor must therefore be derived from the grid spacing, not + from fSample, or the result lands just short of the requested rate. + """ + tvals = rift_tvals() + assert len(tvals) == 614 + assert tvals[1] - tvals[0] > 1.0 / SRATE + assert effective_rate(tvals) == pytest.approx(4086.67, rel=1e-4) + + +def test_reaches_the_requested_rate(toy_lnl): + tvals, lnl = toy_lnl + dense, _ = upsample(tvals, lnl, REQUESTED) + assert effective_rate(dense) >= REQUESTED + + +def test_scales_with_the_request(toy_lnl): + """Regression guard for the old behaviour, which ignored the value.""" + tvals, lnl = toy_lnl + rate_16k = effective_rate(upsample(tvals, lnl, 16384)[0]) + rate_32k = effective_rate(upsample(tvals, lnl, 32768)[0]) + assert rate_16k >= 16384 + assert rate_32k >= 32768 + assert rate_32k > 1.5 * rate_16k + + +def test_does_not_extrapolate_outside_the_original_grid(toy_lnl): + """ + The previous grid, tvals[0] + (dt/2)*arange(2N), ended half a sample past + tvals[-1], where CubicSpline extrapolates. + """ + tvals, lnl = toy_lnl + dense, _ = upsample(tvals, lnl, REQUESTED) + assert dense[0] == pytest.approx(tvals[0]) + assert dense[-1] == pytest.approx(tvals[-1]) + + +def test_preserves_the_original_nodes(toy_lnl): + """Refinement, not re-derivation: lnL at the original times is unchanged.""" + tvals, lnl = toy_lnl + dense, lnl_dense = upsample(tvals, lnl, REQUESTED) + factor = max(2, int(np.ceil(REQUESTED * (tvals[1] - tvals[0])))) + np.testing.assert_allclose(dense[::factor], tvals, rtol=1e-12, atol=1e-12) + np.testing.assert_allclose(lnl_dense[:, ::factor], lnl, rtol=1e-9, atol=1e-9) + + +def test_recovers_the_peak_to_the_requested_resolution(toy_lnl): + """ + The exported geocentre time is drawn from this grid, so the grid spacing + floors the achievable time resolution. + """ + tvals, lnl = toy_lnl + truth = np.array([-0.7e-3, 0.0, 1.3e-3]) + dense, lnl_dense = upsample(tvals, lnl, REQUESTED) + error = np.abs(dense[np.argmax(lnl_dense, axis=1)] - truth).max() + assert error < 1.0 / REQUESTED + + +def test_switch_is_off_at_or_below_fsample(toy_lnl): + tvals, lnl = toy_lnl + for requested in (None, 0, 2048, int(SRATE)): + dense, lnl_dense = upsample(tvals, lnl, requested) + np.testing.assert_array_equal(dense, tvals) + np.testing.assert_array_equal(lnl_dense, lnl) + + +def test_source_matches_reference_implementation(): + """ + Guard against the shipped block and this reference drifting apart - the + tests above are only meaningful if they describe the real code. + """ + if not os.path.exists(ILE_SCRIPT): + pytest.skip("ILE script not found next to the test directory") + with open(ILE_SCRIPT) as handle: + source = handle.read() + + block = re.search( + r"if opts\.srate_resample_time_marginalization and .*?lnLt_norm = " + r"scipy\.special\.logsumexp\(lnLt,axis=-1\)", + source, + re.S, + ) + assert block, "could not locate the upsampling block" + text = block.group(0) + + # The requested rate must actually be used, not just tested for truthiness. + assert "np.ceil(opts.srate_resample_time_marginalization" in text + # ...and the old hardcoded doubling must be gone. + assert "np.arange(2*len(tvals))" not in text + assert "lnLt.shape[1]*2" not in text + + +if __name__ == "__main__": + raise SystemExit(pytest.main([__file__, "-v"])) From 8faa1c7e14e68bf487db5c8cef0af3458b07f0b7 Mon Sep 17 00:00:00 2001 From: Richard O'Shaughnessy Date: Wed, 22 Jul 2026 10:49:48 -0700 Subject: [PATCH 17/63] asimov: allow the RIFT bootstrap source to be named explicitly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bootstrapping RIFT from an existing PE result currently requires expressing that result as an asimov dependency, so _find_posterior can scan `needs:` for a pipeline publishing a 'samples' asset. That is awkward when the file is simply known to be on disk, and it is fragile: only the gwdata pipeline returns 'samples' as a single path to a PESummary metafile. The bilby pipeline returns a *list* of raw bilby result files, h5py rejects it, and the bare `except Exception: pass` swallows the error - leaving the run with no bootstrap at all and no message. Adds a new optional `scheduler: bootstrap file:`, naming the PESummary metafile directly and skipping the dependency scan: scheduler: bootstrap upstream: True bootstrap file: /path/{event}/…/pesummary/samples/posterior_samples.h5 dataset: