simulation_manager: container universe and output-transfer timing

[oshaughnessy-junior]

Paired with the SuperNu OSPool preset extraction — these land together. The preset is what motivated both arguments, and it cannot be extracted onto its project's main without them without hand-rolling the same lines again.

Why

Two things every OSG backend reaches through extra_condor_cmds, which is emitted last and so replaces the queue's own lines rather than extending them.

universe = container + container_image is how OSG documents running in a container today. This queue knew only the legacy +SingularityImage form, so a backend targeting OSG hand-rolled the modern one — silently replacing the queue's own universe line, with the outcome decided by whichever line condor read last.

when_to_transfer_output was hardcoded ON_EXIT, which discards the sandbox when a job is evicted. On a preemptable pool that throws away whatever the job had already written. A backend whose science is output files needs ON_EXIT_OR_EVICT, and had no way to ask for it that did not clobber something.

What

DualCondorRunQueue(
    container_image="osdf:///ospool/ap41/data/<user>/supernu-v2.sif",
    when_to_transfer_output="ON_EXIT_OR_EVICT",
)

container_image selects the universe too — a container_image under a vanilla universe is silently ignored by condor, and a caller asked to remember both will eventually forget one.

The image is not resolved or fetched: an osdf:// or docker:// reference is routinely unreadable from the submit host, so demanding local readability would refuse the ordinary OSG case. Only a value that could corrupt the submit file is rejected.

Setting container_image and use_singularity is refused. Both say "run this in a container"; emitted together, which one wins is a property of the site. The legacy form still works alone — this is not a deprecation, and pools that honour only +SingularityImage exist.

when_to_transfer_output is validated against HTCondor's four legal values here rather than at the schedd. By the time condor_submit refuses a typo, the archive has already recorded the sim as dispatched, so the failure presents as a stuck simulation instead of a configuration error.

universe, container_image and when_to_transfer_output join _PROTECTED_SUBMIT_COMMANDS, consistent with #138: the queue composes them and there is now an argument for each, so leaving the old route open beside the new one just means the next backend author finds it first. Each refusal names the argument to use instead.

Defaults are unchanged — no image is a vanilla job, and the timing stays ON_EXIT.

Tests

34 in a new test_condor_container_universe.py, including condor_submit -dry-run over five shapes (-dry-run contacts no schedd and queues nothing).

27 fail against rift_O4d. Note the method: a bare run against the base fails at collection, because the new WHEN_TO_TRANSFER_OUTPUT constant does not exist there — that proves nothing about the behaviour. The 27 is measured with the import shimmed so collection succeeds. The 7 that pass are the invariants: vanilla by default, the legacy form working alone, ON_EXIT by default, and condor accepting the unchanged shapes.

Full RIFT/simulation_manager/tests/ suite: 184 pass, clean under -W error::RuntimeWarning. The one pre-existing failure in the tree, test/test_database.py::test_dag_chains_levels_per_sim, fails on rift_O4d too.

🤖 Generated with Claude Code

[oshaughnessy-junior]

Paired consumer PR is up: sim_manager_supernu#3, also draft. It is what motivated both arguments here.

Verified end to end against this branch — the preset's options through a real DualCondorRunQueue, and condor_submit -dry-run accepts the result:

universe                = container
container_image         = osdf:///ospool/<ap>/data/<user>/r3/supernu-v2.sif
when_to_transfer_output = ON_EXIT_OR_EVICT
getenv                  = False
periodic_release        = (((HoldReasonCode =?= 34)) && (NumJobStarts < 5)) || (((HoldReasonCode =!= 1) && (NumJobStarts < 50)) && !((HoldReasonCode =?= 34)))
+ProjectName            = "..."

One line for the universe, not two — which was the concrete symptom: the preset appended its own after this queue's universe = vanilla and relied on condor honouring the last one.