Spare localusers from the epilog's killall - #1403
Closed
100milliongold wants to merge 3 commits into
Closed
Conversation
…and recent Slurm
The exclusive-job check in run-parts.sh had two independent failures:
1. It called scontrol/squeue through PATH. slurmd's environment does not
include a custom slurm_install_prefix, so both commands produced empty
output, numcpus_sys and numcpus_job were both "", the comparison was
true, and every job ran the *-exclusive-* prolog/epilog scripts. On a
shared node this reset power limits and clocks on all GPUs and dropped
page caches for every job.
2. It parsed "scontrol show job" with grep -Eio "TRES=cpu=[0-9]+". On
recent Slurm the output has both ReqTRES= and AllocTRES= lines, so the
pattern matched twice and numcpus_job became a multi-line value that
never compared equal. With scontrol on PATH, exclusive jobs were
therefore never detected.
Use {{ slurm_install_prefix }}/bin/squeue by absolute path (the file is
already deployed via the template module) and read allocated CPUs and node
count with -o %C / -o %D instead of parsing scontrol. Guard against an
empty result so a lookup failure means "not exclusive" rather than
"exclusive".
Observed on DGX OS 7.5.0, Slurm 26.05.1, slurm_install_prefix=/raid/slurm/usr/local:
- before the binaries were symlinked into /usr/local/bin, every srun
--gres=gpu:1 job logged "Running .../50-exclusive-gpu" and prolog took
6-8 s (srun: Prolog hung on node);
- after symlinking, a bash -x run of the script showed numcpus_job='1<nl>56'.
Signed-off-by: Jea-Eok-Kim <je.kim@xiilab.com>
This script runs with "set -e", so a bare `numcpus_job=$(squeue ...)` assignment aborts the entire prolog/epilog run when squeue exits nonzero. Redirecting stderr does not suppress the exit status. The result is that a transient squeue failure skips every part script, not just the exclusive ones, and the job fails. Both allocation fields are now fetched by a single `-o "%C %D"` call inside an `if` condition, so the failure is visible and handled. A failed, empty or non-numeric lookup leaves exclusive=0 and logs a warning. The same reasoning is applied to the last-user-job count, with one difference: piping squeue into `wc -l` hides its exit status, and a failed lookup would be read as "no other jobs" and run the *-lastuserjob-* cleanup scripts while another job of the same user is still on the node. A failed lookup now leaves last_user_job=0. Decision matrix, verified on a DGX B300 (Ubuntu 24.04, bash 5.2, 256 CPUs) by substituting a squeue stub that honours the -o format and the -j flag: case before after ---------- ------------------------------ ------------------------------ exclusive rc=0 all three parts ran rc=0 all three parts ran shared rc=0 exclusive part skipped rc=0 exclusive part skipped otherjobs rc=0 lastuserjob part skipped rc=0 lastuserjob part skipped empty rc=0 silently non-exclusive rc=0 non-exclusive + 1 warning nonnumeric rc=0 silently non-exclusive rc=0 non-exclusive + 1 warning fail rc=1 NO part ran at all rc=0 normal part ran + 2 warnings The three normal cases are unchanged, so there is no regression; the failure cases stop taking the whole run down and stop deciding silently.
41-lastuserjob-ssh checks /etc/slurm/localusers.backup before it touches a user, so operator accounts keep their access when a job ends. 40-lastuserjob-processes does not, yet it is the broader of the two: `killall -9 -u` reaps every process the user owns on the node, including an interactive login shell and its sshd session. An operator who submits from a login shell on a compute node is therefore disconnected the instant the epilog runs, even though that account is listed in localusers.backup precisely so it will not be cut off. Observed on a two-node cluster where the controller is also a compute node. Apply the same guard 41 already uses. Job cleanup is unchanged for every user not in that file, and with ProctrackType=proctrack/cgroup the job's own processes are already reaped by Slurm, so this script is a sweep for strays rather than the primary teardown. 42-lastuserjob-cleanup has the same asymmetry -- it deletes the user's files under /tmp and /dev/shm with no exemption check -- but its blast radius is much smaller, so it is left alone here.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happens
epilog.d/41-lastuserjob-sshchecks/etc/slurm/localusers.backupbefore it touches a user:epilog.d/40-lastuserjob-processesdoes not, even though it is the broader of the two:killall -9 -ureaps every process the user owns on that node. That includes an interactive login shell and thesshdsession carrying it.So an operator listed in
localusers.backup— listed there precisely so their access is not cut off — is disconnected the moment their job's epilog runs.How we hit it
Two-node cluster where the controller is also a compute node, so the account submitting jobs also has a login shell on a node that runs the epilog.
and on the operator's terminal, at the same moment:
40leaves no log line of its own here, because itsloggercall is insideif killall ...; thenandkillalltakes down the shell that would have carried the message.There is a second, downstream symptom. Node Health Check runs
check_ps_service -u root -d sshd sshd; on a socket-activatedsshd(Ubuntu 24.04) there is no persistentsshdprocess once the last session dies, so NHC failed 2.5 minutes later:That NHC check is arguably wrong on its own for socket activation, and is not addressed here.
The change
Apply the guard
41already uses, unchanged in form.Cleanup behaviour is identical for every user not in
localusers.backup. And withProctrackType=proctrack/cgroupthe job's own processes are already reaped by Slurm, so this script is a sweep for strays that escaped the cgroup rather than the primary teardown — skipping it for a handful of operator accounts does not leave the node dirty.Verified
Applied to both nodes of the cluster above:
Not changed
42-lastuserjob-cleanuphas the same asymmetry — it removes the user's files under/tmpand/dev/shmwith no exemption check. Its blast radius is much smaller thankillall -9, so it is left alone here rather than widened into this PR. Happy to follow up if you would rather the three scripts agree.