Repository navigation
Multi-process ParFlow stand-alone fails on GNU stack #87
Description
Activity
It fails at the line
pfrun default_single.I double-checked that in the same environment (both GNU and Intel) with GCC- and Intel/LLVM-compiled binaries of a trivial hello-world program works for both compilers.
Yes, but only when I'm submitting it through Slurm (in an interactive job). Otherwise, it is the other way around, I found out. These are maybe two different issues and need some details...
For the GCC+OpenMPI case,
vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3 # make sure that SLURM_JOBID is not set at this point default_single : PASSED vanhulten1@jrlogin09:.../parflow/test/tcl$ salloc --partition=dc-cpu-devel --nodes=1 --account=cdetect --time=0:10:00 salloc: Granted job allocation 13986807 salloc: Waiting for resource configuration salloc: Nodes jrc0096 are ready for job vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3 -------------------------------------------------------------------------- There are not enough slots available in the system to satisfy the 6 slots that were requested by the application: /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025GNU/bin/JURECADC_ParFlow/bin/parflow Either request fewer procs for your application, or make more slots available for use. A "slot" is the PRRTE term for an allocatable unit where we can launch a process. The number of slots available are defined by the environment in which PRRTE processes are run: 1. Hostfile, via "slots=N" clauses (N defaults to number of processor cores if not provided) 2. The --host command line parameter, via a ":N" suffix on the hostname (N defaults to 1 if not provided) 3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.) 4. If none of a hostfile, the --host command line parameter, or an RM is present, PRRTE defaults to the number of processor cores In all the above cases, if you want PRRTE to default to the number of hardware threads instead of the number of processor cores, use the --use-hwthread-cpus option. Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the number of available slots when deciding the number of processes to launch. -------------------------------------------------------------------------- child process exited abnormally while executing "exec sh /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025GNU/bin/JURECADC_ParFlow/bin/run default_single 6" ("eval" body line 1) invoked from within "eval exec sh $Parflow::PARFLOW_DIR/bin/run $runname $NumProcs" invoked from within "if [pfexists Process.Command] { set command [pfget Process.Command] puts [format "Using command : %s" [format $command $NumProcs $runname]] puts [e..." (procedure "pfrun" line 53) invoked from within "pfrun default_single" (file "default_single.tcl" line 324)
For the Intel+ParaStationMPI case,
vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3 srun: error: job_submit_filter: please specify the job's account with -A or --account option srun: error: Hint: to find available budget accounts from each project please run: jutil user projects srun: error: Unable to allocate resources: Invalid account or account/partition combination specified child process exited abnormally while executing "exec sh /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025Intel/bin/JURECADC_ParFlow/bin/run default_single 6" ("eval" body line 1) invoked from within "eval exec sh $Parflow::PARFLOW_DIR/bin/run $runname $NumProcs" invoked from within "if [pfexists Process.Command] { set command [pfget Process.Command] puts [format "Using command : %s" [format $command $NumProcs $runname]] puts [e..." (procedure "pfrun" line 53) invoked from within "pfrun default_single" (file "default_single.tcl" line 324) vanhulten1@jrlogin09:.../parflow/test/tcl$ salloc --partition=dc-cpu-devel --nodes=1 --account=cdetect --time=0:10:00 salloc: Granted job allocation 13986806 salloc: Waiting for resource configuration salloc: Nodes jrc0096 are ready for job vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3 default_single : PASSED
The results are the same when running
gmake testfrom the ParFlow build directory ($TSMP2_DIR/bld/JURECADC_ParFlow/ParFlow/src/ParFlow-build/): starting from Test 9, Intel fails on compute and GNU fails on login node.After some testing, here are my findings and hypotheses.
In the case of GCC+OpenMPI, there is a binary
mpirunavailable, which is taken (precedence oversrun) on the login node when invokingtclsh default_single.tcl 1 2 3. This works fine.
Then when making a reservation,mpirunis of course also available, but this is not the right way to run jobs. Instead,srunshould be used on JSC machines.
In the case of Intel+ParaStationMPI, there is nompirun, just/usr/bin/srun, so the job fails without a reservation (srunis part of Slurm and so a reservation must be made) and with a reservation it succeeds.For the GCC+OpenMPI case, I tried adding
cmake_conf+=" -DMPIEXEC=srun -DMPIEXEC_NUMPROC_FLAG=-n"and settingSLURM_NTASKS=6, but it does not fix the problem.
When I set at runtimePARFLOW_MPIEXEC_EXTRA_FLAGS="--oversubscribe",tclsh default_single.tcl 1 2 3takes longer than a few minutes (it hangs).Likely shell variables (like
PARFLOW_MPIEXEC_EXTRA_FLAGSandSLURM_NTASKS) are overridden by command parameters in the ParFlow runtime system (e.g. in a Tcl module).FYI, ParFlow MPI options are baked in
BuildParFlow.cmake:TSMP2/cmake/BuildParFlow.cmake
Lines 80 to 82 in 0252d09
Can you try to play with these settings Marco? You can also log their values to see how the MPI options change when choosing different toolchains. These options may have to be fixed to account for JSC's
srunas you rightly mentioned.I though, let's set
-DMPIEXEC_EXECUTABLE=srun, but…/bld/JURECADC_ParFlow/ParFlow/src/ParFlow-build/srun: No such file or directoryMPIEXEC_EXECUTABLEclearly is not what I thought it was. I then set-DMPIEXEC=srun -DMPIEXEC_NUMPROC_FLAG=-nbut that turned up the same error, just like setting
SLURM_NTASKSdoes not help. I think it is overridden somewhere in the scripts. I have to dig, or maybe @skollet could help?The problem is that no more than 1 slot is available. This should illustrate it:
vanhulten1@jrlogin09:.../parflow/test/tcl$ salloc --partition=dc-cpu-devel --nodes=1 --account=cdetect --time=0:10:00 salloc: Granted job allocation 13993235 salloc: Waiting for resource configuration salloc: Nodes jrc0736 are ready for job vanhulten1@jrlogin09:.../parflow/test/tcl$ export SLURM_NTASKS=6 vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 1 1 default_single : PASSED vanhulten1@jrlogin09:.../parflow/test/tcl$ tclsh default_single.tcl 1 2 3 -------------------------------------------------------------------------- There are not enough slots available in the system to satisfy the 6 slots that were requested by the application: /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025GNU/bin/JURECADC_ParFlow/bin/parflow Either request fewer procs for your application, or make more slots available for use. A "slot" is the PRRTE term for an allocatable unit where we can launch a process. The number of slots available are defined by the environment in which PRRTE processes are run: 1. Hostfile, via "slots=N" clauses (N defaults to number of processor cores if not provided) 2. The --host command line parameter, via a ":N" suffix on the hostname (N defaults to 1 if not provided) 3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.) 4. If none of a hostfile, the --host command line parameter, or an RM is present, PRRTE defaults to the number of processor cores In all the above cases, if you want PRRTE to default to the number of hardware threads instead of the number of processor cores, use the --use-hwthread-cpus option. Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the number of available slots when deciding the number of processes to launch. -------------------------------------------------------------------------- child process exited abnormally while executing "exec sh /p/project1/cdetect/vanhulten1/models/TSMP2.25811.2025GNU/bin/JURECADC_ParFlow/bin/run default_single 6" ("eval" body line 1) invoked from within "eval exec sh $Parflow::PARFLOW_DIR/bin/run $runname $NumProcs" invoked from within "if [pfexists Process.Command] { set command [pfget Process.Command] puts [format "Using command : %s" [format $command $NumProcs $runname]] puts [e..." (procedure "pfrun" line 53) invoked from within "pfrun default_single" (file "default_single.tcl" line 324)
I got a different error, which gave me a hint:
No PMIx server was reachable, but a PMI1/2 was detected.
It seems that all this time during my GCC+OpenMPI testing,
SLURM_MPI_TYPEwas not set. If this is set topspmix, the job works. Loading the OpenMPI module also sets this variable. I did not think it was an environmental variable, because the same modules are loaded before and after sourcing the environment file. Of course, shell variables can be changed after loading modules, but I do not know why specifically this variable was unset.It may be wise to alway reload your modules (sourcing an env file) after starting an interactive session (or normal job submission, where you probably really have to do this).
Moreover, I noticed that when running salloc and letting Slurm revoke your allocation, you stay in a subshell (which I found out when diffing
setoutput and noticing aSHLVLthat tracks shell nesting).I am checking if this was the only issue, or if something from above was also needed...
...yes, CMake configuration changes were needed and so this note is a side remark. Patch will follow...
- added 2 commits that reference this issue
on Aug 15, 2025 - added a parent issue
on May 27, 2026 - moved this from New features to Housekeeping in TSMP2: Towards Stages/2026 support
on May 27, 2026 - moved this to New features in TSMP2: Towards Stages/2026 support
on May 27, 2026 - moved this from Housekeeping to Issues in TSMP2: Towards Stages/2026 support
on Jun 4, 2026
When compiling ParFlow with the Intel compiler (default), I can successfully run it manually (through tclsh).
However, when compiling ParFlow with GCC and OpenMPI, running fails with
I was sure to use the same environment file before running ParFlow.
ParaStationMPI/5.10.0-1is loadedOpenMPI/5.0.5is loadedI'm using JURECA and Stages/2025.