G2Project.save() launches git and file on every call, which dominates run time for parallel scripted refinements on HPC cluster #364

reproducer.py.txt.py

Summary. With GSASIIscriptable, each G2Project.refine() calls save(), and each save() starts three external processes: git twice (via GitPython) and file once (via platform.architecture()). For a single interactive refinement this costs little, but for many scripted refinements running in parallel on a shared filesystem it dominates: in my case ~90% of all CPU time was kernel overhead and per-node throughput dropped ~9×.

Environment. GSAS-II #5841 via conda (gsas2pkg, briantoby channel), Python 3.11, Linux; EPCC Cirrus, 288-core AMD nodes; conda env and GSAS-II on NFS (/epccfs). Workload: many independent serial Python processes, each running ~89 refine() calls (11-phase Rietveld, 11-step strategy).

Evidence (cluster).

1. strace -f -e trace=execve of one worker (~89 refines): 3,708 execve — 178 × git, 89 × /usr/bin/file, plus ~1,700 failed PATH lookups.
2. With 288 workers per node: ~50% of wall time in wait4 (waiting on child processes), ~50% in openat/newfstatat at 6–23 ms per call (NFS).
3. Slurm accounting, 288 workers x 1 sequence: UserCPU 4:54 h vs SystemCPU 46:29 h.
4. Throughput vs workers/node: 72 -->3,013 sequences/h; 144 -->3,013/h; 288 -->1,492/h. With the workaround below, 288 workers: 674 s --> 72 s median per sequence, ~14,000 sequences/node-h, identical refinement results (same Rwp to all digits).

Evidence (laptop-scale reproducer). The attached script builds an empty G2Project and calls save() 100 times, counting child processes via subprocess.Popen. On a git-based install: 300 child processes (200 × git, 100 × file), 11.4 ms/save on a local SSD -entirely process-spawn overhead. After the fix below: 0 child processes, 0.1 ms/save.

Cause (main @ f75620d):

1. save() → G2fil.get_python_versions() → platform.architecture(), which in CPython runs file -b <sys.executable> on POSIX on every call [GSASIIscriptable.py:1021 → GSASIIfiles.py:123].
2. save() opens a new Repo and reads head.commit on every call -->one git cat-file process per save [GSASIIscriptable.py:1025–1028].
3. GSASIIpath.HowIsG2Installed() is documented as cached, but the two return 'github'+rev statements return before setting G2_installed_result [GSASIIpath.py:208, 211]. For GitHub-based installs (the normal conda/git install) the cache is never filled, so every call reopens the repo and reads remote().urls -->one git remote get-url --all process per save.

Workaround currently in use (in the worker script, before refining):

import platform as _platform
_ARCH = _platform.architecture()
_platform.architecture = lambda *a, **k: _ARCH
G2sc.GSASIIpath.G2_installed_result = 'noVCS'   # version then from git_verinfo.py

Suggested fix (small, behaviour-preserving; saved .gpx metadata unchanged):

1. Set G2_installed_result before the 'github'+rev returns in HowIsG2Installed().
2. Cache the HEAD commit hash (it cannot change during a running process) instead of creating a new Repo per save.
3. Use f"{struct.calcsize('P')*8}bit" in get_python_versions() — same '64bit' string as platform.architecture()[0], no subprocess.

Opening a PR - patch is ready and tested.