Experimental performance family (default-off): load-balance infrastructure, active-box windowing, block-structured AMR, hybrid WENO/Riemann sensors

[sbryngelson]

Summary

An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is s_mpi_decompose_computational_domain, refactored through the new m_box module (byte-identical; covered by the existing suite).

• Load-balance infrastructure — m_box (partition arithmetic), m_load_weight/load_weight_wrt (per-cell load-weight field + imbalance metric), m_sfc_partition/sfc_partition_wrt (Morton-SFC predicted-imbalance diagnostic), m_load_balance/load_balance (weighted static decomposition at init; AMR-fine-work-aware), m_rank_timing/rank_time_wrt (per-rank compute-time diagnostic).
• Active-box windowing — m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.
• Hybrid reconstruction/flux sensors — hybrid_weno and hybrid_riemann (+hybrid_weno_eps, hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).
• Block-structured AMR — m_amr + m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (blocks may span rank seams; fine halo exchange; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.

Full algorithm and user documentation: docs/documentation/amr.md (support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).

AMR physics support matrix (abridged; authoritative table in amr.md)

Supported and golden-tested: single- and multi-fluid (5-eq, mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-block pb/mv side-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) · active_box (blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).

Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) · amr_subcycle under IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).

Validation evidence

• Conservation: reflux-exact (~1e-15 defects) across single/multi-fluid, viscous, bubbles, chemistry, phase change, through subcycled + regridding advances.
• Free-stream preservation: exact (0.0) with subcycling + regrid armed, including 3D, IGR, and Lagrangian configs.
• Decomposition exactness: np=1 == np=2 element-exact with blocks spanning rank seams (fields byte-identical incl. chemistry species).
• Methodology: every feature validated by AMR-vs-no-AMR reference comparison with a control experiment calibrating the expected error scale, plus mechanism evidence from run logs (regrid counts, box coordinates, ghost-point counts) — a golden that does not mechanically exercise its path is treated as no coverage. Liveness goldens for the hybrid sensors are deliberately sized so a silently dead sensor fails them (~5e-4 signal vs 5e-5 tolerance) — they caught a real dead-sensor bug on Cray GPU during CI hardening.
• GPU: full test suite with post-process on 2×V100 — OpenACC 644/644 executed pass, OpenMP offload 643/644 (one slurm launch flake). Speed spot-check (800² 2D blast, 2 ranks): AMR + dynamic regrid costs +3% wall over no-AMR at these sizes; ACC ≈ 12% faster than OMP; acc/omp solutions bitwise identical.
• CI: one fully green 85-lane matrix on record (all GitHub-hosted lanes: gnu/intel × debug/no-debug/reldebug × double/single/mixed, macos, and the NVHPC 23.11–26.3 compat matrix). Fixes landed during hardening were each reproduced locally in the exact failing configuration first (intel FP drift, single-precision tolerance classification, gfortran bounds-check OOB, OpenACC present-table mapping ×3).

Known issues (all non-gating or in progress)

• Frontier/Cray, in progress: two device-mapping issues in the IB machinery under CCE (plain-IBM descriptor corruption on gpu-omp; a present-table crash in the AMR-IB swap on gpu-acc). Being fixed with direct Cray iteration; the CCE bench/case-opt lane failures share the same root.
• NVHPC 24.1/24.3 compat lanes (informational, continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK, -tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.
• Accuracy postures documented rather than hidden: multi-fluid alpha-sum deviation bounded ~5.7e-3 at historical block faces during shock crossing (non-growing); viscous prolongation-ghost seam ~1e-6 np-dependence; IGR seam conservation is truncation-order (no reflux capture from the fused IGR kernels — exact-conservation capture is noted future work).

Review guide

The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).

All parameters ship default-off with case_validator entries, runtime checker gates, and case.md/amr.md documentation.