perf: grind uses minimum wall-clock per-stage time, not running-mean cpu_time

[sbryngelson]

What

Change the grind performance metric (ns/gp/eq/rhs, printed as Performance: and written to time_data.dat) from a running mean of cpu_time to the minimum wall-clock time of a full RK stage.

The timer also moves from s_compute_rhs to the stage loop in s_tvd_rk, so it now brackets the whole per-stage compute+communication — RHS + halo H2D/D2H copies + RK update + source terms + the IBM ghost-point correction — while still excluding file I/O.

Why

The old metric was noisy and incomplete on multi-rank GPU runs:

1. cpu_time is per-process CPU time. An MPI wait that busy-spins the host accumulates CPU time, so idle ranks look busy. At 8 ranks the reduction (maxval(proc_time)) then picks the rank that spun most, not the one that computed most. → system_clock (wall) measures real elapsed time.
2. A running mean is dragged by outlier steps (OS jitter, first-touch/JIT, page migration). → the minimum is the most reproducible estimator: jitter only ever adds time, so the floor reflects the hardware. The min over stages also naturally drops the s == 1 stage that carries any run-time-info/probe I/O.
3. The old timer bracketed only s_compute_rhs, so per-stage work outside it — most notably the IBM correction (s_ibm_correct_state) — was never counted. The ibm benchmark's per-step cost was invisible to grind.

Validation (amdflang 23.1.0, MI250X, -n 8 MPI, gbpp 2)

Same binary, repeated runs:

metric	mean	range	CV
old cpu_time (igr)	1.98	1.65–2.14	9.3%
new wall+min (igr)	1.81	1.67–1.98	5.8%
new wall+min (ibm)	4.09	3.74–4.83	10.4%

igr's run-to-run CV drops ~38%. (Residual spread is contention on a shared dev node; a dedicated CI node will be tighter.) ibm now produces a grind that includes the IBM correction.

Notes for reviewers

• Output-neutral: only perf-timing variables change; no field/physics output is touched, so golden regression files are unaffected.
• This PR's own benchmark comparison is not meaningful — it compares the new metric (this branch) against the old metric (master). The comparison is only valid once merged, when both sides use the same definition. Absolute grind values shift versus historical numbers by design.
• The ETA/progress-bar timer (wall_time_avg) is intentionally left on cpu_time; it is cosmetic and separate from the perf gate.
• Alternative considered: median instead of min would also catch a purely periodic per-step cost (e.g. every-Nth-step) that min can skip, at the cost of slightly more variance. Happy to switch if reviewers prefer.

[Copilot]

Pull request overview

This PR updates the simulation “grind” performance metric to use the minimum wall-clock time of a full TVD-RK stage (compute + comms + RK update + optional IBM correction), instead of a running mean of cpu_time measured only around s_compute_rhs. This better reflects end-to-end stage cost and is less sensitive to MPI host spin-wait and run-to-run jitter.

Changes:

• Move perf timing from s_compute_rhs to the RK stage loop in s_tvd_rk, bracketing the full stage.
• Switch from cpu_time running mean to system_clock wall-clock timing and take the minimum over steady-state stages.
• Simplify s_compute_rhs by removing the time_avg in/out timing parameter and internal timing logic.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/simulation/m_time_steppers.fpp	Adds per-stage wall-clock timing in s_tvd_rk and updates time_avg as the timing floor after warmup.
src/simulation/m_rhs.fpp	Removes RHS-local cpu_time timing and the time_avg argument from s_compute_rhs.

[codecov]

Codecov Report

❌ Patch coverage is 66.66667% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.56%. Comparing base (e090d56) to head (f3a068d).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
src/simulation/m_time_steppers.fpp	64.28%	1 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1650      +/-   ##
==========================================
- Coverage   60.95%   59.56%   -1.39%     
==========================================
  Files          83       83              
  Lines       20003    21115    +1112     
  Branches     2983     3129     +146     
==========================================
+ Hits        12193    12578     +385     
- Misses       5782     6434     +652     
- Partials     2028     2103      +75     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.