benchmark: remeasure samples without marker-time RSS

[Pedro Henrique Penna (ppenna)]

Fixes #193.

Summary

The lifecycle benchmark samples OpenVMM's peak RSS once, at the restored guest's readiness marker. The guest then runs a prequeued nvx-exit 0, so OpenVMM can exit before the coordinator reads its counters.

This PR samples RSS only while OpenVMM is still running. If an attempt loses that race, the benchmark discards it and measures again instead of recording a substituted value. This replaces the post-launch fallback added by ca431cf (#192). That fallback avoided zeros, but the values it recorded come from before the restore.

Root cause

Instrumented runs on bare-metal Linux/KVM, Linux/MSHV, and Windows/WHP hosts showed:

• Linux: VmHWM disappears from /proc/<pid>/status once the unreaped process releases its address space. That happened within 0.18–4.6 ms of the observed marker on KVM and 1.4–1.8 ms on MSHV.
  • One natural KVM miss occurred when the coordinator dequeued the marker 4.5 ms after the reader thread queued it.
  • The original code recorded the failed read as 0. With a 150 ms observer delay, all ten restore samples were zero on both backends, and validate-openvmm failed as in CI.
• Windows: a late read succeeds, because the retained Popen handle keeps a terminated process's counters readable. It returns the lifetime peak working set, which includes teardown.

None of the other readings can stand in for the marker-time sample:

Candidate	Problem measured on bare metal
Post-launch sample (ca431cf)	Taken before the restore: 0.004–0.76 MiB, vs. about 36 MiB (KVM) and 58 MiB (MSHV) at the marker.
Linux wait4() ru_maxrss	Typically 0.1–1.8 MiB below the marker-time VmHWM. It also includes the coordinator image the child inherits before exec: a /bin/true child of a 200 MiB Python process reports 210 MiB.
Windows counters after exit	0.75–0.95 MiB (about 5%) above the marker-time value, because of teardown.

On KVM with ca431cf, I forced the race on 3 of 11 restores. The acceptance artifact then recorded these restore peak-RSS samples (MiB):

36.113, 0.211, 36.309, 36.262, 0.016, 36.027, 36.195, 0.094, 36.188, 36.227

validate-openvmm accepted the artifact (exit 0). MSHV likewise recorded 0.473, 0.160, and 0.277 MiB.

Changes

• live_peak_rss_bytes() returns the peak RSS only while OpenVMM is running, and None once it has exited.
  • Linux: it reads /proc/<pid>/status without polling, so the child is not reaped before the sample and its PID cannot be reused. A missing entry or VmHWM field means the address space is gone.
  • Windows: it accepts a counter read only if the process is still running after the read completes.
  • Errors: non-positive values and unexpected errors raise instead of being masked.
• measure_once() returns None for peak RSS when OpenVMM exited first. Teardown timing and the guest-exit status check are unchanged.
• benchmark() discards and repeats such an attempt, up to 3 attempts per sample.
  • Each attempt reruns before_each, and only the accepted attempt's lifecycle profile is kept.
  • The new peak_rss_remeasured_count field records discarded attempts.
  • Three consecutive misses fail the benchmark without writing a result.
  • Warmups report peak RSS=unavailable.
• Restore-memory workload: it does not prequeue a guest exit, so a missing marker-time RSS is an explicit error.
• Performance collection: positive-value validation in performance.py is unchanged.
• Tests: the two ca431cf tests are replaced by coverage for:
  • a process exiting before the marker-time read, with guest exit status 0 and 1;
  • Linux zombie and missing /proc entries;
  • a Windows process exiting before and during the counter read;
  • remeasurement and exhaustion in benchmark();
  • collection of a remeasured artifact, and rejection of a zero sample.
• Docs: doc/benchmarks.md documents the sampling contract.

Validation

Local (Windows): ruff check, ruff format --check, strict pyright for Linux and Windows (0 errors), the validate-nvx unit suites (328 tests), test_adversarial.py, and test_hosts.py.

Bare-metal runs used CI's lifecycle flags:

--suite e2e --processors 1 --cpus 0-7 --host-cpu-reserve 0 --warmups 1 --runs 10 --timeout 40 --teardown-mode guest-exit --snapshot-profile

To force the race, a runtime-only wrapper delayed selected samples by 150 ms. The wrapper is not part of this PR.

Final tree on f0d103c, with release v0.1.0-dev.31a5478d4e9f (OpenVMM 9073d248):

Check	linux-kvm-baremetal	linux-mshv-baremetal	windows-whp-baremetal
Unit suites	328 pass	328 pass	328 pass
Race forced on every 4th restore attempt	3 remeasured; RSS 35.2–36.3 MiB; validate exit 0	3 remeasured; RSS 55.6–57.6 MiB; validate exit 0	3 remeasured; RSS 18.7–18.8 MiB; validate exit 0
CI-equivalent --suite e2e	0 remeasured; validate exit 0	0 remeasured; validate exit 0	0 remeasured; validate exit 0

Earlier iterations of this patch ran on d2f7947 with release v0.1.0-dev.d2f79471ad83 (OpenVMM dd529ed0). After review, the Windows sampler was tightened to also reject an exit during the read.

• Race forced on every attempt: on all three platforms, the benchmark failed with OpenVMM exited before its peak RSS was sampled at the marker in 3 consecutive attempts for sample 1/1 and wrote no result.
• Natural restore runs with OpenVMM logging enabled: no misses in 143 KVM, 121 MSHV, and 66 WHP marker-time samples.
• CI-equivalent e2e: 3 KVM, 3 MSHV, and 4 WHP runs recorded only positive samples and needed no remeasurement.
• Stability gate: the MSHV snapshot-generation stability gate (exit 75, unrelated to RSS) rejected one CI-equivalent run and one forced-race run. It also rejected 1 of 3 runs of the unmodified code on that host.

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation satisfies #193’s acceptance criteria with focused cross-platform regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes #193 by remeasuring lifecycle samples when OpenVMM exits before marker-time peak RSS collection.

Changes:

• Adds live-process RSS sampling and bounded retries.
• Records discarded attempts while preserving accepted lifecycle profiles.
• Adds cross-platform regression tests and documents sampling semantics.

File	Description
scripts/​nvx_tools/​benchmark.py	Implements live RSS sampling and remeasurement.
scripts/​test_nvx_tools.py	Tests exit races, retries, and exhaustion.
scripts/​test_performance.py	Verifies collection and zero-value rejection.
doc/​benchmarks.md	Documents marker-time RSS behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.