benchmarking/locust: capture cluster hardware facts and density frontiers. by Nishanth29 · Pull Request #1603 · agent-substrate/substrate

Nishanth Kotla (Nishanth29) · 2026-09-10T23:07:18Z

What this PR does

In alignment with the Actor Density Benchmark Specs, this PR teaches the Locust runner to discover cluster capacity, record actor density frontiers, and harvest server-side Prometheus metrics into stats.jsonl and server_summary.json. Discovery lives in its own module, benchmarking/locust/cluster_facts.py, rather than inside runner.py.

Proposed Changes

Cluster hardware discovery (cluster_facts.py, locust.yaml)

Reads allocatable cores, RAM, node count, machine type and worker pod count through the official Kubernetes Python client, so in-cluster and local auth both work without a kubectl subprocess.
Adds a ClusterRole with list on nodes and pods. --no-cluster-facts skips discovery, since listing is expensive on a large cluster.
Anything unreadable stays null. Nothing is guessed or defaulted, so a real reading is always distinguishable from a missing one.

Density frontiers (trial_summary in stats.jsonl)

actors_per_node, actors_per_vcpu, actors_per_gb_ram.
Actors per pod as ap_ratio_p50 / p90 / p99 over the steady-state samples rather than one average, so it is visible whether the system actually pegs at 1.
Raw readings persist beside the derived ones in raw_configuration, so ratios can be re-derived after the fact.

Server ground truth (server_telemetry.py, server_summary.json)

Physical worker assignment from ate_workerpool_workers, as packing percentiles plus the underlying timeseries.
Host Linux kernel PSI stalls for CPU, memory and IO, and CFS throttling.
Snapshot sizes as P50, P90, P95 and mean, checkpoint counts, restore and checkpoint latencies, and throughput. Counts, mean and throughput cover the steady-state window; the quantiles are a 5m rate at window end.
server_telemetry.py uses only the standard library, every call with a timeout. An unreachable Prometheus leaves nulls rather than failing the run, and --prometheus-url retargets it. status.json keeps its existing minimal schema.
Depends on benchmarking: expand cAdvisor scrape allowlist for kernel PSI, networ… #1599 and benchmarking: expose raw atelet and ate metrics via Prometheus export… #1601 landing first: the PSI queries need the container="node" relabel and the snapshot queries need the metrics/raw pipeline. Merged ahead of those, the PSI and snapshot fields come back null.

Docs

benchmarking/README.md covers the new flags and every field in the output files.

How this was tested

15 unit tests across test_cluster_facts.py and test_server_telemetry.py, covering percentile edges, steady-state detection, counter resets across an atelet restart, and the null-versus-zero rules.
Multi-user runs on a benchmark cluster. Frontier fields land in stats.jsonl, server_summary.json is populated, and status.json is unchanged.
Emitted values were re-derived by hand from raw Prometheus and matched.

References

Agent Substrate: Actor Density Benchmark Specs

Tests pass
Appropriate changes to documentation are included in the PR

…iers in runner

Max Smythe (maxsmythe)

Thanks for this! Left a few initial comments.

Max Smythe (maxsmythe) · 2026-09-11T10:05:38Z

    if with_boomer:
-        boomer_cmd = [BOOMER_BINARY, "--user-class", Path(test_file(args.file)).stem]
+        boomer_cmd = [BOOMER_BINARY]
+        if "boomer-glutton" not in os.path.basename(BOOMER_BINARY):


Is all this switching to avoid breaking changes? Shouldn't the binary be bundled in the container? May be able to just straight rename the binary.

yeah, my benchmark cluster was still running an older image from before the #1295 rename, so I needed
these to get my trials running against it. None of it applies to a normal run since the Dockerfile
bundles /app/boomer-worker. Reverting to that and dropping the env knobs.

Max Smythe (maxsmythe) · 2026-09-11T10:07:11Z

+    return int(float(mem_str))
+
+
+def get_cluster_hardware_facts(logs: TextIO | None = None) -> dict[str, any]:


I think this batch of code is large enough and focused enough to warrant its own file.

will move it into its own file

Max Smythe (maxsmythe) · 2026-09-11T10:09:29Z

+        "allocatable_ram_gb": None,
+        "worker_pod_count": None,
+    }
+    # 1. Environment variables override


Why do we need overrides?

I added them as a fallback in case the runner couldn't read the cluster, but this PR adds the RBAC for
that anyway so they aren't really needed. Removing them.

Max Smythe (maxsmythe) · 2026-09-11T10:10:43Z

+    ):
+        return facts
+
+    # 2. In-cluster HTTP API or local kubectl discovery


I think K8s has a Python client we can use for this?

oh yeah... I went with urllib because the locust image is distroless and I was trying to avoid
adding a dependency, but we already install pip packages in there anyway, so that isn't really a
concern. will be switching to the client.

Max Smythe (maxsmythe) · 2026-09-11T10:11:45Z

+                    except Exception:
+                        pass
+        elif shutil.which("kubectl"):
+            # Local workstation fallback via kubectl


Why do we need this? Also I think we may get standard auth for free w/a client

that was just so I could run runner.py from my workstation, but you're right that the client handles
that for us. Dropping the kubectl path.

Max Smythe (maxsmythe) · 2026-09-11T10:13:38Z

                else:
                    stats_generated = jsonl_path.exists()
+                    if stats_generated:
+                        facts = get_cluster_hardware_facts(logs)


for truly large clusters, we may want the ability to disable this (hitting every node may take a long time)

good call, will be adding a --cluster-facts / --no-cluster-facts flag (default on) so it can be skipped
entirely.

Nishanth Kotla (Nishanth29) · 2026-09-12T00:13:07Z

Thanks Max, good catches. All six are fixed: reverted the boomer switching,
moved discovery into cluster_facts.py, dropped the env overrides and the
kubectl fallback, switched to the K8s client, and added --cluster-facts /
--no-cluster-facts so the listing can be skipped on big clusters.

Also fixed some telemetry bugs, rewrote the tests and documented the flags and output fields in the README.

Nishanth Kotla (Nishanth29) added 2 commits September 10, 2026 16:50

benchmarking/locust: capture cluster hardware facts and density front…

8c11e38

…iers in runner

benchmarking/locust: add server-side telemetry harvesting

dc4ddaf

Max Smythe (maxsmythe) requested changes Sep 11, 2026

View reviewed changes

benchmarking/locust: address review feedback

aa26f04

benchmarking/locust: fix telemetry metrics, add tests and docs

7cc81ab

Nishanth Kotla (Nishanth29) changed the title ~~benchmarking/locust: capture cluster hardware facts and density frontiers in runner~~ benchmarking/locust: capture cluster hardware facts and density frontiers. Sep 14, 2026

Nishanth Kotla (Nishanth29) requested a review from Max Smythe (maxsmythe) September 14, 2026 23:28

		return int(float(mem_str))


		def get_cluster_hardware_facts(logs: TextIO \| None = None) -> dict[str, any]:

Uh oh!

Conversation

Nishanth Kotla (Nishanth29) commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Proposed Changes

How this was tested

References

Uh oh!

Max Smythe (maxsmythe) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nishanth Kotla (Nishanth29) commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Nishanth Kotla (Nishanth29) commented Sep 10, 2026 •

edited

Loading

Nishanth Kotla (Nishanth29) commented Sep 12, 2026 •

edited

Loading