docs: add a cluster sizing guide built on hydration history

[maheshwarip]

Docs for hydration history!

[maheshwarip]

@antiguru , @aljoscha could I get your reviews on this one?

[def-]

QA LLM Review

1. MEDIUM -- Guide claims replica peaks correctly size a cluster hosting an upsert source, but the episode closes before the source peaks

doc/user/content/clusters/sizing.md:192

The guide tells readers that "a cluster hosting an upsert source is sized correctly by those peaks even though the source itself never appears per object." A replica episode is defined purely over compute exports, and its peaks are frozen at the moment the collector records it, so on the common create-source-plus-MV layout the row is written minutes into a snapshot that runs for far longer and the source's true peak is never captured. Acting on that number under-provisions the cluster.

Details

replica_collection_sql (src/adapter/src/coord/hydration_history.rs:434-441) builds objects from mz_compute_hydration_times_per_worker only, so a storage ingestion never holds an episode open. The resources CTE (:517-531) reads mz_cluster_replica_resource_usage at the sweep's read timestamp, i.e. whatever the high-water marks are when the sweep happens, and the final anti-join at :554-560 (NOT EXISTS (... h.finished_at >= c.started_at)) means the row is never revised once written. One replica is swept per hydration_history_collection_interval, so recording typically follows episode completion by minutes.

Concretely: CREATE SOURCE (upsert) plus an MV on it, on a fresh replica. The MV's dataflow is installed with an as-of at the source's then-current frontier, hydrates almost immediately against an empty shard, and closes the episode together with the introspection dataflows. The sweep records peak_memory_bytes / peak_disk_bytes from the first minutes of the process. The RocksDB snapshot then runs for half an hour and drives the real peak, and no later compute export means no later episode to capture it.

The reference page has the same overstatement in weaker form at doc/user/content/sql/system-catalog/mz_internal.md:745 ("Their memory and disk usage is still reflected in that table's peaks"). Suggest replacing both with the actual guarantee: the peaks cover the replica processes from start through the moment of collection, so they include storage work only if it happened before that moment, and a cluster whose peak is driven by source snapshotting is not sized by these numbers.

2. MEDIUM -- "The recorded value is never below the true hydration peak" is false for peak_disk_bytes

doc/user/content/clusters/sizing.md:204

The guide asserts that the recorded peaks are operating-system high-water marks and therefore "never below the true hydration peak," presenting them as a safe over-estimate for sizing. That holds for memory but not for disk: peak_disk_bytes is normally derived from statvfs, which has no kernel-side high-water mark and is folded in-process as a maximum over samples. A reader sizing scratch disk on it can under-provision, and this is exactly the column that matters for the upsert workloads the operational guidelines already flag as disk-heavy.

Details

src/metrics/src/usage.rs:71-74 is explicit: DERIVED_PEAKS are "Observations that get a peak folded in this process, because their source has no kernel-side high-water mark. Each is a maximum over samples and therefore a lower bound on the true peak," and (statvfs, fs_used) -> fs_used_peak is the first entry. replica_collection_sql (src/adapter/src/coord/hydration_history.rs:523-530) prefers exactly that value, falling back to the cgroup swap_peak only when it is absent, so on any replica with scratch disk in use the reported figure is the sampled one.

The relation's own column comment already says so ("Filesystem peaks are sampled lower bounds", src/catalog/src/builtin/mz_internal.rs), so the new guide contradicts the reference page it links to. Suggest scoping the bullet to memory and giving disk its own sentence noting the sampled lower bound.

3. LOW -- Both relations are gated on v26.42, but they shipped in v26.40 and v26.41

doc/user/content/sql/system-catalog/mz_internal.md:726

{{< warn-if-unreleased v26.42 >}} on both sections, and the guide's "This guide assumes you are running Materialize v26.42 or later. v26.42 included improvements to allow you to track peak resource usage during hydration" (doc/user/content/clusters/sizing.md:33), name a release that does not contain the work. The banner overstates the requirement for everyone already on a version that has these tables, and the prose sentence keeps doing so permanently for pinned self-managed deployments.

Details

mz_object_hydration_history merged 2026-08-26 (3a822d8457), between the bump to 26.40.0-dev.0 (e60d5e153d, 08-21) and the bump to 26.41.0-dev.0 (43944afc9c, 08-31), so it is in v26.40. mz_replica_hydration_history, which is where the peak columns live, merged 2026-09-02 (46f729653a) with main at 26.41.0-dev.0, so it is in v26.41. This PR's own first commit message stated that mapping before it was changed. Gating on v26.40 and v26.41 respectively also lets the banners clear on their own, whereas the v26.42 gate keeps them up for an extra release cycle.

[maheshwarip]

Verified all three against the source. Findings 1 and 2 were real and are fixed in 64c494c. Finding 3 is a deliberate call by the PR author, so I have left it.

1 — upsert source peaks (fixed). Confirmed: the resources CTE reads mz_cluster_replica_resource_usage at the sweep's read timestamp, and the closing anti-join (NOT EXISTS (... h.finished_at >= c.started_at)) means a row is never revised once written. Combined with episodes being defined over compute exports only, the peaks cover each process from start through the moment of collection and no further, so a snapshot that outlives that moment is never captured. My "sized correctly by those peaks" was wrong in exactly the direction that under-provisions. The note now states the real horizon, says a cluster whose peak is driven by snapshotting is not sized by these numbers, and points at mz_cluster_replica_metrics_history for that case. Same correction applied to the weaker wording on the reference page.

2 — peak_disk_bytes is a lower bound (fixed). Confirmed against DERIVED_PEAKS in src/metrics/src/usage.rs: (statvfs, fs_used) -> fs_used_peak is folded in-process as "a maximum over samples and therefore a lower bound on the true peak," and the collector prefers it whenever a scratch filesystem is present. The "never below the true hydration peak" claim now applies only to peak_memory_bytes, which does come from a kernel high-water mark; disk has its own bullet saying to leave extra headroom. Good catch that the guide was contradicting the column comment on the relation it links to.

3 — version gates. Not changed, deliberately. Your version mapping matches what I originally shipped and what I put on record in this thread: mz_object_hydration_history is in v26.40 and mz_replica_hydration_history in v26.41, neither in v26.39. The v26.42 gate and the prose sentence naming v26.42 were both requested explicitly by @maheshwarip after that evidence was posted, so this is the author's call rather than an oversight — presumably tied to when collection is actually enabled rather than when the relations appear. @maheshwarip, your call if you'd rather go back to per-release gates; the only mechanical wrinkle is that releases/v26.42.md was added for the shortcode to resolve and would no longer be needed.

---

Generated by Claude Code

[maheshwarip]

Human Pranshu here, I can confirm that this will only be announced as a part of v26.42

[maheshwarip]

buildkite/test/console-sql is failing on 7c22e60 (build #134171). I don't believe it belongs to this PR, and I can't re-run it myself — asking for a re-run below.

Why it isn't this PR's. The same job passed on 64c494c, which reported 81 statuses with a combined state of success. This head reports the same 81-status set. The only difference between those two trees is one trailing space removed from doc/user/content/clusters/sizing.md, so the job went pass → fail across a whitespace-only delta in a Markdown file.

Two supporting points:

• The step declares inputs: [console/**, test/console/**] (ci/test/pipeline.template.yml:761). This PR touches neither — the diff is doc/user/**, two files under src/catalog/src/builtin/, and the regenerated test/sqllogictest/autogenerated/mz_{internal,catalog}.slt.
• The entire non-docs change is two column-comment strings: mz_cluster_replica_sizes.memory_bytes ("billionths of a vCPU core" → "bytes") and mz_replica_hydration_history.object_count. No column, type, or value changes, so a Console SQL query cannot be affected. The Console does read memory_bytes (useSmallestClusterReplicaSize, clusterReplicaMetrics), but only as data.

No fix to port. The failure is in code this PR doesn't touch and I have no candidate fix for it.

What I need. This session has no Buildkite credentials (the API and web UI both return 401/403), so I can't read the job log or spend the one re-run the flake rules allow. Could someone with access re-run console-sql, or point me at the log if it's a real break? Note this blocks auto-merge while it's red. I also didn't try reproducing locally: mzcompose console run sql-tests needs a full environmentd build plus Kafka/MySQL/Postgres/SQL Server, which is a poor trade against the pass → fail evidence above.

I'll keep watching the PR until it's green.

---

Generated by Claude Code

[def-]

QA LLM Review

1. MEDIUM -- New step-3 sentence describes peak_memory as hydration-scoped, which the page's own interpretation section says it is not

doc/user/content/clusters/sizing.md:93

The sentence added at step 3, "As the name suggests, peak_memory measures peak memory usage during the hydration event", is not what the column holds: it is the cgroup process-lifetime high-water mark read at collection time, so it carries memory from before the episode and stops at the moment of recording. The bullets at :190 and :206 say exactly that, so the walkthrough now contradicts the interpretation section, and a reader who acts on step 3 alone sizes on a number that means something else.

Details

replica_collection_sql computes peak_memory_bytes as max(value) FILTER (WHERE source = 'cgroup' AND metric = 'memory_peak') over mz_cluster_replica_resource_usage (src/adapter/src/coord/hydration_history.rs:517-531, whose own comment calls these "Process-lifetime resource high-water marks"), and memory_peak is cgroup memory.peak (src/metrics/src/usage.rs:140), which nothing in the collector resets. Both directions of error follow, and the page already documents both: a spike from an unrelated peek, or from a previous episode on the same replica, lands in the next episode's peak_memory (:206-210), while an upsert source still snapshotting when the episode is recorded is not in it at all (:190-199).

Either drop the sentence, since the peak_memory_bytes bullet already owns the semantics, or state the horizon: peak_memory is the highest memory a process on the replica reached from process start through the moment the episode was recorded, which bounds the hydration peak from above rather than isolating it.

Secondary, in the same rewritten paragraph: :95 dropped "which reports per process" from the mz_cluster_replica_sizes pointer. memory_bytes there is the per-process allocation and peak_memory_bytes is likewise a per-process mark, so on a multi-process size the comparison only works if the reader knows not to multiply by processes.

[maheshwarip]

Both points fixed in 6dfade5.

Main finding. Confirmed independently: peak_memory_bytes is max(value) FILTER (WHERE source = 'cgroup' AND metric = 'memory_peak') over mz_cluster_replica_resource_usage at the sweep's read timestamp, memory_peak is cgroup memory.peak, and nothing in the collector resets it. So the value spans process start through the moment of recording, in both directions of error the page already documents. The step-3 sentence now states that horizon and keeps the part that matters for sizing — it bounds the hydration peak from above rather than isolating it — so the walkthrough no longer contradicts the interpretation section.

Secondary. The per-process note is back on the mz_cluster_replica_sizes pointer, with the reason spelled out: memory_bytes there and peak_memory_bytes here are both per-process, so a multi-process size needs no multiplication by processes.

Worth noting for anyone reading the history: this same overstatement was fixed once in 64c494c after your earlier review, then reintroduced by a later prose edit. The peak_memory_bytes bullet in "How should I interpret the hydration metrics?" is the owning statement for these semantics, so a future rewrite of step 3 should defer to it rather than restate it.

Separately, console-sql is red on this branch and I do not think it is this PR's — details in my earlier comment. That one still needs someone with Buildkite access.

---

Generated by Claude Code

[maheshwarip]

Withdrawing my re-run request: console-sql passed on 6dfade5 (build #134175), and the whole build is green at 81/81. The failure on 7c22e60 was a flake, so nobody needs to dig into it. Sorry for the noise.

---

Generated by Claude Code