From d0d8520dfa44bedd0d54ca321cc53575bb69ca49 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 18:17:33 +0000 Subject: [PATCH 1/9] docs: add a cluster sizing guide built on hydration history Sizing a cluster is bounded by hydration, not steady state, and hydration cost is data-dependent enough that it cannot be read off a query plan. The durable hydration history relations now make the previous hydration measurable after the replica that ran it is gone, so the guidance can be "start large, measure, size down" rather than an estimate. Adds `/clusters/sizing/`, which opens with the existing cluster lifecycle content and then walks the procedure: create the cluster at a generous size, wait for hydration, read the episode's duration and resource peaks from `mz_replica_hydration_history` and `mz_object_hydration_history`, pick the smallest size that keeps the peak under a headroom threshold, then resize and confirm against the new episode. Includes what the recorded numbers do and do not mean, and the fallbacks when collection is disabled. The lifecycle section moves to `content/headless/cluster-lifecycle.md` so the concepts page and the sizing guide share one copy, with a pointer added to the durable history alongside the current-state relations. Reference updates: * Gate both hydration history sections on the releases that introduce them (`mz_object_hydration_history` in v26.40, `mz_replica_hydration_history` in v26.41), and document the joins a sizing query needs. * `object_count` counts every maintained compute dataflow in an episode, including the replica's system introspection dataflows, so it exceeds the number of objects a user created and does not match the row count in per-object history. Say so on the column. * `mz_cluster_replica_sizes.memory_bytes` described its unit as billionths of a vCPU core, copied from `cpu_nano_cores`. It is bytes. Every SQL example was run against the emulator. Resource peaks read `NULL` there, since they come from cgroup v2 files that a cgroup v1 host does not expose, so the peak columns in the sample output are illustrative. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E6qZnfQsyVbQ2hydYaYder --- doc/user/content/clusters/_index.md | 1 + .../clusters/operational-guidelines/_index.md | 6 +- doc/user/content/clusters/sizing.md | 387 ++++++++++++++++++ .../content/fundamentals/concepts/clusters.md | 145 +------ .../fundamentals/concepts/hydration.md | 10 + .../content/headless/cluster-lifecycle.md | 149 +++++++ .../content/sql/system-catalog/mz_catalog.md | 2 +- .../content/sql/system-catalog/mz_internal.md | 21 +- src/catalog/src/builtin/mz_catalog.rs | 5 +- src/catalog/src/builtin/mz_internal.rs | 2 +- .../sqllogictest/autogenerated/mz_catalog.slt | 2 +- .../autogenerated/mz_internal.slt | 2 +- 12 files changed, 583 insertions(+), 149 deletions(-) create mode 100644 doc/user/content/clusters/sizing.md create mode 100644 doc/user/content/headless/cluster-lifecycle.md diff --git a/doc/user/content/clusters/_index.md b/doc/user/content/clusters/_index.md index ed91451c25d4b..b67b74b42f3ad 100644 --- a/doc/user/content/clusters/_index.md +++ b/doc/user/content/clusters/_index.md @@ -12,4 +12,5 @@ Clusters provide the compute resources for running dataflows in Materialize. - Learn about [clusters](/fundamentals/concepts/clusters/). - Follow the [operational guidelines](/clusters/operational-guidelines/). +- Choose a [cluster size](/clusters/sizing/). - Understand [system clusters](/clusters/system-clusters/). diff --git a/doc/user/content/clusters/operational-guidelines/_index.md b/doc/user/content/clusters/operational-guidelines/_index.md index d514f6c95f296..00a84d1df84c9 100644 --- a/doc/user/content/clusters/operational-guidelines/_index.md +++ b/doc/user/content/clusters/operational-guidelines/_index.md @@ -66,7 +66,11 @@ For upsert sources, snapshotting is a resource-intensive operation that can requ ## Hydration considerations When sizing a cluster, budget for hydration memory on top of the steady-state -cost. The table below summarizes, per object type, when each object hydrates and +cost. Rather than estimating that budget, start at a size that hydrates +comfortably and size down once you have measured what hydration needed: see +[Cluster sizing](/clusters/sizing/). + +The table below summarizes, per object type, when each object hydrates and the memory it uses. For more on hydration, including strategies to reduce its impact, see [Hydration](/fundamentals/concepts/hydration/). diff --git a/doc/user/content/clusters/sizing.md b/doc/user/content/clusters/sizing.md new file mode 100644 index 0000000000000..e1efdebc3017f --- /dev/null +++ b/doc/user/content/clusters/sizing.md @@ -0,0 +1,387 @@ +--- +title: "Cluster sizing" +description: "Pick a cluster size by measuring what hydration actually needed, then sizing down." +menu: + main: + parent: "clusters" + weight: 5 + name: "Cluster sizing" + identifier: "cluster-sizing" +--- + +A cluster's [size](/sql/create-cluster/#available-sizes) fixes the CPU, memory, +and scratch disk available to every replica of that cluster, and on Materialize +Cloud it fixes the [cost](/materialize-cloud/billing/#compute). The size you +need is set by the most expensive thing the cluster does, and for most clusters +that is [hydration](/fundamentals/concepts/hydration/) rather than steady state. + +Hydration is also the part you cannot predict from the query text. How much +memory a join or an aggregation needs depends on the data: key distribution, +skew, and how much history the inputs carry. So rather than estimate, start at a +size that hydrates comfortably, measure what hydration needed, and then size +down. + +{{% include-headless "/headless/cluster-lifecycle" %}} + +## Why hydration sets the size + +Steady state is the cheap part of a cluster's life. Once a dataflow is hydrated +it holds its arrangements and applies incoming updates, and its memory tracks +the size of the state it maintains. Hydration is different: the replica rebuilds +that state from the storage layer, which means reading the inputs and building +the intermediate arrangements that produce it. Peak memory during hydration is +therefore higher than steady-state memory, often around twice as high, and the +same holds for the time it takes. + +That gap decides two things at once: + +- **A cluster that cannot hydrate serves nothing.** A replica that exceeds its + memory allocation is restarted, and it then attempts the same hydration again. + An undersized cluster does not degrade gracefully into a slow cluster: it + restarts in a loop and never reaches the point where it can answer queries. + +- **A cluster sized for steady state may not survive a restart.** The size that + holds a hydrated dataflow can be too small to rebuild it. Restarts are not + exceptional (a resize, a version upgrade, or a new index all trigger + hydration), so the size has to cover the rebuild, not just the result. + +Both point the same way: choose a size that hydrates, then reduce it with +evidence. + +## Start large, then size down + +The procedure below oversizes the cluster deliberately for one hydration, reads +what that hydration needed from the catalog, and uses those numbers to pick a +steady-state size. + +Steps 3 through 5 read the durable hydration history relations: + +{{< warn-if-unreleased v26.41 >}} + +### 1. Create the cluster at a generous size + +Pick a size you are confident can hydrate the workload, even if it is clearly +more than steady state needs. Oversizing costs money for as long as the cluster +runs at that size. Undersizing costs a hydration that never completes. + +```mzsql +CREATE CLUSTER analytics (SIZE = '400cc'); +``` + +Then create the cluster's indexes and materialized views as usual. + +### 2. Wait for the cluster to hydrate + +Every object has to finish hydrating before the numbers describe the whole +workload. Check that nothing is still hydrating: + +```mzsql +SELECT o.name AS object, h.replica_id, h.hydrated +FROM mz_internal.mz_hydration_statuses AS h +JOIN mz_catalog.mz_objects AS o ON o.id = h.object_id +JOIN mz_catalog.mz_clusters AS c ON c.id = o.cluster_id +WHERE c.name = 'analytics' AND h.hydrated IS NOT TRUE; +``` + +An empty result means every object on the cluster is hydrated. A row with a +`NULL` `replica_id` is an object that has not attached to a replica yet, which +`IS NOT TRUE` catches along with `hydrated = false`. See [Lifecycle of a +cluster](#lifecycle-of-a-cluster) for the states that follow. + + + +### 3. Read what the last hydration needed + +Materialize records completed hydration episodes durably, so the numbers survive +the replica restart or resize that produced them. +[`mz_internal.mz_replica_hydration_history`](/sql/system-catalog/mz_internal/#mz_replica_hydration_history) +holds one row per replica-wide episode, with the resource high-water marks +observed for it: + +```mzsql +SELECT + rh.replica_name AS replica, + rh.size, + h.started_at, + h.finished_at - h.started_at AS hydration_time, + h.object_count, + pg_size_pretty(h.peak_memory_bytes) AS peak_memory, + pg_size_pretty(h.peak_disk_bytes) AS peak_disk +FROM mz_internal.mz_replica_hydration_history AS h +JOIN mz_internal.mz_cluster_replica_history AS rh ON rh.replica_id = h.replica_id +WHERE rh.cluster_name = 'analytics' +ORDER BY h.started_at DESC; +``` + +```none + replica | size | started_at | hydration_time | object_count | peak_memory | peak_disk +---------+-------+-------------------------------+----------------+--------------+-------------+----------- + r1 | 400cc | 2026-09-08 09:12:04.117841+00 | 00:04:11.83 | 41 | 11 GB | 2438 MB +(1 row) +``` + +The join to +[`mz_internal.mz_cluster_replica_history`](/sql/system-catalog/mz_internal/#mz_cluster_replica_history) +is what makes the numbers usable for sizing: it supplies the size the episode +ran at, and it keeps that row after the replica is gone. Hydration history +itself stores only the replica ID, and a resize replaces the replica, so joining +[`mz_cluster_replicas`](/sql/system-catalog/mz_catalog/#mz_cluster_replicas) +instead would drop exactly the episodes you want to compare against. + +Two columns need reading with care: + +- `object_count` counts every maintained dataflow in the episode, which includes + the system introspection dataflows each replica runs. It is normally a few + dozen higher than the number of objects you created, and it is not the number + of rows the per-object table holds for that replica. + +- `peak_memory_bytes` and `peak_disk_bytes` are the largest values reported by + any single process of the replica, not the sum across processes. Memory and + disk limits apply per process, so the maximum is what answers whether any + process came close to its limit. See [Reading the recorded + numbers](#reading-the-recorded-numbers) for what the peaks do and do not + cover. + +To find which object dominated the episode, read the per-object table, +[`mz_internal.mz_object_hydration_history`](/sql/system-catalog/mz_internal/#mz_object_hydration_history). +Its `object_id` is the ID of the object's dataflow, so reach the catalog item +through +[`mz_internal.mz_object_global_ids`](/sql/system-catalog/mz_internal/#mz_object_global_ids): + +```mzsql +SELECT + rh.replica_name AS replica, + o.name AS object, + o.type, + h.hydrated_at - h.installed_at AS hydration_time +FROM mz_internal.mz_object_hydration_history AS h +JOIN mz_internal.mz_object_global_ids AS g ON g.global_id = h.object_id +JOIN mz_catalog.mz_objects AS o ON o.id = g.id +JOIN mz_internal.mz_cluster_replica_history AS rh ON rh.replica_id = h.replica_id +WHERE rh.cluster_name = 'analytics' +ORDER BY hydration_time DESC +LIMIT 5; +``` + +```none + replica | object | type | hydration_time +---------+---------------------+-------------------+----------------- + r1 | auction_summary | materialized-view | 00:04:11.83 + r1 | bids_by_auction | materialized-view | 00:01:47.21 + r1 | bids_by_auction_idx | index | 00:00:22.04 + r1 | auction_summary_idx | index | 00:00:19.88 +(4 rows) +``` + +Every replica records its own rows, so a cluster with a replication factor +above one, or one that has been resized, returns a row per object per replica. +The per-object table carries no resource columns, because peaks are measured per +process and a process runs many dataflows at once. Use it to find the object +whose hydration dominates the episode, then attribute the episode's peak to that +object's cluster placement. If one object accounts for most of the episode, [move +it to its own +cluster](/fundamentals/concepts/hydration/#hydration-strategies) so its +hydration peak stops dictating the size of everything else. + +### 4. Choose a steady-state size + +The episode's peak memory is what the smaller size has to fit, with headroom for +data growth. The following query reports, per cluster, the largest peak still in +the history and the smallest size whose per-process memory keeps that peak under +75%: + +```mzsql +WITH observed AS ( + SELECT + rh.cluster_name AS cluster, + max(h.peak_memory_bytes) AS peak_memory_bytes + FROM mz_internal.mz_replica_hydration_history AS h + JOIN mz_internal.mz_cluster_replica_history AS rh ON rh.replica_id = h.replica_id + WHERE h.peak_memory_bytes IS NOT NULL + GROUP BY rh.cluster_name +) +SELECT + o.cluster, + pg_size_pretty(o.peak_memory_bytes) AS peak_hydration_memory, + ( + SELECT s.size + FROM mz_catalog.mz_cluster_replica_sizes AS s + WHERE o.peak_memory_bytes <= s.memory_bytes * 0.75 + ORDER BY s.memory_bytes + LIMIT 1 + ) AS smallest_size_with_headroom +FROM observed AS o +ORDER BY o.cluster; +``` + +```none + cluster | peak_hydration_memory | smallest_size_with_headroom +-----------+-----------------------+----------------------------- + analytics | 11 GB | 100cc +(1 row) +``` + +The 75% in that query is a starting point, not a guarantee. Raise the headroom +when the inputs are growing, when the workload is seasonal, or when the cluster +also serves ad-hoc `SELECT` queries, since those compete for the same memory and +are not part of a hydration episode. + +Treat the result as the next size to try rather than the final answer. Sizing +down changes the thing you measured: fewer workers per replica changes how the +work is distributed, so the peak at `100cc` is not the peak at `400cc` divided +by four. Step down one size at a time and re-measure after each step. + +{{< tip >}} +If a cluster's peak is dominated by hydration and its steady state is much +cheaper, you can keep it small and let it borrow capacity only while it +hydrates. An [`AUTO SCALING STRATEGY (ON +HYDRATION)`](/sql/alter-cluster/#speed-up-hydration-by-autoscaling-to-a-larger-size) +provisions an extra burst replica at a larger size whenever the cluster has +un-hydrated objects, including after a restart or an upgrade, and removes it once +a steady-size replica catches up. You then pay the hydration size only for the +duration of hydration. +{{< /tip >}} + +### 5. Size down and confirm + +A resize is graceful by default: Materialize hydrates replicas at the new size +alongside the current ones before retiring them, and rolls the resize back if +they do not hydrate within the reconfiguration timeout. See [resizing +process](/sql/alter-cluster/#resizing-process) for the details and how to change +that behavior. + +```mzsql +ALTER CLUSTER analytics SET (SIZE = '100cc'); +``` + +That rollback is what makes stepping down safe to try: a size that cannot +rebuild the state leaves the cluster where it was rather than serving nothing. + +The resize also hydrates the whole workload again, which produces exactly the +measurement you need to confirm the new size. Re-run the query from [step +3](#read-what-the-last-hydration-needed) once the new replica is hydrated: + +```none + replica | size | started_at | hydration_time | object_count | peak_memory | peak_disk +---------+-------+-------------------------------+----------------+--------------+-------------+----------- + r2 | 100cc | 2026-09-08 10:41:22.913044+00 | 00:12:37.42 | 41 | 12 GB | 4310 MB + r1 | 400cc | 2026-09-08 09:12:04.117841+00 | 00:04:11.83 | 41 | 11 GB | 2438 MB +(2 rows) +``` + +This is the outcome to look for, and it is also the point of measuring rather +than estimating. Peak memory barely moved, so `100cc` holds the workload with +the headroom the previous step asked for. Hydration got three times slower, which +is the cost of the smaller size, and whether that matters depends on how long +you can tolerate a restart taking. + +If the new size is too small, no completed episode is recorded for the new +replica at all. Only successful hydration is recorded, so an out-of-memory +restart loop shows up as a missing row plus repeated restarts in +[`mz_internal.mz_cluster_replica_status_history`](/sql/system-catalog/mz_internal/#mz_cluster_replica_status_history): + +```mzsql +SELECT sh.occurred_at, sh.process_id, sh.status, sh.reason +FROM mz_internal.mz_cluster_replica_status_history AS sh +JOIN mz_internal.mz_cluster_replica_history AS rh ON rh.replica_id = sh.replica_id +WHERE rh.cluster_name = 'analytics' +ORDER BY sh.occurred_at DESC +LIMIT 10; +``` + +Repeated `offline` rows with an out-of-memory `reason`, and no new episode in +hydration history, mean the size cannot rebuild the state. Go back to the size +that hydrated, and take a smaller step, or reduce the peak itself with one of +the [hydration +strategies](/fundamentals/concepts/hydration/#hydration-strategies). + +## Reading the recorded numbers + +Hydration history is a best-effort record, not an audit log. Where it is +approximate, it is approximate in ways that matter for sizing: + +- **Only successful episodes are recorded.** There is no row for a hydration + that was killed, canceled, or is still running, and `status` is currently + always `hydrated`. A missing row is a signal in its own right, as in step 5, + but it is never a measurement of a failure. + +- **Short-lived objects can be missed entirely.** Recording works by sampling + each replica in a rotation, so an object that is dropped before its replica's + turn leaves no trace. Nothing incorrect is recorded, the episode is simply + absent. + +- **The peaks are upper bounds on the episode.** They come from operating-system + high-water marks that cover each process's whole lifetime up to the moment the + episode is recorded, so post-hydration work can raise them, and a later + episode can inherit an earlier episode's mark. For sizing this errs the safe + way: the recorded value is never below the true hydration peak. + +- **A peak can be `NULL`.** The values depend on what the platform exposes + (a cgroup memory peak, and a scratch filesystem or swap peak), so they are + absent rather than zero when a deployment does not report them. + +- **Timestamps can carry clock skew.** On a multi-process replica the endpoints + of an interval come from different process clocks, so a recorded duration + includes their skew. This is not usually visible at the minute scale that + matters for sizing. + +- **Rows outlive what they name.** `replica_id`, `cluster_id`, and `object_id` + may all name objects that no longer exist, which is what makes the history + useful across a resize. Join `mz_cluster_replica_history` for replica and + cluster names, and expect the join through `mz_object_global_ids` to drop + objects that have since been dropped. + +- **Rows are retained for 30 days by default.** Sizing decisions should come + from the recent history rather than the earliest episode still stored. + +Both tables live in the [`mz_internal`](/sql/system-catalog/mz_internal/) +schema, which is not part of Materialize's stable interface. + +## If hydration history is empty + +Recording is controlled by the `hydration_history_collection_interval` system +parameter, which sets how often Materialize samples replicas for completed +episodes. A value of zero disables recording, and the tables then stay as they +are: rows already collected remain, and no new ones are added. +`hydration_history_retention_period` bounds how long rows live, and defaults to +30 days. + +On Materialize Cloud, these parameters are managed for you. If both tables are +empty for a cluster that has certainly hydrated, contact +[support](/support/). + +On Materialize Self-Managed, set them as the `mz_system` user, or through the +[system parameters +ConfigMap](/self-managed-deployments/configuration-system-parameters/): + +```mzsql +ALTER SYSTEM SET hydration_history_collection_interval = '60s'; +``` + +A shorter interval records episodes sooner, at the cost of installing a dataflow +on a replica more often. Recording visits one replica per interval, so an +environment with many replicas revisits each one proportionally less often. + +Until history is available, the current-state relations still answer the +narrower question of what is happening now: + +| Relation | What it gives you | +|----------|-------------------| +| [`mz_internal.mz_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_hydration_statuses) | Per-object, per-replica hydration flag, for every object type. | +| [`mz_internal.mz_compute_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_compute_hydration_statuses) | The same flag plus how long hydration took, for indexes and materialized views. | +| [`mz_internal.mz_cluster_replica_metrics_history`](/sql/system-catalog/mz_internal/#mz_cluster_replica_metrics_history) | CPU, memory, and disk sampled about once a minute, retained for 30 days. | + +The two hydration relations report only the current state and are reset by a +replica or Materialize restart. The metrics history survives restarts, but at +roughly one sample a minute it can miss a hydration spike entirely, and it does +not tell you which episode a sample belonged to. That is why these are a +fallback rather than the basis for a sizing decision. + +## Related pages + +- [Hydration](/fundamentals/concepts/hydration/) +- [Clusters](/fundamentals/concepts/clusters/) +- [Operational guidelines](/clusters/operational-guidelines/) +- [`CREATE CLUSTER`](/sql/create-cluster/) +- [`ALTER CLUSTER`](/sql/alter-cluster/) +- [Usage & billing](/materialize-cloud/billing/) diff --git a/doc/user/content/fundamentals/concepts/clusters.md b/doc/user/content/fundamentals/concepts/clusters.md index 762f4ef56e0cc..e12998029e4d7 100644 --- a/doc/user/content/fundamentals/concepts/clusters.md +++ b/doc/user/content/fundamentals/concepts/clusters.md @@ -104,145 +104,7 @@ When provisioning replicas, See also [Hydration considerations](#hydration-considerations). -## Lifecycle of a cluster - -Whenever a cluster starts running a workload (after you create it, resize it, -or one of its replicas restarts), its replicas move through a sequence of states -before results are fully up to date. Knowing which state a cluster is in tells -you whether it is making progress or is stuck. - -The queries below monitor a cluster named `lifecycle_demo` that hosts the -materialized view `bids_by_auction` and its index `bids_by_auction_idx`, both -built on a continuously-updating `AUCTION` load-generator source. Substitute -your own cluster and object names. - -### Provisioning - -Replicas are scheduled and brought online. A cluster with a [replication -factor](#cluster-replicas) of `0` has no compute and never leaves this state. To -monitor progress, check that replicas report `online` in -[`mz_cluster_replica_statuses`](/sql/system-catalog/mz_internal/#mz_cluster_replica_statuses), -and confirm the cluster has replicas via -[`mz_clusters`](/sql/system-catalog/mz_catalog/#mz_clusters). - -```mzsql -SELECT c.name AS cluster, r.name AS replica, r.size, st.status, st.reason -FROM mz_internal.mz_cluster_replica_statuses st -JOIN mz_catalog.mz_cluster_replicas r ON r.id = st.replica_id -JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id -WHERE c.name = 'lifecycle_demo' -ORDER BY r.name; -``` - -```none - cluster | replica | size | status | reason -----------------+---------+------+--------+-------- - lifecycle_demo | r1 | 25cc | online | -(1 row) -``` - -The `reason` column is empty while the replica is `online`, and reports why a -replica is unavailable otherwise. - -### Hydrating - -Each replica reconstructs its in-memory state by reading from Materialize's -storage layer (see [hydration](/fundamentals/concepts/hydration/)). While an object is -hydrating, its `hydrated` flag reads `f` and its lag is reported as `NULL`. To -monitor progress, check the `hydrated` flag per object in -[`mz_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_hydration_statuses), -where the `replica_id` stays blank until the object attaches to a replica. For -indexes and materialized views, -[`mz_compute_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_compute_hydration_statuses) -also reports how long hydration took. - -```mzsql -SELECT o.name AS object, o.type, r.name AS replica, ch.hydrated, ch.hydration_time -FROM mz_internal.mz_compute_hydration_statuses ch -JOIN mz_objects o ON o.id = ch.object_id -JOIN mz_catalog.mz_cluster_replicas r ON r.id = ch.replica_id -WHERE o.name IN ('bids_by_auction', 'bids_by_auction_idx', 'bids_load') -ORDER BY o.name; -``` - -```none - object | type | replica | hydrated | hydration_time ----------------------+-------------------+---------+----------+----------------- - bids_by_auction | materialized-view | r1 | t | 00:00:00.000074 - bids_by_auction_idx | index | r1 | t | 00:00:00.000019 - bids_load | materialized-view | r1 | t | 00:00:05.6032 -(3 rows) -``` - -The light view and index hydrate in microseconds, while the larger `bids_load` -view takes about 5.6 seconds. A larger object with more state to reconstruct -shows a longer, more visible hydration window. - -### Catching up - -Once hydrated, the cluster processes the backlog of input updates that -accumulated while it was unavailable, so its total lag starts high and comes -down. To monitor progress, watch `lag` decrease in -[`mz_wallclock_global_lag_recent_history`](/sql/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history), -or break the lag down by input with -[`mz_materialization_lag`](/sql/system-catalog/mz_internal/#mz_materialization_lag). - -```mzsql -SELECT o.name AS object, l.local_lag, l.global_lag, - si.name AS slowest_local_input, sg.name AS slowest_global_input -FROM mz_internal.mz_materialization_lag l -JOIN mz_objects o ON o.id = l.object_id -LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id -LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id -WHERE o.name IN ('bids_by_auction', 'bids_load') -ORDER BY o.name; -``` - -```none - object | local_lag | global_lag | slowest_local_input | slowest_global_input ------------------+------------------+------------------+---------------------+---------------------- - bids_by_auction | 00:00:34.001 | 00:00:34.001 | bids | bids - bids_load | 00:00:41.001 | 00:00:41.001 | bids | bids -``` - -Both objects trail their slowest input, the `bids` source, by tens of seconds. -As the cluster works through the backlog, these lags fall. - -### Steady state - -The cluster has caught up and its lag holds low and roughly constant, typically -a few seconds. Re-running the lag query confirms the objects have caught up to -their input. - -```mzsql -SELECT o.name AS object, l.local_lag, l.global_lag, - si.name AS slowest_local_input, sg.name AS slowest_global_input -FROM mz_internal.mz_materialization_lag l -JOIN mz_objects o ON o.id = l.object_id -LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id -LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id -WHERE o.name = 'bids_by_auction'; -``` - -```none - object | local_lag | global_lag | slowest_local_input | slowest_global_input ------------------+-----------+------------+---------------------+---------------------- - bids_by_auction | 00:00:00 | 00:00:00 | bids | bids -(1 row) -``` - -Wallclock lag in -[`mz_wallclock_global_lag_recent_history`](/sql/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history) -holds near-constant at a few seconds. A lag that instead climbs steadily, at -about one minute per minute, means the cluster has stopped making progress. - -{{< note >}} -Sources go through an additional -[snapshotting](/fundamentals/concepts/snapshotting/) step the first time they run, reading the -initial state of the upstream system before the states above apply. See -[Troubleshooting](/transform-data/freshness-troubleshooting/) for how to -diagnose a cluster that is not progressing through these states. -{{< /note >}} +{{% include-headless "/headless/cluster-lifecycle" %}} @@ -265,6 +127,11 @@ resize triggers [hydration](#hydration-considerations). During hydration, the cluster keeps serving since Materialize provisions new replicas at the target size and hydrates them before retiring the old ones. +Because hydration, not steady state, sets the floor on a cluster's size, start +at a size that hydrates comfortably and size down once you have measured what +hydration needed. For that procedure and the queries behind it, see [Cluster +sizing](/clusters/sizing/). + ## Hydration considerations {{% include-from-yaml data="hydration-details" name="definition" %}} diff --git a/doc/user/content/fundamentals/concepts/hydration.md b/doc/user/content/fundamentals/concepts/hydration.md index 8d73ee4e489cf..1f6f30f7262bc 100644 --- a/doc/user/content/fundamentals/concepts/hydration.md +++ b/doc/user/content/fundamentals/concepts/hydration.md @@ -28,6 +28,16 @@ table. {{% yaml-table data="hydration-objects-table" %}} +## Measuring hydration + +How much memory and time hydration needs depends on the data, not just on the +query, so the reliable way to find out is to measure a hydration you have +already run. Materialize records completed hydration episodes durably, per +object and per replica, including the resource high-water marks observed for +each replica episode. Those records outlive the replica restart or resize that +produced them, which is what makes them usable for sizing a cluster. See +[Cluster sizing](/clusters/sizing/). + ## Hydration strategies Hydration primarily impacts memory usage, and its speed scales with cluster diff --git a/doc/user/content/headless/cluster-lifecycle.md b/doc/user/content/headless/cluster-lifecycle.md new file mode 100644 index 0000000000000..f424615700f52 --- /dev/null +++ b/doc/user/content/headless/cluster-lifecycle.md @@ -0,0 +1,149 @@ +--- +headless: true +--- +## Lifecycle of a cluster + +Whenever a cluster starts running a workload (after you create it, resize it, +or one of its replicas restarts), its replicas move through a sequence of states +before results are fully up to date. Knowing which state a cluster is in tells +you whether it is making progress or is stuck. + +The queries below monitor a cluster named `lifecycle_demo` that hosts the +materialized view `bids_by_auction` and its index `bids_by_auction_idx`, both +built on a continuously-updating `AUCTION` load-generator source. Substitute +your own cluster and object names. + +### Provisioning + +Replicas are scheduled and brought online. A cluster with a [replication +factor](/fundamentals/concepts/clusters/#cluster-replicas) of `0` has no +compute and never leaves this state. To monitor progress, check that replicas +report `online` in +[`mz_cluster_replica_statuses`](/sql/system-catalog/mz_internal/#mz_cluster_replica_statuses), +and confirm the cluster has replicas via +[`mz_clusters`](/sql/system-catalog/mz_catalog/#mz_clusters). + +```mzsql +SELECT c.name AS cluster, r.name AS replica, r.size, st.status, st.reason +FROM mz_internal.mz_cluster_replica_statuses st +JOIN mz_catalog.mz_cluster_replicas r ON r.id = st.replica_id +JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id +WHERE c.name = 'lifecycle_demo' +ORDER BY r.name; +``` + +```none + cluster | replica | size | status | reason +----------------+---------+------+--------+-------- + lifecycle_demo | r1 | 25cc | online | +(1 row) +``` + +The `reason` column is empty while the replica is `online`, and reports why a +replica is unavailable otherwise. + +### Hydrating + +Each replica reconstructs its in-memory state by reading from Materialize's +storage layer (see [hydration](/fundamentals/concepts/hydration/)). While an object is +hydrating, its `hydrated` flag reads `f` and its lag is reported as `NULL`. To +monitor progress, check the `hydrated` flag per object in +[`mz_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_hydration_statuses), +where the `replica_id` stays blank until the object attaches to a replica. For +indexes and materialized views, +[`mz_compute_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_compute_hydration_statuses) +also reports how long hydration took. + +```mzsql +SELECT o.name AS object, o.type, r.name AS replica, ch.hydrated, ch.hydration_time +FROM mz_internal.mz_compute_hydration_statuses ch +JOIN mz_objects o ON o.id = ch.object_id +JOIN mz_catalog.mz_cluster_replicas r ON r.id = ch.replica_id +WHERE o.name IN ('bids_by_auction', 'bids_by_auction_idx', 'bids_load') +ORDER BY o.name; +``` + +```none + object | type | replica | hydrated | hydration_time +---------------------+-------------------+---------+----------+----------------- + bids_by_auction | materialized-view | r1 | t | 00:00:00.000074 + bids_by_auction_idx | index | r1 | t | 00:00:00.000019 + bids_load | materialized-view | r1 | t | 00:00:05.6032 +(3 rows) +``` + +The light view and index hydrate in microseconds, while the larger `bids_load` +view takes about 5.6 seconds. A larger object with more state to reconstruct +shows a longer, more visible hydration window. + +Both relations report only the current state, so they are wiped when a replica +or Materialize restarts. To compare this hydration against earlier ones, read +the durable [hydration +history](/clusters/sizing/#read-what-the-last-hydration-needed) instead. + +### Catching up + +Once hydrated, the cluster processes the backlog of input updates that +accumulated while it was unavailable, so its total lag starts high and comes +down. To monitor progress, watch `lag` decrease in +[`mz_wallclock_global_lag_recent_history`](/sql/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history), +or break the lag down by input with +[`mz_materialization_lag`](/sql/system-catalog/mz_internal/#mz_materialization_lag). + +```mzsql +SELECT o.name AS object, l.local_lag, l.global_lag, + si.name AS slowest_local_input, sg.name AS slowest_global_input +FROM mz_internal.mz_materialization_lag l +JOIN mz_objects o ON o.id = l.object_id +LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id +LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id +WHERE o.name IN ('bids_by_auction', 'bids_load') +ORDER BY o.name; +``` + +```none + object | local_lag | global_lag | slowest_local_input | slowest_global_input +-----------------+------------------+------------------+---------------------+---------------------- + bids_by_auction | 00:00:34.001 | 00:00:34.001 | bids | bids + bids_load | 00:00:41.001 | 00:00:41.001 | bids | bids +``` + +Both objects trail their slowest input, the `bids` source, by tens of seconds. +As the cluster works through the backlog, these lags fall. + +### Steady state + +The cluster has caught up and its lag holds low and roughly constant, typically +a few seconds. Re-running the lag query confirms the objects have caught up to +their input. + +```mzsql +SELECT o.name AS object, l.local_lag, l.global_lag, + si.name AS slowest_local_input, sg.name AS slowest_global_input +FROM mz_internal.mz_materialization_lag l +JOIN mz_objects o ON o.id = l.object_id +LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id +LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id +WHERE o.name = 'bids_by_auction'; +``` + +```none + object | local_lag | global_lag | slowest_local_input | slowest_global_input +-----------------+-----------+------------+---------------------+---------------------- + bids_by_auction | 00:00:00 | 00:00:00 | bids | bids +(1 row) +``` + +Wallclock lag in +[`mz_wallclock_global_lag_recent_history`](/sql/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history) +holds near-constant at a few seconds. A lag that instead climbs steadily, at +about one minute per minute, means the cluster has stopped making progress. + +{{< note >}} +Sources go through an additional +[snapshotting](/fundamentals/concepts/snapshotting/) step the first time they run, reading the +initial state of the upstream system before the states above apply. See +[Troubleshooting](/transform-data/freshness-troubleshooting/) for how to +diagnose a cluster that is not progressing through these states. +{{< /note >}} + diff --git a/doc/user/content/sql/system-catalog/mz_catalog.md b/doc/user/content/sql/system-catalog/mz_catalog.md index d6ebd9fe49fab..831edab8f3ed8 100644 --- a/doc/user/content/sql/system-catalog/mz_catalog.md +++ b/doc/user/content/sql/system-catalog/mz_catalog.md @@ -111,7 +111,7 @@ any kind of capacity planning. | `processes` | [`uint8`] | The number of processes in the replica. | | `workers` | [`uint8`] | The number of Timely Dataflow workers per process. | | `cpu_nano_cores` | [`uint8`] | The CPU allocation per process, in billionths of a vCPU core. | -| `memory_bytes` | [`uint8`] | The RAM allocation per process, in billionths of a vCPU core. | +| `memory_bytes` | [`uint8`] | The RAM allocation per process, in bytes. | | `disk_bytes` | [`uint8`] | The disk allocation per process. | | `credits_per_hour` | [`numeric`] | The number of compute credits consumed per hour. | diff --git a/doc/user/content/sql/system-catalog/mz_internal.md b/doc/user/content/sql/system-catalog/mz_internal.md index d36965fa807ed..9449091d98136 100644 --- a/doc/user/content/sql/system-catalog/mz_internal.md +++ b/doc/user/content/sql/system-catalog/mz_internal.md @@ -723,6 +723,8 @@ The `mz_object_history` view enriches the [`mz_catalog.mz_objects`](/sql/system- ## `mz_object_hydration_history` +{{< warn-if-unreleased v26.40 >}} + The `mz_object_hydration_history` table records completed hydration of indexes and materialized views, with one row for each time a dataflow hydrated on a replica. By default, rows are retained for 30 days while collection is enabled. Disabling @@ -737,6 +739,13 @@ multi-process replica, timestamps come from process-local logging clocks and inc their clock skew. A process whose clock is ahead can be absent at the sampled logical timestamp, so the recorded finish can precede the latest process's finish. +`object_id` is a global ID rather than a catalog item ID, so join +[`mz_object_global_ids`](#mz_object_global_ids) to reach the index or +materialized view. To recover the name and size of a replica that has since been +replaced, join [`mz_cluster_replica_history`](#mz_cluster_replica_history). For +how to use these columns to choose a cluster size, see [Cluster +sizing](/clusters/sizing/). + | Field | Type | Meaning | | -------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------ | @@ -750,6 +759,8 @@ logical timestamp, so the recorded finish can precede the latest process's finis ## `mz_replica_hydration_history` +{{< warn-if-unreleased v26.41 >}} + The `mz_replica_hydration_history` table records successful replica hydration episodes. An episode begins when a maintained compute dataflow is installed on a fully hydrated replica and finishes when every running maintained compute @@ -761,6 +772,14 @@ is recorded. Resource peaks cover the replica processes' lifetimes through collection, not only the hydration episode. On a multi-process replica, the table records the largest peak reported by any process. +Because memory and disk limits apply to each replica process independently, +compare `peak_memory_bytes` against +[`mz_cluster_replica_sizes.memory_bytes`](/sql/system-catalog/mz_catalog/#mz_cluster_replica_sizes), +which is also per process. Join +[`mz_cluster_replica_history`](#mz_cluster_replica_history) for the size the +episode ran at, since that row survives the replica. For how to use these +columns to choose a cluster size, see [Cluster sizing](/clusters/sizing/). + | Field | Type | Meaning | | ------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------ | @@ -768,7 +787,7 @@ table records the largest peak reported by any process. | `cluster_id` | [`text`] | The ID of the replica's cluster. | | `started_at` | [`timestamp with time zone`] | The earliest maintained compute dataflow installation in the hydration episode. | | `finished_at` | [`timestamp with time zone`] | The latest maintained compute dataflow hydration in the hydration episode. | -| `object_count` | [`uint8`] | The number of maintained compute dataflows in the hydration episode. | +| `object_count` | [`uint8`] | The number of maintained compute dataflows in the hydration episode. Includes the replica's system introspection dataflows, so it exceeds the number of indexes and materialized views you created. | | `peak_memory_bytes` | [`uint8`] | The largest process-lifetime cgroup memory high-water mark reported by any process when the collector recorded the episode. `NULL` if the platform reports no cgroup memory peak. | | `peak_disk_bytes` | [`uint8`] | The largest process-lifetime scratch-filesystem or swap high-water mark reported by any process when the collector recorded the episode. Filesystem peaks are sampled lower bounds. `NULL` if neither measurement is available. | | `status` | [`text`] | The hydration episode's status. Currently always `hydrated`. | diff --git a/src/catalog/src/builtin/mz_catalog.rs b/src/catalog/src/builtin/mz_catalog.rs index bdd9cd3969b5e..1e77a097bbddf 100644 --- a/src/catalog/src/builtin/mz_catalog.rs +++ b/src/catalog/src/builtin/mz_catalog.rs @@ -2804,10 +2804,7 @@ pub static MZ_CLUSTER_REPLICA_SIZES: LazyLock = LazyLock::new(|| B "cpu_nano_cores", "The CPU allocation per process, in billionths of a vCPU core.", ), - ( - "memory_bytes", - "The RAM allocation per process, in billionths of a vCPU core.", - ), + ("memory_bytes", "The RAM allocation per process, in bytes."), ("disk_bytes", "The disk allocation per process."), ( "credits_per_hour", diff --git a/src/catalog/src/builtin/mz_internal.rs b/src/catalog/src/builtin/mz_internal.rs index 006492fbdf5d2..2cb92ab492b86 100644 --- a/src/catalog/src/builtin/mz_internal.rs +++ b/src/catalog/src/builtin/mz_internal.rs @@ -5234,7 +5234,7 @@ pub static MZ_REPLICA_HYDRATION_HISTORY: LazyLock = LazyLock::new( ), ( "object_count", - "The number of maintained compute dataflows in the hydration episode.", + "The number of maintained compute dataflows in the hydration episode. Includes the replica's system introspection dataflows, so it exceeds the number of indexes and materialized views you created.", ), ( "peak_memory_bytes", diff --git a/test/sqllogictest/autogenerated/mz_catalog.slt b/test/sqllogictest/autogenerated/mz_catalog.slt index 4079dd2ff2592..35d6eff460b69 100644 --- a/test/sqllogictest/autogenerated/mz_catalog.slt +++ b/test/sqllogictest/autogenerated/mz_catalog.slt @@ -76,7 +76,7 @@ size text The␠human-readable␠replica␠size. processes uint8 The␠number␠of␠processes␠in␠the␠replica. workers uint8 The␠number␠of␠Timely␠Dataflow␠workers␠per␠process. cpu_nano_cores uint8 The␠CPU␠allocation␠per␠process,␠in␠billionths␠of␠a␠vCPU␠core. -memory_bytes uint8 The␠RAM␠allocation␠per␠process,␠in␠billionths␠of␠a␠vCPU␠core. +memory_bytes uint8 The␠RAM␠allocation␠per␠process,␠in␠bytes. disk_bytes uint8 The␠disk␠allocation␠per␠process. credits_per_hour numeric The␠number␠of␠compute␠credits␠consumed␠per␠hour. diff --git a/test/sqllogictest/autogenerated/mz_internal.slt b/test/sqllogictest/autogenerated/mz_internal.slt index 66cd3a52e3a4e..679787936fa86 100644 --- a/test/sqllogictest/autogenerated/mz_internal.slt +++ b/test/sqllogictest/autogenerated/mz_internal.slt @@ -406,7 +406,7 @@ replica_id text The␠ID␠of␠the␠cluster␠replica.␠May␠name␠a␠re cluster_id text The␠ID␠of␠the␠replica's␠cluster. started_at timestamp␠with␠time␠zone The␠earliest␠maintained␠compute␠dataflow␠installation␠in␠the␠hydration␠episode. finished_at timestamp␠with␠time␠zone The␠latest␠maintained␠compute␠dataflow␠hydration␠in␠the␠hydration␠episode. -object_count uint8 The␠number␠of␠maintained␠compute␠dataflows␠in␠the␠hydration␠episode. +object_count uint8 The␠number␠of␠maintained␠compute␠dataflows␠in␠the␠hydration␠episode.␠Includes␠the␠replica's␠system␠introspection␠dataflows,␠so␠it␠exceeds␠the␠number␠of␠indexes␠and␠materialized␠views␠you␠created. peak_memory_bytes uint8 The␠largest␠process-lifetime␠cgroup␠memory␠high-water␠mark␠reported␠by␠any␠process␠when␠the␠collector␠recorded␠the␠episode.␠`NULL`␠if␠the␠platform␠reports␠no␠cgroup␠memory␠peak. peak_disk_bytes uint8 The␠largest␠process-lifetime␠scratch-filesystem␠or␠swap␠high-water␠mark␠reported␠by␠any␠process␠when␠the␠collector␠recorded␠the␠episode.␠Filesystem␠peaks␠are␠sampled␠lower␠bounds.␠`NULL`␠if␠neither␠measurement␠is␠available. status text The␠hydration␠episode's␠status.␠Currently␠always␠`hydrated`. From b6e4721c7c7f71601d7f618ea9b1656802e7e30f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 18:23:42 +0000 Subject: [PATCH 2/9] docs: drop the trailing blank line in the cluster lifecycle fragment Extracting the lifecycle section carried over the blank line that separated it from the next heading, which `check-whitespace` rejects at EOF. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E6qZnfQsyVbQ2hydYaYder --- doc/user/content/headless/cluster-lifecycle.md | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/user/content/headless/cluster-lifecycle.md b/doc/user/content/headless/cluster-lifecycle.md index f424615700f52..85fd41216a4ca 100644 --- a/doc/user/content/headless/cluster-lifecycle.md +++ b/doc/user/content/headless/cluster-lifecycle.md @@ -146,4 +146,3 @@ initial state of the upstream system before the states above apply. See [Troubleshooting](/transform-data/freshness-troubleshooting/) for how to diagnose a cluster that is not progressing through these states. {{< /note >}} - From a13ec273a315354f065f2089e52586b72f3e99a8 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 18:56:16 +0000 Subject: [PATCH 3/9] docs: apply review feedback to the cluster sizing guide Retitles the page "Optimize cluster size" and reshapes it around the questions a reader arrives with, per review. * Rewrite the intro to lead with provisioning for peak resource usage, and link out to the cluster lifecycle rather than including it. The headless fragment existed only for that include, so it goes away and the lifecycle section returns inline to the clusters concept page unchanged. * Replace the "Why hydration sets the size" section with a short note. * State the version assumption as prose: the guide assumes v26.42 or later, which is where peak resource usage during hydration became trackable. * Drop the "Choose a steady-state size" step. Its headroom query inferred a target size from a peak, which is not a rule we want to publish. Step 3 now points at `mz_cluster_replica_sizes` for the per-process memory a candidate size provides, and sizing down follows from that. * Rewrite the resize step around `ALTER CLUSTER` being graceful, and cut the paragraphs on rollback semantics, per-process peak measurement, and reading `object_count`. * Frame the remaining sections as questions, and add "How do I speed up hydration?" covering autoscaling and pointing at the hydration strategies. Both hydration history relations are gated on v26.42, which needs a release page for the shortcode to resolve, so this adds `releases/v26.42.md` with the next weekly date. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E6qZnfQsyVbQ2hydYaYder --- doc/user/content/clusters/_index.md | 2 +- .../clusters/operational-guidelines/_index.md | 2 +- doc/user/content/clusters/sizing.md | 211 +++++------------- .../content/fundamentals/concepts/clusters.md | 152 ++++++++++++- .../fundamentals/concepts/hydration.md | 2 +- .../content/headless/cluster-lifecycle.md | 148 ------------ doc/user/content/releases/v26.42.md | 9 + .../content/sql/system-catalog/mz_internal.md | 10 +- 8 files changed, 214 insertions(+), 322 deletions(-) delete mode 100644 doc/user/content/headless/cluster-lifecycle.md create mode 100644 doc/user/content/releases/v26.42.md diff --git a/doc/user/content/clusters/_index.md b/doc/user/content/clusters/_index.md index b67b74b42f3ad..150fbcccf4797 100644 --- a/doc/user/content/clusters/_index.md +++ b/doc/user/content/clusters/_index.md @@ -12,5 +12,5 @@ Clusters provide the compute resources for running dataflows in Materialize. - Learn about [clusters](/fundamentals/concepts/clusters/). - Follow the [operational guidelines](/clusters/operational-guidelines/). -- Choose a [cluster size](/clusters/sizing/). +- Learn how to [optimize cluster size](/clusters/sizing/). - Understand [system clusters](/clusters/system-clusters/). diff --git a/doc/user/content/clusters/operational-guidelines/_index.md b/doc/user/content/clusters/operational-guidelines/_index.md index 00a84d1df84c9..16cf938c97634 100644 --- a/doc/user/content/clusters/operational-guidelines/_index.md +++ b/doc/user/content/clusters/operational-guidelines/_index.md @@ -68,7 +68,7 @@ For upsert sources, snapshotting is a resource-intensive operation that can requ When sizing a cluster, budget for hydration memory on top of the steady-state cost. Rather than estimating that budget, start at a size that hydrates comfortably and size down once you have measured what hydration needed: see -[Cluster sizing](/clusters/sizing/). +[Optimize cluster size](/clusters/sizing/). The table below summarizes, per object type, when each object hydrates and the memory it uses. For more on hydration, including strategies to reduce its diff --git a/doc/user/content/clusters/sizing.md b/doc/user/content/clusters/sizing.md index e1efdebc3017f..f3fa57b7ed391 100644 --- a/doc/user/content/clusters/sizing.md +++ b/doc/user/content/clusters/sizing.md @@ -1,68 +1,42 @@ --- -title: "Cluster sizing" -description: "Pick a cluster size by measuring what hydration actually needed, then sizing down." +title: "Optimize cluster size" +description: "Optimize your cluster size by observing the resources it requires to hydrate." menu: main: parent: "clusters" weight: 5 - name: "Cluster sizing" + name: "Optimize cluster size" identifier: "cluster-sizing" --- -A cluster's [size](/sql/create-cluster/#available-sizes) fixes the CPU, memory, -and scratch disk available to every replica of that cluster, and on Materialize -Cloud it fixes the [cost](/materialize-cloud/billing/#compute). The size you -need is set by the most expensive thing the cluster does, and for most clusters -that is [hydration](/fundamentals/concepts/hydration/) rather than steady state. - -Hydration is also the part you cannot predict from the query text. How much -memory a join or an aggregation needs depends on the data: key distribution, -skew, and how much history the inputs carry. So rather than estimate, start at a -size that hydrates comfortably, measure what hydration needed, and then size -down. - -{{% include-headless "/headless/cluster-lifecycle" %}} - -## Why hydration sets the size - -Steady state is the cheap part of a cluster's life. Once a dataflow is hydrated -it holds its arrangements and applies incoming updates, and its memory tracks -the size of the state it maintains. Hydration is different: the replica rebuilds -that state from the storage layer, which means reading the inputs and building -the intermediate arrangements that produce it. Peak memory during hydration is -therefore higher than steady-state memory, often around twice as high, and the -same holds for the time it takes. - -That gap decides two things at once: - -- **A cluster that cannot hydrate serves nothing.** A replica that exceeds its - memory allocation is restarted, and it then attempts the same hydration again. - An undersized cluster does not degrade gracefully into a slow cluster: it - restarts in a loop and never reaches the point where it can answer queries. - -- **A cluster sized for steady state may not survive a restart.** The size that - holds a hydrated dataflow can be too small to rebuild it. Restarts are not - exceptional (a resize, a version upgrade, or a new index all trigger - hydration), so the size has to cover the rebuild, not just the result. - -Both point the same way: choose a size that hydrates, then reduce it with -evidence. +A cluster's [size](/sql/create-cluster/#available-sizes) defines the CPU, +memory, and scratch disk available to every replica. On Materialize Cloud, this +determines the [cost](/materialize-cloud/billing/#compute) of the cluster. +Clusters should be provisioned for peak resource usage, to ensure that they can +handle the load placed on them. For most clusters, peak resource usage happens +during [hydration](/fundamentals/concepts/hydration/). + +This guide will walk you through how to estimate resources required for +hydration. Before reading this guide, make sure you understand the [lifecycle of +a cluster](/fundamentals/concepts/clusters/#lifecycle-of-a-cluster). + +{{< note >}} +Hydration rebuilds a dataflow's in-memory state from the storage layer, which +takes more memory than maintaining that state afterwards. The size that holds a +hydrated cluster is therefore not always the size that can rebuild it, and a +replica that runs out of memory while hydrating restarts and tries again rather +than running slower. +{{< /note >}} ## Start large, then size down -The procedure below oversizes the cluster deliberately for one hydration, reads -what that hydration needed from the catalog, and uses those numbers to pick a -steady-state size. - -Steps 3 through 5 read the durable hydration history relations: - -{{< warn-if-unreleased v26.41 >}} +This guide assumes you are running Materialize v26.42 or later. v26.42 included +improvements to allow you to track peak resource usage during hydration. ### 1. Create the cluster at a generous size Pick a size you are confident can hydrate the workload, even if it is clearly -more than steady state needs. Oversizing costs money for as long as the cluster -runs at that size. Undersizing costs a hydration that never completes. +more than steady state needs. ```mzsql CREATE CLUSTER analytics (SIZE = '400cc'); @@ -85,8 +59,7 @@ WHERE c.name = 'analytics' AND h.hydrated IS NOT TRUE; An empty result means every object on the cluster is hydrated. A row with a `NULL` `replica_id` is an object that has not attached to a replica yet, which -`IS NOT TRUE` catches along with `hydrated = false`. See [Lifecycle of a -cluster](#lifecycle-of-a-cluster) for the states that follow. +`IS NOT TRUE` catches along with `hydrated = false`. @@ -120,27 +93,9 @@ ORDER BY h.started_at DESC; (1 row) ``` -The join to -[`mz_internal.mz_cluster_replica_history`](/sql/system-catalog/mz_internal/#mz_cluster_replica_history) -is what makes the numbers usable for sizing: it supplies the size the episode -ran at, and it keeps that row after the replica is gone. Hydration history -itself stores only the replica ID, and a resize replaces the replica, so joining -[`mz_cluster_replicas`](/sql/system-catalog/mz_catalog/#mz_cluster_replicas) -instead would drop exactly the episodes you want to compare against. - -Two columns need reading with care: - -- `object_count` counts every maintained dataflow in the episode, which includes - the system introspection dataflows each replica runs. It is normally a few - dozen higher than the number of objects you created, and it is not the number - of rows the per-object table holds for that replica. - -- `peak_memory_bytes` and `peak_disk_bytes` are the largest values reported by - any single process of the replica, not the sum across processes. Memory and - disk limits apply per process, so the maximum is what answers whether any - process came close to its limit. See [Reading the recorded - numbers](#reading-the-recorded-numbers) for what the peaks do and do not - cover. +Compare `peak_memory` against the memory the candidate size provides, which +[`mz_catalog.mz_cluster_replica_sizes`](/sql/system-catalog/mz_catalog/#mz_cluster_replica_sizes) +reports per process, and leave headroom for the inputs to grow. To find which object dominated the episode, read the per-object table, [`mz_internal.mz_object_hydration_history`](/sql/system-catalog/mz_internal/#mz_object_hydration_history). @@ -175,87 +130,22 @@ LIMIT 5; Every replica records its own rows, so a cluster with a replication factor above one, or one that has been resized, returns a row per object per replica. -The per-object table carries no resource columns, because peaks are measured per -process and a process runs many dataflows at once. Use it to find the object -whose hydration dominates the episode, then attribute the episode's peak to that -object's cluster placement. If one object accounts for most of the episode, [move -it to its own +If one object accounts for most of the episode, [move it to its own cluster](/fundamentals/concepts/hydration/#hydration-strategies) so its hydration peak stops dictating the size of everything else. -### 4. Choose a steady-state size +### 4. Size down -The episode's peak memory is what the smaller size has to fit, with headroom for -data growth. The following query reports, per cluster, the largest peak still in -the history and the smallest size whose per-process memory keeps that peak under -75%: - -```mzsql -WITH observed AS ( - SELECT - rh.cluster_name AS cluster, - max(h.peak_memory_bytes) AS peak_memory_bytes - FROM mz_internal.mz_replica_hydration_history AS h - JOIN mz_internal.mz_cluster_replica_history AS rh ON rh.replica_id = h.replica_id - WHERE h.peak_memory_bytes IS NOT NULL - GROUP BY rh.cluster_name -) -SELECT - o.cluster, - pg_size_pretty(o.peak_memory_bytes) AS peak_hydration_memory, - ( - SELECT s.size - FROM mz_catalog.mz_cluster_replica_sizes AS s - WHERE o.peak_memory_bytes <= s.memory_bytes * 0.75 - ORDER BY s.memory_bytes - LIMIT 1 - ) AS smallest_size_with_headroom -FROM observed AS o -ORDER BY o.cluster; -``` - -```none - cluster | peak_hydration_memory | smallest_size_with_headroom ------------+-----------------------+----------------------------- - analytics | 11 GB | 100cc -(1 row) -``` - -The 75% in that query is a starting point, not a guarantee. Raise the headroom -when the inputs are growing, when the workload is seasonal, or when the cluster -also serves ad-hoc `SELECT` queries, since those compete for the same memory and -are not part of a hydration episode. - -Treat the result as the next size to try rather than the final answer. Sizing -down changes the thing you measured: fewer workers per replica changes how the -work is distributed, so the peak at `100cc` is not the peak at `400cc` divided -by four. Step down one size at a time and re-measure after each step. - -{{< tip >}} -If a cluster's peak is dominated by hydration and its steady state is much -cheaper, you can keep it small and let it borrow capacity only while it -hydrates. An [`AUTO SCALING STRATEGY (ON -HYDRATION)`](/sql/alter-cluster/#speed-up-hydration-by-autoscaling-to-a-larger-size) -provisions an extra burst replica at a larger size whenever the cluster has -un-hydrated objects, including after a restart or an upgrade, and removes it once -a steady-size replica catches up. You then pay the hydration size only for the -duration of hydration. -{{< /tip >}} - -### 5. Size down and confirm - -A resize is graceful by default: Materialize hydrates replicas at the new size -alongside the current ones before retiring them, and rolls the resize back if -they do not hydrate within the reconfiguration timeout. See [resizing -process](/sql/alter-cluster/#resizing-process) for the details and how to change -that behavior. +Once you have found the appropriate size, you can downsize by altering the +cluster: ```mzsql ALTER CLUSTER analytics SET (SIZE = '100cc'); ``` -That rollback is what makes stepping down safe to try: a size that cannot -rebuild the state leaves the cluster where it was rather than serving nothing. +`ALTER CLUSTER` operations are graceful. This means the smaller cluster will +hydrate in parallel, and Materialize will cut over to the smaller cluster when +it is ready. The resize also hydrates the whole workload again, which produces exactly the measurement you need to confirm the new size. Re-run the query from [step @@ -269,12 +159,6 @@ measurement you need to confirm the new size. Re-run the query from [step (2 rows) ``` -This is the outcome to look for, and it is also the point of measuring rather -than estimating. Peak memory barely moved, so `100cc` holds the workload with -the headroom the previous step asked for. Hydration got three times slower, which -is the cost of the smaller size, and whether that matters depends on how long -you can tolerate a restart taking. - If the new size is too small, no completed episode is recorded for the new replica at all. Only successful hydration is recorded, so an out-of-memory restart loop shows up as a missing row plus repeated restarts in @@ -295,14 +179,14 @@ that hydrated, and take a smaller step, or reduce the peak itself with one of the [hydration strategies](/fundamentals/concepts/hydration/#hydration-strategies). -## Reading the recorded numbers +## How should I interpret the hydration metrics? Hydration history is a best-effort record, not an audit log. Where it is approximate, it is approximate in ways that matter for sizing: - **Only successful episodes are recorded.** There is no row for a hydration that was killed, canceled, or is still running, and `status` is currently - always `hydrated`. A missing row is a signal in its own right, as in step 5, + always `hydrated`. A missing row is a signal in its own right, as in step 4, but it is never a measurement of a failure. - **Short-lived objects can be missed entirely.** Recording works by sampling @@ -316,10 +200,6 @@ approximate, it is approximate in ways that matter for sizing: episode can inherit an earlier episode's mark. For sizing this errs the safe way: the recorded value is never below the true hydration peak. -- **A peak can be `NULL`.** The values depend on what the platform exposes - (a cgroup memory peak, and a scratch filesystem or swap peak), so they are - absent rather than zero when a deployment does not report them. - - **Timestamps can carry clock skew.** On a multi-process replica the endpoints of an interval come from different process clocks, so a recorded duration includes their skew. This is not usually visible at the minute scale that @@ -334,10 +214,7 @@ approximate, it is approximate in ways that matter for sizing: - **Rows are retained for 30 days by default.** Sizing decisions should come from the recent history rather than the earliest episode still stored. -Both tables live in the [`mz_internal`](/sql/system-catalog/mz_internal/) -schema, which is not part of Materialize's stable interface. - -## If hydration history is empty +## What should I do if hydration history is empty? Recording is controlled by the `hydration_history_collection_interval` system parameter, which sets how often Materialize samples replicas for completed @@ -377,6 +254,18 @@ roughly one sample a minute it can miss a hydration spike entirely, and it does not tell you which episode a sample belonged to. That is why these are a fallback rather than the basis for a sizing decision. +## How do I speed up hydration? + +Hydration speed scales with cluster size, so a cluster can borrow capacity for +hydration alone rather than running at the larger size permanently. An [`AUTO +SCALING STRATEGY (ON +HYDRATION)`](/sql/alter-cluster/#speed-up-hydration-by-autoscaling-to-a-larger-size) +provisions an extra burst replica at a larger size whenever the cluster has +un-hydrated objects, and removes it once a steady-size replica catches up. + +To reduce the work hydration has to do in the first place, see [hydration +strategies](/fundamentals/concepts/hydration/#hydration-strategies). + ## Related pages - [Hydration](/fundamentals/concepts/hydration/) diff --git a/doc/user/content/fundamentals/concepts/clusters.md b/doc/user/content/fundamentals/concepts/clusters.md index e12998029e4d7..224a9117ee385 100644 --- a/doc/user/content/fundamentals/concepts/clusters.md +++ b/doc/user/content/fundamentals/concepts/clusters.md @@ -104,7 +104,150 @@ When provisioning replicas, See also [Hydration considerations](#hydration-considerations). -{{% include-headless "/headless/cluster-lifecycle" %}} +## Lifecycle of a cluster + +Whenever a cluster starts running a workload (after you create it, resize it, +or one of its replicas restarts), its replicas move through a sequence of states +before results are fully up to date. Knowing which state a cluster is in tells +you whether it is making progress or is stuck. + +The queries below monitor a cluster named `lifecycle_demo` that hosts the +materialized view `bids_by_auction` and its index `bids_by_auction_idx`, both +built on a continuously-updating `AUCTION` load-generator source. Substitute +your own cluster and object names. + +### Provisioning + +Replicas are scheduled and brought online. A cluster with a [replication +factor](#cluster-replicas) of `0` has no compute and never leaves this state. To +monitor progress, check that replicas report `online` in +[`mz_cluster_replica_statuses`](/sql/system-catalog/mz_internal/#mz_cluster_replica_statuses), +and confirm the cluster has replicas via +[`mz_clusters`](/sql/system-catalog/mz_catalog/#mz_clusters). + +```mzsql +SELECT c.name AS cluster, r.name AS replica, r.size, st.status, st.reason +FROM mz_internal.mz_cluster_replica_statuses st +JOIN mz_catalog.mz_cluster_replicas r ON r.id = st.replica_id +JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id +WHERE c.name = 'lifecycle_demo' +ORDER BY r.name; +``` + +```none + cluster | replica | size | status | reason +----------------+---------+------+--------+-------- + lifecycle_demo | r1 | 25cc | online | +(1 row) +``` + +The `reason` column is empty while the replica is `online`, and reports why a +replica is unavailable otherwise. + +### Hydrating + +Each replica reconstructs its in-memory state by reading from Materialize's +storage layer (see [hydration](/fundamentals/concepts/hydration/)). While an object is +hydrating, its `hydrated` flag reads `f` and its lag is reported as `NULL`. To +monitor progress, check the `hydrated` flag per object in +[`mz_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_hydration_statuses), +where the `replica_id` stays blank until the object attaches to a replica. For +indexes and materialized views, +[`mz_compute_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_compute_hydration_statuses) +also reports how long hydration took. + +```mzsql +SELECT o.name AS object, o.type, r.name AS replica, ch.hydrated, ch.hydration_time +FROM mz_internal.mz_compute_hydration_statuses ch +JOIN mz_objects o ON o.id = ch.object_id +JOIN mz_catalog.mz_cluster_replicas r ON r.id = ch.replica_id +WHERE o.name IN ('bids_by_auction', 'bids_by_auction_idx', 'bids_load') +ORDER BY o.name; +``` + +```none + object | type | replica | hydrated | hydration_time +---------------------+-------------------+---------+----------+----------------- + bids_by_auction | materialized-view | r1 | t | 00:00:00.000074 + bids_by_auction_idx | index | r1 | t | 00:00:00.000019 + bids_load | materialized-view | r1 | t | 00:00:05.6032 +(3 rows) +``` + +The light view and index hydrate in microseconds, while the larger `bids_load` +view takes about 5.6 seconds. A larger object with more state to reconstruct +shows a longer, more visible hydration window. + +Both relations report only the current state, so they are wiped when a replica +or Materialize restarts. To compare this hydration against earlier ones, and to +see the memory and disk it needed, read the durable hydration history described +in [Optimize cluster size](/clusters/sizing/). + +### Catching up + +Once hydrated, the cluster processes the backlog of input updates that +accumulated while it was unavailable, so its total lag starts high and comes +down. To monitor progress, watch `lag` decrease in +[`mz_wallclock_global_lag_recent_history`](/sql/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history), +or break the lag down by input with +[`mz_materialization_lag`](/sql/system-catalog/mz_internal/#mz_materialization_lag). + +```mzsql +SELECT o.name AS object, l.local_lag, l.global_lag, + si.name AS slowest_local_input, sg.name AS slowest_global_input +FROM mz_internal.mz_materialization_lag l +JOIN mz_objects o ON o.id = l.object_id +LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id +LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id +WHERE o.name IN ('bids_by_auction', 'bids_load') +ORDER BY o.name; +``` + +```none + object | local_lag | global_lag | slowest_local_input | slowest_global_input +-----------------+------------------+------------------+---------------------+---------------------- + bids_by_auction | 00:00:34.001 | 00:00:34.001 | bids | bids + bids_load | 00:00:41.001 | 00:00:41.001 | bids | bids +``` + +Both objects trail their slowest input, the `bids` source, by tens of seconds. +As the cluster works through the backlog, these lags fall. + +### Steady state + +The cluster has caught up and its lag holds low and roughly constant, typically +a few seconds. Re-running the lag query confirms the objects have caught up to +their input. + +```mzsql +SELECT o.name AS object, l.local_lag, l.global_lag, + si.name AS slowest_local_input, sg.name AS slowest_global_input +FROM mz_internal.mz_materialization_lag l +JOIN mz_objects o ON o.id = l.object_id +LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id +LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id +WHERE o.name = 'bids_by_auction'; +``` + +```none + object | local_lag | global_lag | slowest_local_input | slowest_global_input +-----------------+-----------+------------+---------------------+---------------------- + bids_by_auction | 00:00:00 | 00:00:00 | bids | bids +(1 row) +``` + +Wallclock lag in +[`mz_wallclock_global_lag_recent_history`](/sql/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history) +holds near-constant at a few seconds. A lag that instead climbs steadily, at +about one minute per minute, means the cluster has stopped making progress. + +{{< note >}} +Sources go through an additional +[snapshotting](/fundamentals/concepts/snapshotting/) step the first time they run, reading the +initial state of the upstream system before the states above apply. See +[Troubleshooting](/transform-data/freshness-troubleshooting/) for how to +diagnose a cluster that is not progressing through these states. +{{< /note >}} @@ -127,10 +270,9 @@ resize triggers [hydration](#hydration-considerations). During hydration, the cluster keeps serving since Materialize provisions new replicas at the target size and hydrates them before retiring the old ones. -Because hydration, not steady state, sets the floor on a cluster's size, start -at a size that hydrates comfortably and size down once you have measured what -hydration needed. For that procedure and the queries behind it, see [Cluster -sizing](/clusters/sizing/). +Because peak resource usage normally happens during hydration, size a cluster +for the resources hydration needs. For how to measure those, see [Optimize +cluster size](/clusters/sizing/). ## Hydration considerations diff --git a/doc/user/content/fundamentals/concepts/hydration.md b/doc/user/content/fundamentals/concepts/hydration.md index 1f6f30f7262bc..01c355819dc76 100644 --- a/doc/user/content/fundamentals/concepts/hydration.md +++ b/doc/user/content/fundamentals/concepts/hydration.md @@ -36,7 +36,7 @@ already run. Materialize records completed hydration episodes durably, per object and per replica, including the resource high-water marks observed for each replica episode. Those records outlive the replica restart or resize that produced them, which is what makes them usable for sizing a cluster. See -[Cluster sizing](/clusters/sizing/). +[Optimize cluster size](/clusters/sizing/). ## Hydration strategies diff --git a/doc/user/content/headless/cluster-lifecycle.md b/doc/user/content/headless/cluster-lifecycle.md deleted file mode 100644 index 85fd41216a4ca..0000000000000 --- a/doc/user/content/headless/cluster-lifecycle.md +++ /dev/null @@ -1,148 +0,0 @@ ---- -headless: true ---- -## Lifecycle of a cluster - -Whenever a cluster starts running a workload (after you create it, resize it, -or one of its replicas restarts), its replicas move through a sequence of states -before results are fully up to date. Knowing which state a cluster is in tells -you whether it is making progress or is stuck. - -The queries below monitor a cluster named `lifecycle_demo` that hosts the -materialized view `bids_by_auction` and its index `bids_by_auction_idx`, both -built on a continuously-updating `AUCTION` load-generator source. Substitute -your own cluster and object names. - -### Provisioning - -Replicas are scheduled and brought online. A cluster with a [replication -factor](/fundamentals/concepts/clusters/#cluster-replicas) of `0` has no -compute and never leaves this state. To monitor progress, check that replicas -report `online` in -[`mz_cluster_replica_statuses`](/sql/system-catalog/mz_internal/#mz_cluster_replica_statuses), -and confirm the cluster has replicas via -[`mz_clusters`](/sql/system-catalog/mz_catalog/#mz_clusters). - -```mzsql -SELECT c.name AS cluster, r.name AS replica, r.size, st.status, st.reason -FROM mz_internal.mz_cluster_replica_statuses st -JOIN mz_catalog.mz_cluster_replicas r ON r.id = st.replica_id -JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id -WHERE c.name = 'lifecycle_demo' -ORDER BY r.name; -``` - -```none - cluster | replica | size | status | reason -----------------+---------+------+--------+-------- - lifecycle_demo | r1 | 25cc | online | -(1 row) -``` - -The `reason` column is empty while the replica is `online`, and reports why a -replica is unavailable otherwise. - -### Hydrating - -Each replica reconstructs its in-memory state by reading from Materialize's -storage layer (see [hydration](/fundamentals/concepts/hydration/)). While an object is -hydrating, its `hydrated` flag reads `f` and its lag is reported as `NULL`. To -monitor progress, check the `hydrated` flag per object in -[`mz_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_hydration_statuses), -where the `replica_id` stays blank until the object attaches to a replica. For -indexes and materialized views, -[`mz_compute_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_compute_hydration_statuses) -also reports how long hydration took. - -```mzsql -SELECT o.name AS object, o.type, r.name AS replica, ch.hydrated, ch.hydration_time -FROM mz_internal.mz_compute_hydration_statuses ch -JOIN mz_objects o ON o.id = ch.object_id -JOIN mz_catalog.mz_cluster_replicas r ON r.id = ch.replica_id -WHERE o.name IN ('bids_by_auction', 'bids_by_auction_idx', 'bids_load') -ORDER BY o.name; -``` - -```none - object | type | replica | hydrated | hydration_time ----------------------+-------------------+---------+----------+----------------- - bids_by_auction | materialized-view | r1 | t | 00:00:00.000074 - bids_by_auction_idx | index | r1 | t | 00:00:00.000019 - bids_load | materialized-view | r1 | t | 00:00:05.6032 -(3 rows) -``` - -The light view and index hydrate in microseconds, while the larger `bids_load` -view takes about 5.6 seconds. A larger object with more state to reconstruct -shows a longer, more visible hydration window. - -Both relations report only the current state, so they are wiped when a replica -or Materialize restarts. To compare this hydration against earlier ones, read -the durable [hydration -history](/clusters/sizing/#read-what-the-last-hydration-needed) instead. - -### Catching up - -Once hydrated, the cluster processes the backlog of input updates that -accumulated while it was unavailable, so its total lag starts high and comes -down. To monitor progress, watch `lag` decrease in -[`mz_wallclock_global_lag_recent_history`](/sql/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history), -or break the lag down by input with -[`mz_materialization_lag`](/sql/system-catalog/mz_internal/#mz_materialization_lag). - -```mzsql -SELECT o.name AS object, l.local_lag, l.global_lag, - si.name AS slowest_local_input, sg.name AS slowest_global_input -FROM mz_internal.mz_materialization_lag l -JOIN mz_objects o ON o.id = l.object_id -LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id -LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id -WHERE o.name IN ('bids_by_auction', 'bids_load') -ORDER BY o.name; -``` - -```none - object | local_lag | global_lag | slowest_local_input | slowest_global_input ------------------+------------------+------------------+---------------------+---------------------- - bids_by_auction | 00:00:34.001 | 00:00:34.001 | bids | bids - bids_load | 00:00:41.001 | 00:00:41.001 | bids | bids -``` - -Both objects trail their slowest input, the `bids` source, by tens of seconds. -As the cluster works through the backlog, these lags fall. - -### Steady state - -The cluster has caught up and its lag holds low and roughly constant, typically -a few seconds. Re-running the lag query confirms the objects have caught up to -their input. - -```mzsql -SELECT o.name AS object, l.local_lag, l.global_lag, - si.name AS slowest_local_input, sg.name AS slowest_global_input -FROM mz_internal.mz_materialization_lag l -JOIN mz_objects o ON o.id = l.object_id -LEFT JOIN mz_objects si ON si.id = l.slowest_local_input_id -LEFT JOIN mz_objects sg ON sg.id = l.slowest_global_input_id -WHERE o.name = 'bids_by_auction'; -``` - -```none - object | local_lag | global_lag | slowest_local_input | slowest_global_input ------------------+-----------+------------+---------------------+---------------------- - bids_by_auction | 00:00:00 | 00:00:00 | bids | bids -(1 row) -``` - -Wallclock lag in -[`mz_wallclock_global_lag_recent_history`](/sql/system-catalog/mz_internal/#mz_wallclock_global_lag_recent_history) -holds near-constant at a few seconds. A lag that instead climbs steadily, at -about one minute per minute, means the cluster has stopped making progress. - -{{< note >}} -Sources go through an additional -[snapshotting](/fundamentals/concepts/snapshotting/) step the first time they run, reading the -initial state of the upstream system before the states above apply. See -[Troubleshooting](/transform-data/freshness-troubleshooting/) for how to -diagnose a cluster that is not progressing through these states. -{{< /note >}} diff --git a/doc/user/content/releases/v26.42.md b/doc/user/content/releases/v26.42.md new file mode 100644 index 0000000000000..9577b977777b6 --- /dev/null +++ b/doc/user/content/releases/v26.42.md @@ -0,0 +1,9 @@ +--- +title: Materialize v26.42 +date: 2026-09-16 +released: false +patch: 0 +publish_helm_chart: true +build: + render: never +--- diff --git a/doc/user/content/sql/system-catalog/mz_internal.md b/doc/user/content/sql/system-catalog/mz_internal.md index 9449091d98136..7d013b2e704ce 100644 --- a/doc/user/content/sql/system-catalog/mz_internal.md +++ b/doc/user/content/sql/system-catalog/mz_internal.md @@ -723,7 +723,7 @@ The `mz_object_history` view enriches the [`mz_catalog.mz_objects`](/sql/system- ## `mz_object_hydration_history` -{{< warn-if-unreleased v26.40 >}} +{{< warn-if-unreleased v26.42 >}} The `mz_object_hydration_history` table records completed hydration of indexes and materialized views, with one row for each time a dataflow hydrated on a replica. @@ -743,8 +743,8 @@ logical timestamp, so the recorded finish can precede the latest process's finis [`mz_object_global_ids`](#mz_object_global_ids) to reach the index or materialized view. To recover the name and size of a replica that has since been replaced, join [`mz_cluster_replica_history`](#mz_cluster_replica_history). For -how to use these columns to choose a cluster size, see [Cluster -sizing](/clusters/sizing/). +how to use these columns to choose a cluster size, see [Optimize cluster +size](/clusters/sizing/). | Field | Type | Meaning | @@ -759,7 +759,7 @@ sizing](/clusters/sizing/). ## `mz_replica_hydration_history` -{{< warn-if-unreleased v26.41 >}} +{{< warn-if-unreleased v26.42 >}} The `mz_replica_hydration_history` table records successful replica hydration episodes. An episode begins when a maintained compute dataflow is installed on @@ -778,7 +778,7 @@ compare `peak_memory_bytes` against which is also per process. Join [`mz_cluster_replica_history`](#mz_cluster_replica_history) for the size the episode ran at, since that row survives the replica. For how to use these -columns to choose a cluster size, see [Cluster sizing](/clusters/sizing/). +columns to choose a cluster size, see [Optimize cluster size](/clusters/sizing/). | Field | Type | Meaning | From 1736dc1198bdea37da2203c103a91f961f47fd47 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 19:28:54 +0000 Subject: [PATCH 4/9] docs: note that object hydration history skips sources Object history records only indexes and materialized views, so a source contributes no rows and does not hold a replica episode open. Upsert sources are the case where that matters, since they are memory-heavy enough that a reader would expect to find them. The distinction is easy to misread as "no metrics for upsert sources," so both notes say what is still measured: the replica peaks cover whole processes, so an upsert source's memory and disk are included there and a cluster hosting one is sized correctly by those peaks. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E6qZnfQsyVbQ2hydYaYder --- doc/user/content/clusters/sizing.md | 7 +++++++ doc/user/content/sql/system-catalog/mz_internal.md | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/doc/user/content/clusters/sizing.md b/doc/user/content/clusters/sizing.md index f3fa57b7ed391..864375f2202bf 100644 --- a/doc/user/content/clusters/sizing.md +++ b/doc/user/content/clusters/sizing.md @@ -189,6 +189,13 @@ approximate, it is approximate in ways that matter for sizing: always `hydrated`. A missing row is a signal in its own right, as in step 4, but it is never a measurement of a failure. +- **Only indexes and materialized views are tracked per object.** Sources, + including upsert sources, contribute no rows to object history and do not hold + a replica episode open. Their memory and disk usage is still counted in the + replica peaks, which measure whole processes, so a cluster hosting an upsert + source is sized correctly by those peaks even though the source itself never + appears per object. + - **Short-lived objects can be missed entirely.** Recording works by sampling each replica in a rotation, so an object that is dropped before its replica's turn leaves no trace. Nothing incorrect is recorded, the episode is simply diff --git a/doc/user/content/sql/system-catalog/mz_internal.md b/doc/user/content/sql/system-catalog/mz_internal.md index 7d013b2e704ce..d27f5e9766766 100644 --- a/doc/user/content/sql/system-catalog/mz_internal.md +++ b/doc/user/content/sql/system-catalog/mz_internal.md @@ -739,6 +739,12 @@ multi-process replica, timestamps come from process-local logging clocks and inc their clock skew. A process whose clock is ahead can be absent at the sampled logical timestamp, so the recorded finish can precede the latest process's finish. +Only indexes and materialized views are recorded. Sources, including upsert +sources, contribute no rows here and do not gate the completion of a replica +episode in [`mz_replica_hydration_history`](#mz_replica_hydration_history). +Their memory and disk usage is still reflected in that table's peaks, which +measure whole replica processes rather than individual dataflows. + `object_id` is a global ID rather than a catalog item ID, so join [`mz_object_global_ids`](#mz_object_global_ids) to reach the index or materialized view. To recover the name and size of a replica that has since been From 12ceac50335453674f75ebd79326681e44ce2cce Mon Sep 17 00:00:00 2001 From: Pranshu Maheshwari Date: Wed, 9 Sep 2026 15:45:00 -0400 Subject: [PATCH 5/9] Update sizing.md --- doc/user/content/clusters/sizing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user/content/clusters/sizing.md b/doc/user/content/clusters/sizing.md index 864375f2202bf..8a2e6fbf5d97e 100644 --- a/doc/user/content/clusters/sizing.md +++ b/doc/user/content/clusters/sizing.md @@ -1,5 +1,5 @@ --- -title: "Optimize cluster size" +title: "Optimize cluster sizes for hydration" description: "Optimize your cluster size by observing the resources it requires to hydrate." menu: main: From 64c494cbe9958cf47b1f43372a0d73fbbc46c381 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 20:22:00 +0000 Subject: [PATCH 6/9] docs: correct the hydration peak guarantees Two overstatements in the interpretation guidance, both caught in review and both confirmed against the collector. An episode's peaks are read from `mz_cluster_replica_resource_usage` at the sweep's read timestamp and the row is never revised, so they cover each process from start through the moment of collection and no further. Since an episode closes on the compute dataflows alone, an upsert source's snapshot usually runs past that point and its peak is never recorded. Saying such a cluster is "sized correctly by those peaks" invites exactly the under-provisioning this guide is meant to prevent, so the note now states the real horizon and sends readers to the sampled metrics history for snapshot-driven peaks. `peak_disk_bytes` is also not an upper bound. `statvfs` exposes no kernel high-water mark, so `fs_used_peak` is folded in-process as a maximum over samples, which `DERIVED_PEAKS` documents as a lower bound on the true peak, and the collector prefers that value whenever a scratch filesystem is present. The "never below the true peak" claim now applies only to `peak_memory_bytes`, which does come from a kernel high-water mark, and disk gets its own bullet saying to leave extra headroom. This also stops the guide contradicting the column comment on the relation it links to. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E6qZnfQsyVbQ2hydYaYder --- doc/user/content/clusters/sizing.md | 27 ++++++++++++------- .../content/sql/system-catalog/mz_internal.md | 5 ++-- 2 files changed, 21 insertions(+), 11 deletions(-) diff --git a/doc/user/content/clusters/sizing.md b/doc/user/content/clusters/sizing.md index 8a2e6fbf5d97e..572986d63b3ac 100644 --- a/doc/user/content/clusters/sizing.md +++ b/doc/user/content/clusters/sizing.md @@ -191,21 +191,30 @@ approximate, it is approximate in ways that matter for sizing: - **Only indexes and materialized views are tracked per object.** Sources, including upsert sources, contribute no rows to object history and do not hold - a replica episode open. Their memory and disk usage is still counted in the - replica peaks, which measure whole processes, so a cluster hosting an upsert - source is sized correctly by those peaks even though the source itself never - appears per object. + a replica episode open. The replica peaks measure whole processes, so they + include a source's memory and disk only for the work it had finished by the + moment the episode was recorded. An episode closes on the compute dataflows, + and [snapshotting](/fundamentals/concepts/snapshotting/) an upsert source + often runs well past that, so a cluster whose peak is driven by snapshotting + is not sized by these numbers. Read + [`mz_internal.mz_cluster_replica_metrics_history`](/sql/system-catalog/mz_internal/#mz_cluster_replica_metrics_history) + for that instead. - **Short-lived objects can be missed entirely.** Recording works by sampling each replica in a rotation, so an object that is dropped before its replica's turn leaves no trace. Nothing incorrect is recorded, the episode is simply absent. -- **The peaks are upper bounds on the episode.** They come from operating-system - high-water marks that cover each process's whole lifetime up to the moment the - episode is recorded, so post-hydration work can raise them, and a later - episode can inherit an earlier episode's mark. For sizing this errs the safe - way: the recorded value is never below the true hydration peak. +- **`peak_memory_bytes` is an upper bound on the episode.** It comes from the + kernel's own high-water mark, covering each process's whole lifetime up to the + moment the episode is recorded, so post-hydration work can raise it and a + later episode can inherit an earlier episode's mark. For sizing memory this + errs the safe way: the recorded value is never below the true hydration peak. + +- **`peak_disk_bytes` is a lower bound.** Where a scratch filesystem is in use + there is no kernel high-water mark to read, so the value is a maximum over + samples and can miss a spike between two of them. Leave more headroom on disk + than the number by itself implies. - **Timestamps can carry clock skew.** On a multi-process replica the endpoints of an interval come from different process clocks, so a recorded duration diff --git a/doc/user/content/sql/system-catalog/mz_internal.md b/doc/user/content/sql/system-catalog/mz_internal.md index d27f5e9766766..559314d47bb1a 100644 --- a/doc/user/content/sql/system-catalog/mz_internal.md +++ b/doc/user/content/sql/system-catalog/mz_internal.md @@ -742,8 +742,9 @@ logical timestamp, so the recorded finish can precede the latest process's finis Only indexes and materialized views are recorded. Sources, including upsert sources, contribute no rows here and do not gate the completion of a replica episode in [`mz_replica_hydration_history`](#mz_replica_hydration_history). -Their memory and disk usage is still reflected in that table's peaks, which -measure whole replica processes rather than individual dataflows. +That table's peaks measure whole replica processes rather than individual +dataflows, so they include a source's memory and disk only for work finished +before the episode was recorded. `object_id` is a global ID rather than a catalog item ID, so join [`mz_object_global_ids`](#mz_object_global_ids) to reach the index or From a5d2463a37e42fe7c5522647eb58ed85015e6d39 Mon Sep 17 00:00:00 2001 From: Pranshu Maheshwari Date: Wed, 9 Sep 2026 18:28:41 -0400 Subject: [PATCH 7/9] Update sizing.md --- doc/user/content/clusters/sizing.md | 36 ++++++++++++++--------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/doc/user/content/clusters/sizing.md b/doc/user/content/clusters/sizing.md index 572986d63b3ac..d84e2223ccb0d 100644 --- a/doc/user/content/clusters/sizing.md +++ b/doc/user/content/clusters/sizing.md @@ -1,6 +1,6 @@ --- -title: "Optimize cluster sizes for hydration" -description: "Optimize your cluster size by observing the resources it requires to hydrate." +title: "Size clusters for hydration" +description: "Measure the resources your cluster requires to hydrate, and optimize your cluster size accordingly" menu: main: parent: "clusters" @@ -28,9 +28,9 @@ replica that runs out of memory while hydrating restarts and tries again rather than running slower. {{< /note >}} -## Start large, then size down +## Determine the right size by starting large, and then size down -This guide assumes you are running Materialize v26.42 or later. v26.42 included +This guide assumes you are running Materialize v26.42 or later. v26.42 added improvements to allow you to track peak resource usage during hydration. ### 1. Create the cluster at a generous size @@ -63,13 +63,10 @@ An empty result means every object on the cluster is hydrated. A row with a -### 3. Read what the last hydration needed +### 3. Read what the last hydration required -Materialize records completed hydration episodes durably, so the numbers survive -the replica restart or resize that produced them. -[`mz_internal.mz_replica_hydration_history`](/sql/system-catalog/mz_internal/#mz_replica_hydration_history) -holds one row per replica-wide episode, with the resource high-water marks -observed for it: +Materialize records completed hydration episodes. [`mz_internal.mz_replica_hydration_history`](/sql/system-catalog/mz_internal/#mz_replica_hydration_history) +holds one row per replica-wide hydration episode: ```mzsql SELECT @@ -93,9 +90,11 @@ ORDER BY h.started_at DESC; (1 row) ``` -Compare `peak_memory` against the memory the candidate size provides, which -[`mz_catalog.mz_cluster_replica_sizes`](/sql/system-catalog/mz_catalog/#mz_cluster_replica_sizes) -reports per process, and leave headroom for the inputs to grow. +As the name suggests, `peak_memory` measures peak memory usage during the hydration event. + +Compare `peak_memory` against the replica sizes in +[`mz_catalog.mz_cluster_replica_sizes`](/sql/system-catalog/mz_catalog/#mz_cluster_replica_sizes), and use this to +determine the ideal cluster size. To find which object dominated the episode, read the per-object table, [`mz_internal.mz_object_hydration_history`](/sql/system-catalog/mz_internal/#mz_object_hydration_history). @@ -136,7 +135,7 @@ hydration peak stops dictating the size of everything else. ### 4. Size down -Once you have found the appropriate size, you can downsize by altering the +Once you have identified the appropriate size, you can downsize by altering the cluster: ```mzsql @@ -173,11 +172,10 @@ ORDER BY sh.occurred_at DESC LIMIT 10; ``` -Repeated `offline` rows with an out-of-memory `reason`, and no new episode in -hydration history, mean the size cannot rebuild the state. Go back to the size -that hydrated, and take a smaller step, or reduce the peak itself with one of -the [hydration -strategies](/fundamentals/concepts/hydration/#hydration-strategies). +If you see repeated `offline` rows with an out-of-memory `reason`, that means +the new size is too small. Size up, or consider using one of our [hydration +strategies](/fundamentals/concepts/hydration/#hydration-strategies) to reduce +the memory required for hydration. ## How should I interpret the hydration metrics? From 7c22e60df7b1ab37a77c385c8f8c612a1a4157c0 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 22:30:41 +0000 Subject: [PATCH 8/9] docs: strip trailing whitespace in the cluster sizing guide `check-whitespace` rejects it, so `lint-and-rustfmt` would have failed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E6qZnfQsyVbQ2hydYaYder --- doc/user/content/clusters/sizing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user/content/clusters/sizing.md b/doc/user/content/clusters/sizing.md index d84e2223ccb0d..177007268caa8 100644 --- a/doc/user/content/clusters/sizing.md +++ b/doc/user/content/clusters/sizing.md @@ -92,7 +92,7 @@ ORDER BY h.started_at DESC; As the name suggests, `peak_memory` measures peak memory usage during the hydration event. -Compare `peak_memory` against the replica sizes in +Compare `peak_memory` against the replica sizes in [`mz_catalog.mz_cluster_replica_sizes`](/sql/system-catalog/mz_catalog/#mz_cluster_replica_sizes), and use this to determine the ideal cluster size. From 6dfade53231daad5ae116188ed3f33699558ff52 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 23:05:29 +0000 Subject: [PATCH 9/9] docs: scope the peak_memory description to what the column holds `peak_memory_bytes` is the cgroup `memory.peak` high-water mark read at collection time, and nothing in the collector resets it, so it carries memory from before the episode and stops at the moment of recording. Describing it as memory usage "during the hydration event" contradicted the two bullets in this page's own interpretation section, which document both directions of error. State the horizon instead, and keep the point that matters for sizing: the value bounds the hydration peak from above rather than isolating it. The comparison against `mz_cluster_replica_sizes` also regained the per-process note. Both `memory_bytes` there and `peak_memory_bytes` here are per-process, so a multi-process size needs no multiplication. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01E6qZnfQsyVbQ2hydYaYder --- doc/user/content/clusters/sizing.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/user/content/clusters/sizing.md b/doc/user/content/clusters/sizing.md index 177007268caa8..d88fa273473c0 100644 --- a/doc/user/content/clusters/sizing.md +++ b/doc/user/content/clusters/sizing.md @@ -90,11 +90,15 @@ ORDER BY h.started_at DESC; (1 row) ``` -As the name suggests, `peak_memory` measures peak memory usage during the hydration event. +`peak_memory` is the highest memory any process on the replica reached, from +process start through the moment the episode was recorded. For sizing that is +the useful direction: it bounds the hydration peak rather than under-reporting +it. Compare `peak_memory` against the replica sizes in [`mz_catalog.mz_cluster_replica_sizes`](/sql/system-catalog/mz_catalog/#mz_cluster_replica_sizes), and use this to -determine the ideal cluster size. +determine the ideal cluster size. Both figures are per process, so on a +multi-process size do not multiply by `processes`. To find which object dominated the episode, read the per-object table, [`mz_internal.mz_object_hydration_history`](/sql/system-catalog/mz_internal/#mz_object_hydration_history).