diff --git a/doc/user/content/clusters/_index.md b/doc/user/content/clusters/_index.md index ed91451c25d4b..150fbcccf4797 100644 --- a/doc/user/content/clusters/_index.md +++ b/doc/user/content/clusters/_index.md @@ -12,4 +12,5 @@ Clusters provide the compute resources for running dataflows in Materialize. - Learn about [clusters](/fundamentals/concepts/clusters/). - Follow the [operational guidelines](/clusters/operational-guidelines/). +- Learn how to [optimize cluster size](/clusters/sizing/). - Understand [system clusters](/clusters/system-clusters/). diff --git a/doc/user/content/clusters/operational-guidelines/_index.md b/doc/user/content/clusters/operational-guidelines/_index.md index d514f6c95f296..16cf938c97634 100644 --- a/doc/user/content/clusters/operational-guidelines/_index.md +++ b/doc/user/content/clusters/operational-guidelines/_index.md @@ -66,7 +66,11 @@ For upsert sources, snapshotting is a resource-intensive operation that can requ ## Hydration considerations When sizing a cluster, budget for hydration memory on top of the steady-state -cost. The table below summarizes, per object type, when each object hydrates and +cost. Rather than estimating that budget, start at a size that hydrates +comfortably and size down once you have measured what hydration needed: see +[Optimize cluster size](/clusters/sizing/). + +The table below summarizes, per object type, when each object hydrates and the memory it uses. For more on hydration, including strategies to reduce its impact, see [Hydration](/fundamentals/concepts/hydration/). diff --git a/doc/user/content/clusters/sizing.md b/doc/user/content/clusters/sizing.md new file mode 100644 index 0000000000000..d88fa273473c0 --- /dev/null +++ b/doc/user/content/clusters/sizing.md @@ -0,0 +1,294 @@ +--- +title: "Size clusters for hydration" +description: "Measure the resources your cluster requires to hydrate, and optimize your cluster size accordingly" +menu: + main: + parent: "clusters" + weight: 5 + name: "Optimize cluster size" + identifier: "cluster-sizing" +--- + +A cluster's [size](/sql/create-cluster/#available-sizes) defines the CPU, +memory, and scratch disk available to every replica. On Materialize Cloud, this +determines the [cost](/materialize-cloud/billing/#compute) of the cluster. +Clusters should be provisioned for peak resource usage, to ensure that they can +handle the load placed on them. For most clusters, peak resource usage happens +during [hydration](/fundamentals/concepts/hydration/). + +This guide will walk you through how to estimate resources required for +hydration. Before reading this guide, make sure you understand the [lifecycle of +a cluster](/fundamentals/concepts/clusters/#lifecycle-of-a-cluster). + +{{< note >}} +Hydration rebuilds a dataflow's in-memory state from the storage layer, which +takes more memory than maintaining that state afterwards. The size that holds a +hydrated cluster is therefore not always the size that can rebuild it, and a +replica that runs out of memory while hydrating restarts and tries again rather +than running slower. +{{< /note >}} + +## Determine the right size by starting large, and then size down + +This guide assumes you are running Materialize v26.42 or later. v26.42 added +improvements to allow you to track peak resource usage during hydration. + +### 1. Create the cluster at a generous size + +Pick a size you are confident can hydrate the workload, even if it is clearly +more than steady state needs. + +```mzsql +CREATE CLUSTER analytics (SIZE = '400cc'); +``` + +Then create the cluster's indexes and materialized views as usual. + +### 2. Wait for the cluster to hydrate + +Every object has to finish hydrating before the numbers describe the whole +workload. Check that nothing is still hydrating: + +```mzsql +SELECT o.name AS object, h.replica_id, h.hydrated +FROM mz_internal.mz_hydration_statuses AS h +JOIN mz_catalog.mz_objects AS o ON o.id = h.object_id +JOIN mz_catalog.mz_clusters AS c ON c.id = o.cluster_id +WHERE c.name = 'analytics' AND h.hydrated IS NOT TRUE; +``` + +An empty result means every object on the cluster is hydrated. A row with a +`NULL` `replica_id` is an object that has not attached to a replica yet, which +`IS NOT TRUE` catches along with `hydrated = false`. + + + +### 3. Read what the last hydration required + +Materialize records completed hydration episodes. [`mz_internal.mz_replica_hydration_history`](/sql/system-catalog/mz_internal/#mz_replica_hydration_history) +holds one row per replica-wide hydration episode: + +```mzsql +SELECT + rh.replica_name AS replica, + rh.size, + h.started_at, + h.finished_at - h.started_at AS hydration_time, + h.object_count, + pg_size_pretty(h.peak_memory_bytes) AS peak_memory, + pg_size_pretty(h.peak_disk_bytes) AS peak_disk +FROM mz_internal.mz_replica_hydration_history AS h +JOIN mz_internal.mz_cluster_replica_history AS rh ON rh.replica_id = h.replica_id +WHERE rh.cluster_name = 'analytics' +ORDER BY h.started_at DESC; +``` + +```none + replica | size | started_at | hydration_time | object_count | peak_memory | peak_disk +---------+-------+-------------------------------+----------------+--------------+-------------+----------- + r1 | 400cc | 2026-09-08 09:12:04.117841+00 | 00:04:11.83 | 41 | 11 GB | 2438 MB +(1 row) +``` + +`peak_memory` is the highest memory any process on the replica reached, from +process start through the moment the episode was recorded. For sizing that is +the useful direction: it bounds the hydration peak rather than under-reporting +it. + +Compare `peak_memory` against the replica sizes in +[`mz_catalog.mz_cluster_replica_sizes`](/sql/system-catalog/mz_catalog/#mz_cluster_replica_sizes), and use this to +determine the ideal cluster size. Both figures are per process, so on a +multi-process size do not multiply by `processes`. + +To find which object dominated the episode, read the per-object table, +[`mz_internal.mz_object_hydration_history`](/sql/system-catalog/mz_internal/#mz_object_hydration_history). +Its `object_id` is the ID of the object's dataflow, so reach the catalog item +through +[`mz_internal.mz_object_global_ids`](/sql/system-catalog/mz_internal/#mz_object_global_ids): + +```mzsql +SELECT + rh.replica_name AS replica, + o.name AS object, + o.type, + h.hydrated_at - h.installed_at AS hydration_time +FROM mz_internal.mz_object_hydration_history AS h +JOIN mz_internal.mz_object_global_ids AS g ON g.global_id = h.object_id +JOIN mz_catalog.mz_objects AS o ON o.id = g.id +JOIN mz_internal.mz_cluster_replica_history AS rh ON rh.replica_id = h.replica_id +WHERE rh.cluster_name = 'analytics' +ORDER BY hydration_time DESC +LIMIT 5; +``` + +```none + replica | object | type | hydration_time +---------+---------------------+-------------------+----------------- + r1 | auction_summary | materialized-view | 00:04:11.83 + r1 | bids_by_auction | materialized-view | 00:01:47.21 + r1 | bids_by_auction_idx | index | 00:00:22.04 + r1 | auction_summary_idx | index | 00:00:19.88 +(4 rows) +``` + +Every replica records its own rows, so a cluster with a replication factor +above one, or one that has been resized, returns a row per object per replica. +If one object accounts for most of the episode, [move it to its own +cluster](/fundamentals/concepts/hydration/#hydration-strategies) so its +hydration peak stops dictating the size of everything else. + +### 4. Size down + +Once you have identified the appropriate size, you can downsize by altering the +cluster: + +```mzsql +ALTER CLUSTER analytics SET (SIZE = '100cc'); +``` + +`ALTER CLUSTER` operations are graceful. This means the smaller cluster will +hydrate in parallel, and Materialize will cut over to the smaller cluster when +it is ready. + +The resize also hydrates the whole workload again, which produces exactly the +measurement you need to confirm the new size. Re-run the query from [step +3](#read-what-the-last-hydration-needed) once the new replica is hydrated: + +```none + replica | size | started_at | hydration_time | object_count | peak_memory | peak_disk +---------+-------+-------------------------------+----------------+--------------+-------------+----------- + r2 | 100cc | 2026-09-08 10:41:22.913044+00 | 00:12:37.42 | 41 | 12 GB | 4310 MB + r1 | 400cc | 2026-09-08 09:12:04.117841+00 | 00:04:11.83 | 41 | 11 GB | 2438 MB +(2 rows) +``` + +If the new size is too small, no completed episode is recorded for the new +replica at all. Only successful hydration is recorded, so an out-of-memory +restart loop shows up as a missing row plus repeated restarts in +[`mz_internal.mz_cluster_replica_status_history`](/sql/system-catalog/mz_internal/#mz_cluster_replica_status_history): + +```mzsql +SELECT sh.occurred_at, sh.process_id, sh.status, sh.reason +FROM mz_internal.mz_cluster_replica_status_history AS sh +JOIN mz_internal.mz_cluster_replica_history AS rh ON rh.replica_id = sh.replica_id +WHERE rh.cluster_name = 'analytics' +ORDER BY sh.occurred_at DESC +LIMIT 10; +``` + +If you see repeated `offline` rows with an out-of-memory `reason`, that means +the new size is too small. Size up, or consider using one of our [hydration +strategies](/fundamentals/concepts/hydration/#hydration-strategies) to reduce +the memory required for hydration. + +## How should I interpret the hydration metrics? + +Hydration history is a best-effort record, not an audit log. Where it is +approximate, it is approximate in ways that matter for sizing: + +- **Only successful episodes are recorded.** There is no row for a hydration + that was killed, canceled, or is still running, and `status` is currently + always `hydrated`. A missing row is a signal in its own right, as in step 4, + but it is never a measurement of a failure. + +- **Only indexes and materialized views are tracked per object.** Sources, + including upsert sources, contribute no rows to object history and do not hold + a replica episode open. The replica peaks measure whole processes, so they + include a source's memory and disk only for the work it had finished by the + moment the episode was recorded. An episode closes on the compute dataflows, + and [snapshotting](/fundamentals/concepts/snapshotting/) an upsert source + often runs well past that, so a cluster whose peak is driven by snapshotting + is not sized by these numbers. Read + [`mz_internal.mz_cluster_replica_metrics_history`](/sql/system-catalog/mz_internal/#mz_cluster_replica_metrics_history) + for that instead. + +- **Short-lived objects can be missed entirely.** Recording works by sampling + each replica in a rotation, so an object that is dropped before its replica's + turn leaves no trace. Nothing incorrect is recorded, the episode is simply + absent. + +- **`peak_memory_bytes` is an upper bound on the episode.** It comes from the + kernel's own high-water mark, covering each process's whole lifetime up to the + moment the episode is recorded, so post-hydration work can raise it and a + later episode can inherit an earlier episode's mark. For sizing memory this + errs the safe way: the recorded value is never below the true hydration peak. + +- **`peak_disk_bytes` is a lower bound.** Where a scratch filesystem is in use + there is no kernel high-water mark to read, so the value is a maximum over + samples and can miss a spike between two of them. Leave more headroom on disk + than the number by itself implies. + +- **Timestamps can carry clock skew.** On a multi-process replica the endpoints + of an interval come from different process clocks, so a recorded duration + includes their skew. This is not usually visible at the minute scale that + matters for sizing. + +- **Rows outlive what they name.** `replica_id`, `cluster_id`, and `object_id` + may all name objects that no longer exist, which is what makes the history + useful across a resize. Join `mz_cluster_replica_history` for replica and + cluster names, and expect the join through `mz_object_global_ids` to drop + objects that have since been dropped. + +- **Rows are retained for 30 days by default.** Sizing decisions should come + from the recent history rather than the earliest episode still stored. + +## What should I do if hydration history is empty? + +Recording is controlled by the `hydration_history_collection_interval` system +parameter, which sets how often Materialize samples replicas for completed +episodes. A value of zero disables recording, and the tables then stay as they +are: rows already collected remain, and no new ones are added. +`hydration_history_retention_period` bounds how long rows live, and defaults to +30 days. + +On Materialize Cloud, these parameters are managed for you. If both tables are +empty for a cluster that has certainly hydrated, contact +[support](/support/). + +On Materialize Self-Managed, set them as the `mz_system` user, or through the +[system parameters +ConfigMap](/self-managed-deployments/configuration-system-parameters/): + +```mzsql +ALTER SYSTEM SET hydration_history_collection_interval = '60s'; +``` + +A shorter interval records episodes sooner, at the cost of installing a dataflow +on a replica more often. Recording visits one replica per interval, so an +environment with many replicas revisits each one proportionally less often. + +Until history is available, the current-state relations still answer the +narrower question of what is happening now: + +| Relation | What it gives you | +|----------|-------------------| +| [`mz_internal.mz_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_hydration_statuses) | Per-object, per-replica hydration flag, for every object type. | +| [`mz_internal.mz_compute_hydration_statuses`](/sql/system-catalog/mz_internal/#mz_compute_hydration_statuses) | The same flag plus how long hydration took, for indexes and materialized views. | +| [`mz_internal.mz_cluster_replica_metrics_history`](/sql/system-catalog/mz_internal/#mz_cluster_replica_metrics_history) | CPU, memory, and disk sampled about once a minute, retained for 30 days. | + +The two hydration relations report only the current state and are reset by a +replica or Materialize restart. The metrics history survives restarts, but at +roughly one sample a minute it can miss a hydration spike entirely, and it does +not tell you which episode a sample belonged to. That is why these are a +fallback rather than the basis for a sizing decision. + +## How do I speed up hydration? + +Hydration speed scales with cluster size, so a cluster can borrow capacity for +hydration alone rather than running at the larger size permanently. An [`AUTO +SCALING STRATEGY (ON +HYDRATION)`](/sql/alter-cluster/#speed-up-hydration-by-autoscaling-to-a-larger-size) +provisions an extra burst replica at a larger size whenever the cluster has +un-hydrated objects, and removes it once a steady-size replica catches up. + +To reduce the work hydration has to do in the first place, see [hydration +strategies](/fundamentals/concepts/hydration/#hydration-strategies). + +## Related pages + +- [Hydration](/fundamentals/concepts/hydration/) +- [Clusters](/fundamentals/concepts/clusters/) +- [Operational guidelines](/clusters/operational-guidelines/) +- [`CREATE CLUSTER`](/sql/create-cluster/) +- [`ALTER CLUSTER`](/sql/alter-cluster/) +- [Usage & billing](/materialize-cloud/billing/) diff --git a/doc/user/content/fundamentals/concepts/clusters.md b/doc/user/content/fundamentals/concepts/clusters.md index 762f4ef56e0cc..224a9117ee385 100644 --- a/doc/user/content/fundamentals/concepts/clusters.md +++ b/doc/user/content/fundamentals/concepts/clusters.md @@ -178,6 +178,11 @@ The light view and index hydrate in microseconds, while the larger `bids_load` view takes about 5.6 seconds. A larger object with more state to reconstruct shows a longer, more visible hydration window. +Both relations report only the current state, so they are wiped when a replica +or Materialize restarts. To compare this hydration against earlier ones, and to +see the memory and disk it needed, read the durable hydration history described +in [Optimize cluster size](/clusters/sizing/). + ### Catching up Once hydrated, the cluster processes the backlog of input updates that @@ -265,6 +270,10 @@ resize triggers [hydration](#hydration-considerations). During hydration, the cluster keeps serving since Materialize provisions new replicas at the target size and hydrates them before retiring the old ones. +Because peak resource usage normally happens during hydration, size a cluster +for the resources hydration needs. For how to measure those, see [Optimize +cluster size](/clusters/sizing/). + ## Hydration considerations {{% include-from-yaml data="hydration-details" name="definition" %}} diff --git a/doc/user/content/fundamentals/concepts/hydration.md b/doc/user/content/fundamentals/concepts/hydration.md index 8d73ee4e489cf..01c355819dc76 100644 --- a/doc/user/content/fundamentals/concepts/hydration.md +++ b/doc/user/content/fundamentals/concepts/hydration.md @@ -28,6 +28,16 @@ table. {{% yaml-table data="hydration-objects-table" %}} +## Measuring hydration + +How much memory and time hydration needs depends on the data, not just on the +query, so the reliable way to find out is to measure a hydration you have +already run. Materialize records completed hydration episodes durably, per +object and per replica, including the resource high-water marks observed for +each replica episode. Those records outlive the replica restart or resize that +produced them, which is what makes them usable for sizing a cluster. See +[Optimize cluster size](/clusters/sizing/). + ## Hydration strategies Hydration primarily impacts memory usage, and its speed scales with cluster diff --git a/doc/user/content/sql/system-catalog/mz_catalog.md b/doc/user/content/sql/system-catalog/mz_catalog.md index d6ebd9fe49fab..831edab8f3ed8 100644 --- a/doc/user/content/sql/system-catalog/mz_catalog.md +++ b/doc/user/content/sql/system-catalog/mz_catalog.md @@ -111,7 +111,7 @@ any kind of capacity planning. | `processes` | [`uint8`] | The number of processes in the replica. | | `workers` | [`uint8`] | The number of Timely Dataflow workers per process. | | `cpu_nano_cores` | [`uint8`] | The CPU allocation per process, in billionths of a vCPU core. | -| `memory_bytes` | [`uint8`] | The RAM allocation per process, in billionths of a vCPU core. | +| `memory_bytes` | [`uint8`] | The RAM allocation per process, in bytes. | | `disk_bytes` | [`uint8`] | The disk allocation per process. | | `credits_per_hour` | [`numeric`] | The number of compute credits consumed per hour. | diff --git a/doc/user/content/sql/system-catalog/mz_internal.md b/doc/user/content/sql/system-catalog/mz_internal.md index d36965fa807ed..559314d47bb1a 100644 --- a/doc/user/content/sql/system-catalog/mz_internal.md +++ b/doc/user/content/sql/system-catalog/mz_internal.md @@ -723,6 +723,8 @@ The `mz_object_history` view enriches the [`mz_catalog.mz_objects`](/sql/system- ## `mz_object_hydration_history` +{{< warn-if-unreleased v26.42 >}} + The `mz_object_hydration_history` table records completed hydration of indexes and materialized views, with one row for each time a dataflow hydrated on a replica. By default, rows are retained for 30 days while collection is enabled. Disabling @@ -737,6 +739,20 @@ multi-process replica, timestamps come from process-local logging clocks and inc their clock skew. A process whose clock is ahead can be absent at the sampled logical timestamp, so the recorded finish can precede the latest process's finish. +Only indexes and materialized views are recorded. Sources, including upsert +sources, contribute no rows here and do not gate the completion of a replica +episode in [`mz_replica_hydration_history`](#mz_replica_hydration_history). +That table's peaks measure whole replica processes rather than individual +dataflows, so they include a source's memory and disk only for work finished +before the episode was recorded. + +`object_id` is a global ID rather than a catalog item ID, so join +[`mz_object_global_ids`](#mz_object_global_ids) to reach the index or +materialized view. To recover the name and size of a replica that has since been +replaced, join [`mz_cluster_replica_history`](#mz_cluster_replica_history). For +how to use these columns to choose a cluster size, see [Optimize cluster +size](/clusters/sizing/). + | Field | Type | Meaning | | -------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------ | @@ -750,6 +766,8 @@ logical timestamp, so the recorded finish can precede the latest process's finis ## `mz_replica_hydration_history` +{{< warn-if-unreleased v26.42 >}} + The `mz_replica_hydration_history` table records successful replica hydration episodes. An episode begins when a maintained compute dataflow is installed on a fully hydrated replica and finishes when every running maintained compute @@ -761,6 +779,14 @@ is recorded. Resource peaks cover the replica processes' lifetimes through collection, not only the hydration episode. On a multi-process replica, the table records the largest peak reported by any process. +Because memory and disk limits apply to each replica process independently, +compare `peak_memory_bytes` against +[`mz_cluster_replica_sizes.memory_bytes`](/sql/system-catalog/mz_catalog/#mz_cluster_replica_sizes), +which is also per process. Join +[`mz_cluster_replica_history`](#mz_cluster_replica_history) for the size the +episode ran at, since that row survives the replica. For how to use these +columns to choose a cluster size, see [Optimize cluster size](/clusters/sizing/). + | Field | Type | Meaning | | ------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------ | @@ -768,7 +794,7 @@ table records the largest peak reported by any process. | `cluster_id` | [`text`] | The ID of the replica's cluster. | | `started_at` | [`timestamp with time zone`] | The earliest maintained compute dataflow installation in the hydration episode. | | `finished_at` | [`timestamp with time zone`] | The latest maintained compute dataflow hydration in the hydration episode. | -| `object_count` | [`uint8`] | The number of maintained compute dataflows in the hydration episode. | +| `object_count` | [`uint8`] | The number of maintained compute dataflows in the hydration episode. Includes the replica's system introspection dataflows, so it exceeds the number of indexes and materialized views you created. | | `peak_memory_bytes` | [`uint8`] | The largest process-lifetime cgroup memory high-water mark reported by any process when the collector recorded the episode. `NULL` if the platform reports no cgroup memory peak. | | `peak_disk_bytes` | [`uint8`] | The largest process-lifetime scratch-filesystem or swap high-water mark reported by any process when the collector recorded the episode. Filesystem peaks are sampled lower bounds. `NULL` if neither measurement is available. | | `status` | [`text`] | The hydration episode's status. Currently always `hydrated`. | diff --git a/src/catalog/src/builtin/mz_catalog.rs b/src/catalog/src/builtin/mz_catalog.rs index bdd9cd3969b5e..1e77a097bbddf 100644 --- a/src/catalog/src/builtin/mz_catalog.rs +++ b/src/catalog/src/builtin/mz_catalog.rs @@ -2804,10 +2804,7 @@ pub static MZ_CLUSTER_REPLICA_SIZES: LazyLock = LazyLock::new(|| B "cpu_nano_cores", "The CPU allocation per process, in billionths of a vCPU core.", ), - ( - "memory_bytes", - "The RAM allocation per process, in billionths of a vCPU core.", - ), + ("memory_bytes", "The RAM allocation per process, in bytes."), ("disk_bytes", "The disk allocation per process."), ( "credits_per_hour", diff --git a/src/catalog/src/builtin/mz_internal.rs b/src/catalog/src/builtin/mz_internal.rs index 006492fbdf5d2..2cb92ab492b86 100644 --- a/src/catalog/src/builtin/mz_internal.rs +++ b/src/catalog/src/builtin/mz_internal.rs @@ -5234,7 +5234,7 @@ pub static MZ_REPLICA_HYDRATION_HISTORY: LazyLock = LazyLock::new( ), ( "object_count", - "The number of maintained compute dataflows in the hydration episode.", + "The number of maintained compute dataflows in the hydration episode. Includes the replica's system introspection dataflows, so it exceeds the number of indexes and materialized views you created.", ), ( "peak_memory_bytes", diff --git a/test/sqllogictest/autogenerated/mz_catalog.slt b/test/sqllogictest/autogenerated/mz_catalog.slt index 4079dd2ff2592..35d6eff460b69 100644 --- a/test/sqllogictest/autogenerated/mz_catalog.slt +++ b/test/sqllogictest/autogenerated/mz_catalog.slt @@ -76,7 +76,7 @@ size text The␠human-readable␠replica␠size. processes uint8 The␠number␠of␠processes␠in␠the␠replica. workers uint8 The␠number␠of␠Timely␠Dataflow␠workers␠per␠process. cpu_nano_cores uint8 The␠CPU␠allocation␠per␠process,␠in␠billionths␠of␠a␠vCPU␠core. -memory_bytes uint8 The␠RAM␠allocation␠per␠process,␠in␠billionths␠of␠a␠vCPU␠core. +memory_bytes uint8 The␠RAM␠allocation␠per␠process,␠in␠bytes. disk_bytes uint8 The␠disk␠allocation␠per␠process. credits_per_hour numeric The␠number␠of␠compute␠credits␠consumed␠per␠hour. diff --git a/test/sqllogictest/autogenerated/mz_internal.slt b/test/sqllogictest/autogenerated/mz_internal.slt index 66cd3a52e3a4e..679787936fa86 100644 --- a/test/sqllogictest/autogenerated/mz_internal.slt +++ b/test/sqllogictest/autogenerated/mz_internal.slt @@ -406,7 +406,7 @@ replica_id text The␠ID␠of␠the␠cluster␠replica.␠May␠name␠a␠re cluster_id text The␠ID␠of␠the␠replica's␠cluster. started_at timestamp␠with␠time␠zone The␠earliest␠maintained␠compute␠dataflow␠installation␠in␠the␠hydration␠episode. finished_at timestamp␠with␠time␠zone The␠latest␠maintained␠compute␠dataflow␠hydration␠in␠the␠hydration␠episode. -object_count uint8 The␠number␠of␠maintained␠compute␠dataflows␠in␠the␠hydration␠episode. +object_count uint8 The␠number␠of␠maintained␠compute␠dataflows␠in␠the␠hydration␠episode.␠Includes␠the␠replica's␠system␠introspection␠dataflows,␠so␠it␠exceeds␠the␠number␠of␠indexes␠and␠materialized␠views␠you␠created. peak_memory_bytes uint8 The␠largest␠process-lifetime␠cgroup␠memory␠high-water␠mark␠reported␠by␠any␠process␠when␠the␠collector␠recorded␠the␠episode.␠`NULL`␠if␠the␠platform␠reports␠no␠cgroup␠memory␠peak. peak_disk_bytes uint8 The␠largest␠process-lifetime␠scratch-filesystem␠or␠swap␠high-water␠mark␠reported␠by␠any␠process␠when␠the␠collector␠recorded␠the␠episode.␠Filesystem␠peaks␠are␠sampled␠lower␠bounds.␠`NULL`␠if␠neither␠measurement␠is␠available. status text The␠hydration␠episode's␠status.␠Currently␠always␠`hydrated`.