From 6c0c054d55f5eb155291ae4c9712cf23c12c8936 Mon Sep 17 00:00:00 2001 From: Tyler Hartwig Date: Thu, 10 Sep 2026 16:54:27 -0400 Subject: [PATCH 1/4] docs: serving during source stalls Document which query shapes Materialize can keep serving while an upstream source is stalled, by isolation level, plus the patterns that keep serving available through an upstream outage: * maintain cross-source queries as indexed/materialized views and read them directly (single-collection reads serve stale), and * the frontier-alignment workaround: a pre-created materialized view over the same inputs holds their read frontiers together so ad hoc slow-path queries can still pick a valid timestamp. Also expands the stalled-source subsection of the serving troubleshooting page with a pointer to the new page. Every behavior claim was verified empirically on v26.36.0 with two independent stall methods (ingestion cluster at replication factor 0, and stopping the upstream Kafka broker), including the negative cases: an aligner created after the stall does not help, reads of stalled data in explicit transactions block (healthy-only transactions keep serving), and SUBSCRIBE over a stalled+live join blocks like the equivalent SELECT. Co-Authored-By: Claude Fable 5 --- .../content/serve-results/source-stalls.md | 185 ++++++++++++++++++ .../content/serve-results/troubleshooting.md | 9 + 2 files changed, 194 insertions(+) create mode 100644 doc/user/content/serve-results/source-stalls.md diff --git a/doc/user/content/serve-results/source-stalls.md b/doc/user/content/serve-results/source-stalls.md new file mode 100644 index 0000000000000..a30a7504bf6f1 --- /dev/null +++ b/doc/user/content/serve-results/source-stalls.md @@ -0,0 +1,185 @@ +--- +title: "Serving during source stalls" +description: "Which queries Materialize can keep serving while an upstream source is stalled, and how to design for availability during upstream outages." +menu: + main: + name: "Serve during source stalls" + parent: serve-results + identifier: 'serve-results-source-stalls' + weight: 16 +--- + +When an upstream system becomes unavailable (a Kafka broker outage, a paused +replication slot, an ingestion cluster with no running replicas), the affected +sources stop making progress, or **stall**. A stalled source does not have to +stop your application from reading: in many cases, Materialize keeps serving +queries from the last consistently ingested data. + +Whether a given query keeps serving depends on two things: the +[isolation level](/serve-results/isolation-level/) of the session, and the +**shape of the query**. This page describes the behavior and the patterns that +keep serving available through an upstream outage. + +## How a stall affects queries + +Materialize serves every query at a single logical timestamp that must be +valid for **all** of the query's inputs: for each input collection, the +timestamp must lie between its [read frontier and write +frontier](/sql/explain-timestamp/#details). When a source stalls, the write +frontiers of the collections that depend on it freeze at the moment of the +stall, while unaffected collections keep advancing. + +Queries confined to the stalled data can still be served at the frozen +timestamp. The results are stale, but consistent. Queries that mix stalled and +still-advancing inputs may find that no common timestamp exists, in which case +the query **blocks** until one does (typically, when the source resumes). + +## Behavior by query shape + +Under the **serializable** isolation level, queries whose inputs stalled +together keep serving; queries that mix stalled and live inputs block. Under +**strict serializable**, any query that reads stalled data blocks, because the +query timestamp must also reflect real-time recent writes. Queries that read +no stalled data at all are unaffected under either isolation level. + +| Query shape | Serializable | Strict serializable | +| ----------- | ------------ | ------------------- | +| Point lookup on an index over stalled data | Serves stale | Blocks | +| Direct read of a table created from the stalled source (`CREATE TABLE ... FROM SOURCE`) | Serves stale | Blocks | +| Aggregation or full scan over one stalled collection | Serves stale | Blocks | +| Query whose inputs all stalled together (e.g., views over the same source) | Serves stale | Blocks | +| `SUBSCRIBE` to a single stalled collection | Serves stale | Blocks | +| Join between a stalled collection and a healthy one | Blocks | Blocks | +| Query mixing a stalled collection and a user-writable table | Blocks | Blocks | +| Read of stalled data inside an explicit transaction | Blocks | Blocks | + +Blocked queries **wait**; they do not error. A blocked query completes once +the source resumes and its write frontier passes the query's timestamp, with +no work lost. Note that `statement_timeout` does not interrupt this wait: it +cancels queries that are executing, but does not fire while a query is waiting +for its timestamp to become available. Use client-side timeouts to bound the +wait. + +To see why a specific query does or does not serve, use [`EXPLAIN +TIMESTAMP`](/sql/explain-timestamp/): it reports `can respond immediately: +true/false` along with the read and write frontiers of every input. + +## Keep serving across sources: maintain the query + +The recommended pattern for queries that span sources is to maintain the query +as an [indexed view](/concepts/views/#indexes-on-views) or [materialized +view](/concepts/views/#materialized-views), and have the application read from +that object directly: + +```mzsql +CREATE VIEW order_enrichment AS + SELECT o.id, o.total, c.name, c.region + FROM kafka_orders o + JOIN pg_customers c ON o.customer_id = c.id; + +CREATE INDEX order_enrichment_idx ON order_enrichment (id); +``` + +The maintained object's write frontier follows its *slowest* input, so when +one of the sources stalls, the object as a whole freezes consistently. Reading +it is then a single-collection query, so it keeps serving stale results for as +long as the stall lasts. This is also the recommended pattern for query +latency in general, since point lookups on the index are served directly from +memory. + +## Keep ad hoc queries serving: align frontiers with a maintained object + +If your application must issue **ad hoc** queries that reference multiple +collections (for example, generated queries from a BI tool joining a stalled +source to a live one), you can keep those queries servable during a stall by +maintaining *any* object that reads the same set of inputs: + +```mzsql +-- A small maintained object whose only purpose is to hold the read +-- frontiers of its inputs together. +CREATE MATERIALIZED VIEW frontier_alignment AS + SELECT max(id) FROM ( + SELECT id FROM kafka_orders + UNION ALL + SELECT id FROM pg_customers + ); +``` + +Because the maintained object must remain readable at its own (frozen) write +frontier, Materialize holds back the read frontiers of **all** of its inputs. +That keeps the stalled and live collections' frontier intervals overlapping, +so an ad hoc query over any subset of those inputs can still select a valid +timestamp and serve stale results instead of blocking. + +For this to work, note: + +- **The object must exist before the stall.** Once the live inputs' read + frontiers have advanced past the stalled collection's write frontier, + compaction has already discarded the historical data. Creating the aligning + object after the fact does not help, and the new object itself cannot serve + until the source resumes. + +- **The object must genuinely read every input you want covered.** The + optimizer removes inputs it can prove are unused (for example, behind + `WHERE false`). Check `EXPLAIN` on the object's definition to confirm all + intended inputs appear in the plan. + +- **Holding back read frontiers has a cost.** For the duration of the stall, + Materialize retains historical data for the covered inputs that it would + otherwise compact away. + +- **Reads of stalled data inside explicit transactions are not rescued** + (see below). + +## Explicit transactions + +An explicit read transaction ([`BEGIN ... COMMIT`](/sql/begin/)) does not take +its read set from the queries inside it: it pins a +[timedomain](/sql/begin/#same-timedomain-error) covering every object in the +referenced schemas, plus the system catalog. The transaction timestamp must +sit above the read frontiers of everything in that timedomain, and those +frontiers keep advancing during a stall. As a consequence, once the stall +outlasts the compaction window, **any read of stalled data inside an explicit +transaction blocks**, including query shapes that serve fine outside a +transaction, such as an indexed point lookup. + +Transactions that read only healthy objects keep serving, under both +isolation levels, even when a stalled source exists in the same schema. + +For reads of stalled data during a stall, issue single-statement queries, or +use [`SUBSCRIBE`](/sql/subscribe/) with a cursor. `SUBSCRIBE` does not pin a +timedomain, so subscribing to a single stalled collection serves stale even +inside a transaction. `SUBSCRIBE` remains subject to the query-shape rules +above, however: a subscription over a join of stalled and live inputs blocks +just like the equivalent `SELECT`. + +{{< if-released "v26.29" >}} +## Fail fast instead of blocking + +If your application would rather receive an error than wait, or than serve +data past a staleness threshold, use the [bounded +staleness](/serve-results/isolation-level/#bounded-staleness) isolation +level. During a stall: + +- Query shapes that serve stale under serializable also serve stale under + bounded staleness, as long as the stall is younger than the configured + bound. + +- Query shapes that would block under serializable, and any query reading + stalled data once the stall exceeds the bound, **error immediately** with + `SQLSTATE 40001` instead of blocking, giving the application a clean signal + to retry or fall back. + +```mzsql +SET TRANSACTION_ISOLATION TO 'bounded staleness 1m'; +``` +{{< /if-released >}} + +## Related pages + +- [Isolation levels](/serve-results/isolation-level/) +- [`EXPLAIN TIMESTAMP`](/sql/explain-timestamp/) +- [`BEGIN`](/sql/begin/) +- [`SUBSCRIBE`](/sql/subscribe/) +- [Troubleshooting serving](/serve-results/troubleshooting/) +- [Ingest data: troubleshooting](/ingest-data/troubleshooting/) diff --git a/doc/user/content/serve-results/troubleshooting.md b/doc/user/content/serve-results/troubleshooting.md index c0d778b6f4e80..6eb2ce6b99391 100644 --- a/doc/user/content/serve-results/troubleshooting.md +++ b/doc/user/content/serve-results/troubleshooting.md @@ -222,6 +222,15 @@ guidance. To detect and address stalled sources, follow the [`Ingest data` troubleshooting](/ingest-data/troubleshooting) guide. +While a source is stalled, whether a query blocks or keeps serving (stale) +results depends on the isolation level and the shape of the query. Queries that +read only data that stalled together keep serving under the `serializable` +isolation level, while queries that mix stalled and still-advancing inputs +block, as does any query that reads stalled data under `strict serializable` +or inside an explicit transaction. For the full behavior matrix and patterns +that keep serving available through an upstream outage, see [Serving during +source stalls](/serve-results/source-stalls/). + ### Snapshotting source When a source is created, it must first _snapshot_ the existing data from the From c898920fa5412f4914c92e80f3cee10f841c7136 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 06:27:04 +0000 Subject: [PATCH 2/4] docs: address review feedback on source-stalls guide Rephrases the frontier-overlap explanation with a concrete two-source example and pulls the read/write frontier definitions into a note, renames the query-shape section to mention isolation level, and collapses "Explicit transactions" into a one-line "Don't use transactions" callout per review. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01H5FCMmD7mTXkmcc85p245z --- .../content/serve-results/source-stalls.md | 45 +++++++------------ 1 file changed, 17 insertions(+), 28 deletions(-) diff --git a/doc/user/content/serve-results/source-stalls.md b/doc/user/content/serve-results/source-stalls.md index a30a7504bf6f1..aa7ff1dfd4548 100644 --- a/doc/user/content/serve-results/source-stalls.md +++ b/doc/user/content/serve-results/source-stalls.md @@ -22,19 +22,25 @@ keep serving available through an upstream outage. ## How a stall affects queries -Materialize serves every query at a single logical timestamp that must be -valid for **all** of the query's inputs: for each input collection, the -timestamp must lie between its [read frontier and write -frontier](/sql/explain-timestamp/#details). When a source stalls, the write -frontiers of the collections that depend on it freeze at the moment of the -stall, while unaffected collections keep advancing. +Materialize serves every query at a single logical timestamp. This means that +each input must have overlapping timestamps. Imagine you have two sources, A +and B. When source A stalls, its timestamp is frozen at the point of the +stall, but source B will continue advancing. + +{{< note >}} +Every collection has a **read frontier** (the earliest timestamp it can still +answer correctly, advanced by compaction) and a **write frontier** (all data +before this point has been fully processed). A query's timestamp must fall +between the [read and write frontiers](/sql/explain-timestamp/#details) of +every input it reads. +{{< /note >}} Queries confined to the stalled data can still be served at the frozen timestamp. The results are stale, but consistent. Queries that mix stalled and still-advancing inputs may find that no common timestamp exists, in which case the query **blocks** until one does (typically, when the source resumes). -## Behavior by query shape +## Behavior by query shape and isolation level Under the **serializable** isolation level, queries whose inputs stalled together keep serving; queries that mix stalled and live inputs block. Under @@ -131,27 +137,10 @@ For this to work, note: - **Reads of stalled data inside explicit transactions are not rescued** (see below). -## Explicit transactions - -An explicit read transaction ([`BEGIN ... COMMIT`](/sql/begin/)) does not take -its read set from the queries inside it: it pins a -[timedomain](/sql/begin/#same-timedomain-error) covering every object in the -referenced schemas, plus the system catalog. The transaction timestamp must -sit above the read frontiers of everything in that timedomain, and those -frontiers keep advancing during a stall. As a consequence, once the stall -outlasts the compaction window, **any read of stalled data inside an explicit -transaction blocks**, including query shapes that serve fine outside a -transaction, such as an indexed point lookup. - -Transactions that read only healthy objects keep serving, under both -isolation levels, even when a stalled source exists in the same schema. - -For reads of stalled data during a stall, issue single-statement queries, or -use [`SUBSCRIBE`](/sql/subscribe/) with a cursor. `SUBSCRIBE` does not pin a -timedomain, so subscribing to a single stalled collection serves stale even -inside a transaction. `SUBSCRIBE` remains subject to the query-shape rules -above, however: a subscription over a join of stalled and live inputs blocks -just like the equivalent `SELECT`. +## Don't use transactions + +Explicit read transactions that touch stalled data are unable to serve +during a stall. Issue single-statement queries instead. {{< if-released "v26.29" >}} ## Fail fast instead of blocking From a102a30a3ba10ef8b5988d688a1eaad964494c8b Mon Sep 17 00:00:00 2001 From: Pranshu Maheshwari Date: Fri, 11 Sep 2026 11:36:54 -0400 Subject: [PATCH 3/4] Update doc/user/content/serve-results/source-stalls.md Co-authored-by: Moritz Hoffmann --- doc/user/content/serve-results/source-stalls.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user/content/serve-results/source-stalls.md b/doc/user/content/serve-results/source-stalls.md index aa7ff1dfd4548..6f90219cc81f6 100644 --- a/doc/user/content/serve-results/source-stalls.md +++ b/doc/user/content/serve-results/source-stalls.md @@ -91,7 +91,7 @@ one of the sources stalls, the object as a whole freezes consistently. Reading it is then a single-collection query, so it keeps serving stale results for as long as the stall lasts. This is also the recommended pattern for query latency in general, since point lookups on the index are served directly from -memory. +the index. ## Keep ad hoc queries serving: align frontiers with a maintained object From 061ea2124736bb9390fe25be4b7743b0f7a63f86 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 05:36:50 +0000 Subject: [PATCH 4/4] docs: address review feedback on source-stalls guide Fixes the lint-docs failure: the "keep serving across sources" section linked through the /concepts/views/ alias rather than the canonical /fundamentals/concepts/views/ path, and htmltest cannot resolve an anchor against an alias's redirect stub. Addresses the outstanding review feedback: - distinguish a frozen write frontier (this page's scope) from the transient `stalled` health status and a durable ingestion error - drop the "align frontiers with a maintained object" pattern for ad hoc cross-source queries: verified against the compute controller that a materialized view only holds back its inputs by one step, and internal discussion never endorsed the trick beyond a last resort - correct the explicit-transaction scope to the actual schema-wide timedomain, not just objects the transaction reads - scope the `statement_timeout` note to its actual write-operation behavior, and add a client-side-cancellation note - add `mz_frontiers`/`mz_source_statuses` as diagnostics - distinguish the two bounded-staleness failure modes (retryable SQLSTATE 40001 vs. a structural no-overlap error) Verified: local `hugo --gc` build plus `htmltest` against the full generated site (1121 documents) both pass. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_013WvybEHysqeefsajGiFbvR --- .../content/serve-results/source-stalls.md | 131 ++++++++++-------- 1 file changed, 70 insertions(+), 61 deletions(-) diff --git a/doc/user/content/serve-results/source-stalls.md b/doc/user/content/serve-results/source-stalls.md index 6f90219cc81f6..9869cb2c7d83a 100644 --- a/doc/user/content/serve-results/source-stalls.md +++ b/doc/user/content/serve-results/source-stalls.md @@ -11,9 +11,24 @@ menu: When an upstream system becomes unavailable (a Kafka broker outage, a paused replication slot, an ingestion cluster with no running replicas), the affected -sources stop making progress, or **stall**. A stalled source does not have to -stop your application from reading: in many cases, Materialize keeps serving -queries from the last consistently ingested data. +source's write frontier stops advancing, or **stalls**. This is one of three +things "stalled" can mean, and the only one this page is about: + +* **Write frontier stalled (this page).** The last consistently ingested data + is intact and nothing new is written. Reads below the frozen write frontier + keep serving. +* **[`mz_source_statuses`](/sql/system-catalog/mz_internal/#mz_source_statuses) + reports `stalled`.** A transient status carrying an error string, for + example a connectivity hiccup. It does not poison the collection; ingestion + resumes on its own once the underlying issue clears. +* **A durable ingestion error**, such as a decoding or replication error, + written into the collection's own error stream. From that timestamp + forward, every read of the collection returns the error, at every isolation + level, and none of the patterns on this page recover from it. + +A stalled write frontier does not have to stop your application from reading: +in many cases, Materialize keeps serving queries from the last consistently +ingested data. Whether a given query keeps serving depends on two things: the [isolation level](/serve-results/isolation-level/) of the session, and the @@ -57,25 +72,36 @@ no stalled data at all are unaffected under either isolation level. | `SUBSCRIBE` to a single stalled collection | Serves stale | Blocks | | Join between a stalled collection and a healthy one | Blocks | Blocks | | Query mixing a stalled collection and a user-writable table | Blocks | Blocks | -| Read of stalled data inside an explicit transaction | Blocks | Blocks | +| Explicit transaction, if a stalled collection shares a schema with anything the transaction reads | Blocks | Blocks | -Blocked queries **wait**; they do not error. A blocked query completes once -the source resumes and its write frontier passes the query's timestamp, with -no work lost. Note that `statement_timeout` does not interrupt this wait: it -cancels queries that are executing, but does not fire while a query is waiting -for its timestamp to become available. Use client-side timeouts to bound the -wait. +Blocked queries **wait**; they do not error, and a client-side cancellation +(a driver-level statement timeout, or `Ctrl-C`) still reaches them right +away. A blocked query otherwise completes once the source resumes and its +write frontier passes the query's timestamp, with no work lost. + +Note that server-side `statement_timeout` does not bound this wait: it is +existing, general behavior scoped to the read portion of write statements +(`INSERT ... SELECT`, and the `WHERE` of `UPDATE`/`DELETE`), not to blocked +reads, so it never fires while a query is waiting for its timestamp to +become available. Use a client-side timeout to bound the wait on a plain +`SELECT`. To see why a specific query does or does not serve, use [`EXPLAIN TIMESTAMP`](/sql/explain-timestamp/): it reports `can respond immediately: -true/false` along with the read and write frontiers of every input. +true/false` along with the read and write frontiers of every input. To check +frontiers across every object at once, +[`mz_internal.mz_frontiers`](/sql/system-catalog/mz_internal/#mz_frontiers) +reports the read and write frontier of every source, sink, table, index, and +materialized view, and +[`mz_internal.mz_source_statuses`](/sql/system-catalog/mz_internal/#mz_source_statuses) +reports whether a source is merely `stalled` or has hit a durable error. ## Keep serving across sources: maintain the query The recommended pattern for queries that span sources is to maintain the query -as an [indexed view](/concepts/views/#indexes-on-views) or [materialized -view](/concepts/views/#materialized-views), and have the application read from -that object directly: +as an [indexed view](/fundamentals/concepts/views/#indexes-on-views) or +[materialized view](/fundamentals/concepts/views/#materialized-views), and +have the application read from that object directly: ```mzsql CREATE VIEW order_enrichment AS @@ -93,54 +119,33 @@ long as the stall lasts. This is also the recommended pattern for query latency in general, since point lookups on the index are served directly from the index. -## Keep ad hoc queries serving: align frontiers with a maintained object - -If your application must issue **ad hoc** queries that reference multiple -collections (for example, generated queries from a BI tool joining a stalled -source to a live one), you can keep those queries servable during a stall by -maintaining *any* object that reads the same set of inputs: - -```mzsql --- A small maintained object whose only purpose is to hold the read --- frontiers of its inputs together. -CREATE MATERIALIZED VIEW frontier_alignment AS - SELECT max(id) FROM ( - SELECT id FROM kafka_orders - UNION ALL - SELECT id FROM pg_customers - ); -``` - -Because the maintained object must remain readable at its own (frozen) write -frontier, Materialize holds back the read frontiers of **all** of its inputs. -That keeps the stalled and live collections' frontier intervals overlapping, -so an ad hoc query over any subset of those inputs can still select a valid -timestamp and serve stale results instead of blocking. - -For this to work, note: - -- **The object must exist before the stall.** Once the live inputs' read - frontiers have advanced past the stalled collection's write frontier, - compaction has already discarded the historical data. Creating the aligning - object after the fact does not help, and the new object itself cannot serve - until the source resumes. +## Ad hoc queries across a stalled and a live source -- **The object must genuinely read every input you want covered.** The - optimizer removes inputs it can prove are unused (for example, behind - `WHERE false`). Check `EXPLAIN` on the object's definition to confirm all - intended inputs appear in the plan. +An ad hoc query that joins a stalled collection to one still advancing (for +example, a generated query from a BI tool) has no supported way to keep +serving: Materialize can only wait for the two collections' timestamp ranges +to overlap again, or fail. -- **Holding back read frontiers has a cost.** For the duration of the stall, - Materialize retains historical data for the covered inputs that it would - otherwise compact away. +It is possible to hand-roll an overlap by maintaining an unrelated object +that reads the same inputs, since a maintained object holds back its inputs' +read frontiers for as long as it stays readable. This is not a supported +pattern: for a materialized view, the held-back window is exactly one step +behind the object's own write frontier, not a window you control, and the +optimizer can prune an input it proves unused, silently narrowing what the +trick actually covers. Treat it as a last resort, not a technique to build on. -- **Reads of stalled data inside explicit transactions are not rescued** - (see below). +If your application issues this kind of ad hoc query, convert it to the +maintained-query pattern above instead of trying to align frontiers after +the fact. -## Don't use transactions +## Avoid explicit transactions during a stall -Explicit read transactions that touch stalled data are unable to serve -during a stall. Issue single-statement queries instead. +Materialize picks one timestamp for an entire transaction, valid across +every object in every schema referenced by its first statement, not just the +objects the transaction actually reads. If a stalled collection shares a +schema with a healthy table you query inside `BEGIN`, the transaction blocks +even though it never reads the stalled collection. Issue single-statement +queries instead of wrapping reads in an explicit transaction during a stall. {{< if-released "v26.29" >}} ## Fail fast instead of blocking @@ -154,10 +159,12 @@ level. During a stall: bounded staleness, as long as the stall is younger than the configured bound. -- Query shapes that would block under serializable, and any query reading - stalled data once the stall exceeds the bound, **error immediately** with - `SQLSTATE 40001` instead of blocking, giving the application a clean signal - to retry or fall back. +- A query that would block under serializable instead errors once the data + available is too stale for your bound. When a valid, if stale, timestamp + exists, Materialize raises a serialization failure (`SQLSTATE 40001`) so + your application gets a clean, retryable signal. When the inputs' timestamp + ranges never overlap at all, it raises a different, non-retryable error + instead. Design your fallback to handle both. ```mzsql SET TRANSACTION_ISOLATION TO 'bounded staleness 1m'; @@ -168,6 +175,8 @@ SET TRANSACTION_ISOLATION TO 'bounded staleness 1m'; - [Isolation levels](/serve-results/isolation-level/) - [`EXPLAIN TIMESTAMP`](/sql/explain-timestamp/) +- [`mz_internal.mz_frontiers`](/sql/system-catalog/mz_internal/#mz_frontiers) +- [`mz_internal.mz_source_statuses`](/sql/system-catalog/mz_internal/#mz_source_statuses) - [`BEGIN`](/sql/begin/) - [`SUBSCRIBE`](/sql/subscribe/) - [Troubleshooting serving](/serve-results/troubleshooting/)