docs: serving during source stalls

[tylerhartwig]

Motivation

A customer asked whether a stalled upstream source has to stop Materialize from serving reads. The answer ("often no, and there are patterns that keep serving available") was worked out empirically on a Slack thread and in a repro environment, but is not documented anywhere. This adds a page to the Serve results section covering the behavior and the recommended patterns.

Changes

• New page serve-results/source-stalls.md ("Serving during source stalls"):
  • a behavior matrix of query shape x isolation level (serves stale vs. blocks),
  • why: single-timestamp selection against each input's read/write frontier interval,
  • the recommended pattern: maintain cross-source queries as indexed/materialized views and read them directly,
  • the frontier-alignment workaround for ad hoc slow-path queries: a pre-created materialized view over the same inputs holds their read frontiers together, so ad hoc queries mixing stalled and live inputs can still select a timestamp (with its caveats: must exist before the stall, must genuinely read all inputs per EXPLAIN, holds back compaction, does not rescue reads of stalled data in explicit transactions),
  • explicit-transaction behavior (timedomain pinning) and SUBSCRIBE scope,
  • fail-fast behavior under bounded staleness (gated on if-released v26.29, matching isolation-level.md),
  • statement_timeout caveat and EXPLAIN TIMESTAMP as the diagnostic.
• serve-results/troubleshooting.md: the "Stalled source" subsection now summarizes the block-vs-serve rule and links to the new page.

Verification

Every behavior claim in the page was verified empirically against Materialize v26.36.0 (Docker), with two independent stall methods producing identical results: scaling the ingestion cluster to zero replicas, and stopping the upstream Kafka broker. This includes the full 8 query shapes x 4 isolation settings matrix and the negative cases: an aligner MV created after the stall does not help, transactions reading only healthy objects keep serving while reads of stalled data inside transactions block, and SUBSCRIBE over a stalled+live join blocks like the equivalent SELECT.

Docs build (hugo --gc) passes with no warnings; bin/format-docs applied; the repo's docs-pre-pr-review-claude.sh review was run and its findings addressed.

🤖 Generated with Claude Code

[maheshwarip]

@antiguru how do you feel about this section? It sorta feels like hacking retain history together; I think it is a neat trick but I'm not sure if we should recommend it to users!

[linear-code]

EDU-70

[maheshwarip]

Pushed c898920 addressing the open review comments:

• Reworked the frontier-overlap intro with the A/B stall example, and pulled the read/write frontier definitions into a note (double-checked write-frontier wording against src/persist-client/src/read.rs — it's an exclusive bound, not "latest fully processed timestamp").
• Renamed "Behavior by query shape" → "Behavior by query shape and isolation level".
• Collapsed "Explicit transactions" into "Don't use transactions", scoped to "transactions that touch stalled data" rather than all transactions, since timedomain_for (src/adapter/src/coord/timeline.rs) only pins objects in the referenced schema — an unqualified "unable to serve during a stall" would have overstated it.

Left the "align frontiers with a maintained object" thread open — that's a product question for @antiguru, not something to resolve on my own.

---

Generated by Claude Code

[antiguru]

Review of the page. The headline is the stall taxonomy; the rest are smaller corrections.

Which kind of stall this page is about

Three different failure modes share the word "stall", and the page silently assumes one of them. A reader who arrives with a different one in mind will read the whole page as wrong.

1. The write frontier stops advancing. The last consistently ingested data is intact, nothing is written into the collection, and reads below the frozen write frontier keep serving. This is what the page documents, and what both repros (zero replicas, stopped broker) produce.
2. The stalled health status. mz_source_statuses.status = 'stalled' (src/storage/src/healthcheck.rs:174) is a transient error state carrying an error string. It does not poison the collection. But ingest-data/troubleshooting.md:48 describes stalled as "you likely have a configuration issue", so the word already means something error-shaped to a reader coming from that page.
3. A definite error. A SourceError or DecodeError is written into the collection's error stream (src/storage-types/src/errors.rs:377). From that timestamp forward, every read of the collection returns the error, at every isolation level. Only reads strictly before the error timestamp still serve, and no pattern on this page changes that.

The intro should define which sense it means, and say explicitly that a source that has written an error into its collection is a different failure mode that none of these patterns recover. Without that, the behavior matrix reads as a promise the product cannot keep for case 3.

Transactions block more broadly than the table says

The matrix row is "read of stalled data inside an explicit transaction". The actual scope is wider. Inside a transaction the timestamp is determined over the whole timedomain, which is every object in every schema referenced by the first statement, plus pg_catalog (src/adapter/src/coord/sequencer/inner/peek.rs:1035-1051, src/adapter/src/coord/timeline.rs:413-421). So BEGIN; SELECT healthy_table; blocks when a stalled collection merely lives in the same schema, without the transaction ever reading it. The empirical result that transactions over healthy objects keep serving most likely held because those objects sat in a different schema. Worth re-testing with the stalled source in the same schema, and widening the guidance accordingly, which also strengthens the "avoid explicit transactions" advice.

The frontier-alignment section

On the open product question: the instinct in that thread is right, this is hand-rolled retain history. Since retain history is not generally available, it is not an alternative we can point at instead, so the call is whether to ship the trick or drop the section. Two corrections that apply either way:

• The overlap window is exactly one millisecond, not a range. A materialized view is a write-only compute collection, so its input read holds are downgraded to write_frontier.step_back() (src/compute-client/src/controller/instance.rs:1908-1928). Ad hoc queries then serve at exactly the millisecond before the stall, for the whole duration of the stall. An aligning index would instead give a window as wide as the compaction window. Documenting a pattern that rests on that specific branch is a maintenance liability, and the page should not describe it as "an overlapping interval" when it is a single point.
• max(id) is a hierarchical reduce that arranges every input row. count(*) is accumulable, holds the same read holds, and costs almost no memory. The UNION ALL also requires the two id columns to share a type, which the example does not say.

statement_timeout

The claim is correct, but it is existing documented behavior rather than a stall-specific quirk, and the page should present it that way. The parameter is scoped to write operations by design: see doc/user/content/headless/configuration-parameters.md:65 ("the read portion of write operations"), which is what /sql/set and /sql/show render, and the variable's own description in src/sql/src/session/vars/definitions.rs:384. The origin is #12427, which added the timeout so that a wedged read-then-write could not hold the global write lock and block unrelated writes; it protects a shared resource rather than bounding query latency, and was never generalized.

Suggest linking /sql/set/#other-configuration-parameters and MaterializeInc/database-issues#8708, which is exactly this surprise on a broken source and is still open. MaterializeInc/database-issues#3085 tracks the general ask. Also worth stating that cancellation does work, since that is what makes the client-side timeout advice sound: a queued peek is cancelled and responds Canceled (src/compute/src/compute_state.rs:951).

One inconsistency noticed while checking: that configuration parameters table lists the default as 10s, while the code default is 60s. Nothing in the repo sets 10s, so the table may be stale.

Smaller items

• The bounded staleness claim that shapes which block under serializable "error immediately with SQLSTATE 40001" needs one more check. In the mixed stalled-plus-live case there may be no common timestamp at all, rather than one that merely exceeds the bound, and those two situations may not produce the same error.
• The heading "Don't use transactions" now contradicts its own body, which was scoped to transactions touching stalled data. Given the timedomain finding above, retitling to something like "Avoid explicit transactions during a stall" is both shorter and more accurate.
• Add mz_internal.mz_frontiers and mz_source_statuses alongside EXPLAIN TIMESTAMP as diagnostics, so a reader can identify which input is frozen and whether the source is merely frozen or errored.
• Menu weight 16 slots cleanly between isolation-level (15) and fdw (20).

---

Posted by Claude Code on behalf of @antiguru