[leedqin]

Motivation

Ninth PR in the cluster details redesign stack (#39509), on top of #40043. The redesign's prototype has a Hydration report tab. This PR adds the data a Hydration tab needs, built on the hydration history the database now records. The next PR in the stack, #40074, adds the tab. Part of CNS-173.

Description

• History, read with react-query since the history tables have no index:
  • a cluster's hydration episodes from mz_replica_hydration_history, one row per episode with the highest peaks across its processes. Gated on 26.43, which records a row per process, and leaves out older rows. Episodes with no objects stay in: on a replica with nothing to hydrate, such as one running only sources, they're the record that a boot finished;
  • the cluster's replicas from mz_cluster_replica_history, so a dropped replica (a resize, a burst replica) keeps its name and size. Fetched again only when the cluster's current replicas change, keeping the same cluster's answer up meanwhile, and filtered to the window as it moves on;
  • an episode's objects from mz_object_hydration_history, with catalog ids through mz_object_global_ids. Fetched once per episode, since a finished episode doesn't change;
  • the highest CPU sampled during an episode, from mz_cluster_replica_metrics_history.
• Live state, in SUBSCRIBE-fed TanStack DB collections:
  • whether each user object on each of the cluster's replicas is hydrated. A lookup on mz_compute_hydration_times_ind, on the fast path;
  • each object's state on one replica (waiting for a hydration slot, working, hydrated), from that replica's mz_compute_hydration_times_per_worker in a session pinned to the replica. Gated on 26.39. It runs on the replica, not on mz_catalog_server, so the tab mounts it only for a replica that is hydrating.
• Model (hydrationModel.ts):
  • an episode is a full hydration when it starts within two minutes of its replica coming online or being created, or covers every object the cluster had when it finished. The second rule catches a zero-downtime upgrade's rehydration, which runs while the environment is read-only and so records no boot. Objects created during an episode raise its threshold as much as its count, so a burst of new objects isn't full. Any other episode is additive: it covers objects added to an already-hydrated replica. The database records no cause, so this is inferred from status history and the objects' creation times;
  • a replica that goes offline before it finishes hydrating, such as an out-of-memory restart loop, is an unfinished hydration. The history only records episodes that finish, so these come from status history. An attempt runs from the replica's previous restart or its creation, not its last online row, since a zero-downtime upgrade writes that row after the replica hydrated. An attempt that began before the window is skipped, since the episodes before the window aren't read. An offline row counts as a restart only after an online one, since a new replica's pod on Kubernetes records an offline row before it first comes online. An object created during an attempt starts a hydration of its own, so the attempt finished only if a hydration finished after the last such object;
  • an episode's objects are split into time waiting for one of the hydration slots and time working.
• useObjectCreationTimes keeps its earlier answer up while the objects overlap, as after one is created, so a new index doesn't blank what depends on it.
• useClusterHydrationData and useHydrationDetail load and derive what the tab shows. The detail's object timings and CPU sample load and fail on their own, and the timings don't wait on a catalog that failed.
• Fix: SubscribeManager starts a clean snapshot when it resends a subscribe the server ended, as when the subscribe's target replica restarts. The old run's rows and error used to stay, so a per-replica subscribe read as failed for as long as it stayed mounted. The held state stays until the new run updates it, so a statement that fails every time keeps its error without flickering.
• Fix: collection subscribes now keep their session variables across connection attempts. Each attempt sent the defaults instead, so a cluster passed to a collection subscribe never reached the server. No existing caller passed one. Only variables that are set are passed, so unpinned subscribes keep running on mz_catalog_server.
• types/materialize.d.ts is regenerated with yarn gen:types. Most of its diff is the generator catching up with the catalog: builtin materialized views are no longer Generated<>, and relations added since the last run (including the two hydration histories and mz_object_global_ids) appear.

yarn gen:types doesn't run against Materialize as is: kysely 0.28's Postgres introspector calls pg_get_serial_sequence, which Materialize lacks, and lists only tables and views, which leaves out the catalog's builtin materialized views. The types here were generated with both patched locally in node_modules. Fixing the script is a follow-up.

Cost on mz_catalog_server: the episode query and the object query read storage, scoped to one cluster or replica and a time bound, as the redesign's status timeline already does. Episodes poll at the history's 60s collection interval. The replica history scans its index, fetched once per change to the replicas.

The Metrics tab's "Hydrating · N objects" badge counts introspection dataflows as well as user objects, so for a moment after a replica boots it can disagree with this PR's per-replica progress.

Tests

• hydrationHistory.test.sql.ts: each builder against shadowed relations (grouping and peaks per episode, the 26.43 filter, episodes with no objects, dropped replicas in the window, user objects only, per-worker states combined into one per object, catalog ids for dropped objects), and against the real catalog, including a new index's hydration streamed from a session pinned to the quickstart replica.
• hydrationModel.test.ts: full and additive episodes, a zero-downtime upgrade's rehydration counted as full by the objects it covered, a burst of new objects that outnumbers the existing ones, out-of-memory restarts that never finished, a restart merged across processes, restarts after a finished hydration and on drop, a restart while hydrating an object created after the replica finished, an attempt that began before the window, an upgrade's late online row, a new pod's offline row, a boot with nothing to hydrate, the latest full hydration by finish, dropped replicas outside the window, heap limits, and object spans.
• useLiveHydration.test.ts: progress counted per replica from the stream on mz_catalog_server, progress complete at once for a cluster with no replicas, per-object states read in a session pinned to the replica, following the replica's new process after its subscribe fails over a restart, and no subscribe before 26.39.
• SubscribeManager.test.ts: a subscribe the server ends and runs again drops its error and old rows, and one that fails every time keeps reporting its error.
• replicaProcessMetrics.test.sql.ts reads its snapshot through the new shared readSubscribeSnapshot helper.

🤖 Generated with Claude Code

---

[linear-code]

CNS-173

[def-]

QA LLM Review

1. MEDIUM -- A restart after a hydration that finished before the window is reported as an unfinished hydration

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:229

findUnfinishedHydrations takes the replica's last boot from the full retained status history, but only checks for a finished hydration in hydrations, and the episodes query only returns episodes finished inside the window. So when a long-running replica goes offline inside the window, for example a steady-state OOM, the model reports a hydration that "went offline before it finished" and dates its start to the boot days earlier. With the default 1-hour window this is the usual case for any replica that restarts, not a rare one.

Details

The bounds don't match:

• replicaBootTimes (hydrationModel.ts:119) gets boots from createdAt and from every process-0 online row. buildReplicaStatusHistoryQuery has no time bound, and mergeHydrationReplicas gives current replicas their original created_at.
• useClusterHydrationEpisodes (queries.ts:1316) fetches finished_at >= now - timePeriodMinutes (hydrationHistory.ts:62).
• bootMs is the latest boot before the offline event, which can be before the window. The finished check at hydrationModel.ts:232 then looks for an episode finishing between bootMs and the offline event among only the in-window episodes, finds none, and emits the entry.

I checked this by running the model on: replica created and online at 2026-10-18 10:00, OOM-killed at 2026-10-20 11:30, back online at 11:31 and rehydrated by 11:35, window starting at 11:00. The output is [{startedAt: "2026-10-18T10:00:30Z", endedAt: "2026-10-20T11:30:00Z", reason: "oom-killed"}]. When the 2026-10-18 episode is also passed in, which the window excludes, the output is []. The existing test "skips a restart after the replica finished hydrating" only covers a hydration inside the window.

Two possible fixes: skip a restart whose bootMs is before startMs, which only loses the first attempt of a crash loop that began before the window, or also fetch each replica's latest episode that finished before the window.

[def-]

QA LLM Review

1. MEDIUM -- A zero-downtime upgrade's rehydration is classified as a new-objects episode

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:152

On Kubernetes, the full rehydration a 0dt upgrade runs on every replica finishes before status history records any boot for it, so transformHydrations labels it newObjects. Releases make this the most common full hydration in cloud. Each one drops out of latestFullHydration and typicalFullHydration and is shown as a hydration of new objects.

Details

• The new generation runs each replica on new -gen-N processes (src/controller/src/clusters.rs:1034). These install and hydrate every dataflow while environmentd is read-only, and the episode's started_at is their earliest installed_at.
• A read-only environmentd writes no status-history rows (src/adapter/src/coord/message_handler.rs:1077). The first online row for those processes is written when the promoted generation reboots as leader and its orchestrator watch reports the already-running pods ready. That row is stamped with the reboot time, since the pod Ready condition's last_probe_time is normally unset (src/orchestrator-kubernetes/src/lib.rs:1401).
• Promotion waits until clusters have stayed hydrated for with_0dt_caught_up_check_stability_period. That is 10 minutes by default, and the flag has no LaunchDarkly override (KNOWN_MISSING_FROM_LD). So the online row always comes more than the hydration time plus 10 minutes after the episode started, far outside BOOT_WINDOW_MS.

I ran the model on that timeline: the gen-N processes boot at 10:00, the episode runs from 10:00:05 to 10:06, and the leader's online row lands at 10:16:30. The output is kind: "newObjects", and latestFullHydration returns undefined. The promotion row also becomes the replica's latest boot. So a later OOM at 10:40 is reported as an unfinished hydration that started at 10:16:30, although the replica had been hydrated since 10:06.

Boot detection needs a signal that also sees read-only boots. now() - mz_uptime(), which the base stack already uses for the resource usage timeline, doesn't help here: it marks the leader reboot, the same moment as the online row.

[def-]

QA LLM Review

1. MEDIUM -- Every restart of a replica with no indexes or materialized views is reported as an unfinished hydration

console/src/api/materialize/cluster/hydrationHistory.ts:61

The episodes query drops zero-object episodes, but findUnfinishedHydrations (hydrationModel.ts:232) treats a restart as finished only if some episode finished between the replica's boot and the restart. So on a cluster without indexes or materialized views, such as a source or sink cluster, or a cluster used only for ad hoc queries, every OOM kill or other offline event in the window is listed as a hydration that never finished, however long the replica had been up.

Details

The collector does record an episode after each boot of such a replica. Since 26.43, object_count counts only non-introspection exports, so that episode has object_count 0 (test/testdrive/hydration-status.td:69 expects hydrated 0 for a replica with no objects). .where("object_count", ">", "0") removes exactly those rows, so hydrations never holds an entry for the replica.

I ran the model on a source-only replica that came online at 2026-10-21 14:01 and was OOM-killed at 2026-10-22 03:00. The output is an unfinished hydration from 14:01 to 03:00 with reason oom-killed. When the zero-object episode recorded at 14:01 is passed in, the output is empty.

This is separate from the earlier window-boundary finding. The boot here is inside the window, and any online row in the 7-day default window (an earlier restart, an environmentd restart, a release) puts it there. In #40074 these entries become unfinished rows in the history table and chart. The verdict is unaffected, since live progress counts zero objects and reads as hydrated.

One option is to keep zero-object episodes for the unfinished check and filter them only from the hydrations that are displayed.

[leedqin]

Thanks, all three are addressed in 9a2a02e:

1. Window boundary: an attempt now runs from the replica's previous restart or its creation, and one that began before the window is skipped rather than reported.
2. Zero-downtime upgrades: an episode that covers every object the cluster had when it began counts as full, and online rows no longer bound an attempt, so the promotion's late online row doesn't produce a false restart. Objects dropped since aren't counted, so after a drop a partial hydration can read as full. That's noted in the code.
3. Zero-object episodes: the episodes query keeps them, and they count as a finished boot. The tab still leaves them out of the chart and table.

Each has a regression test in hydrationModel.test.ts, and the SQL test now expects the zero-object episode.

[def-]

QA LLM Review

1. MEDIUM -- On Kubernetes, every new replica reports an unfinished hydration at its creation

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:210

findUnfinishedHydrations treats every process-0 offline row as a restart. On Kubernetes, a new replica writes an offline row with no reason as soon as its pod is created, before it has ever come online. So every replica created in the window (a new cluster, a resize, a burst replica) gets an unfinished hydration that runs from its creation to a fraction of a second later. In #40074, a replica doing its first hydration then reads as restarting, and the tab shows an error verdict.

Details

How the row gets written:

• The coordinator starts each process of a new replica at Offline(Some(Initializing)) (src/adapter/src/coord/ddl.rs:639, src/adapter/src/coord.rs:1723).
• The pod watch passes the pod's ADDED event through (touched_objects). A pod that was just created has no Ready condition, so the event maps to Offline(None), stamped with the current time (src/orchestrator-kubernetes/src/lib.rs:1389).
• Offline(None) is not equal to Offline(Some(Initializing)), so the cluster-event handler writes an offline row with a NULL reason (src/adapter/src/coord/message_handler.rs:1059, :1077). The process orchestrator reports online straight away, which is why local runs don't show the row. I traced this through the code and haven't seen it on a live cluster.

I ran the model on this case: replica created at 10:00:00.000, offline/NULL at 10:00:00.300, online at 10:01:30, an episode from 10:01:32 to 10:06:00, and a window starting at 09:00. The output is [{startedAt: 10:00:00.000, endedAt: 10:00:00.300, reason: null}], both while the replica is hydrating and after it finishes. Without the pending-pod row the output is [].

In #40074, summarizeReplicasNow counts that entry as a restart because nothing has finished yet. So during the first hydration, the verdict says the replica "hasn't finished hydrating since 10:00" and "restarted once". Once hydration finishes, the history still keeps a "Restarted before finishing" row. None of the fixtures in hydrationModel.test.ts has an offline row before the first online.

One fix: count an offline row as a restart only when the replica's previous process-0 transition was online.

[leedqin]

Thanks, fixed in 44073ca. An offline row now counts as a restart only when the replica's previous process-0 transition was online, so the row a new pod records before it first comes online no longer reads as an unfinished hydration. There's a regression test in hydrationModel.test.ts.

This push also makes live progress complete at once for a cluster with no replicas, for the finding on #40074.

[def-]

QA LLM Review

1. MEDIUM -- New objects created together are classified as a full hydration once they outnumber the cluster's existing objects

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:164

coversAll compares the episode's object count with the cluster's object count when the episode started. Objects created after the episode started add to the episode's count but not to that threshold. So an episode that hydrates only new objects counts as full whenever it covers more objects than the cluster already had, even though it never touched the existing ones. That partial episode then becomes the replica's latestFullHydration and enters typicalFullHydration.

Details

At its start the cluster holds its N existing objects plus the first new one, which is created just before it is installed (useClusterHydrationData.ts:122). So the threshold is N+1, and any burst of K ≥ N+1 new objects whose hydrations overlap, and so form one episode, is labelled full.

I ran the model on this case: a replica up since 08:00 with one index, then two new objects created at 10:00:00 and 10:00:01 that hydrate together by 10:03. The output is kind: "full", and latestFullHydration and the typical duration are both that 3-minute episode. Without objectCountAt the same episode is newObjects. With 5 existing objects and 6 new ones, the result is also full.

In #40074 that duration feeds "Its last full hydration took …" while a replica rehydrates after a restart, and also the hydrated verdict and the per-size typical value. Both then understate how long a full rehydration takes. The NOTE about dropped objects doesn't cover this case, since nothing was dropped.

Comparing against the objects the cluster had when the episode finished (objectCountAt(finishedMs)) would make objects created during the episode raise the threshold by as much as they raise the count.

[def-]

QA LLM Review

1. MEDIUM -- After a replica restarts, its live per-object hydration reads as an error for as long as the hook stays mounted

console/src/platform/clusters/ClusterPage/Hydration/useLiveHydration.ts:179

When the replica that useLiveObjectHydrations is pinned to restarts, for example after an OOM kill partway through hydrating, its subscribe fails with "target replica failed or was dropped". The subscribe is then sent again and streams the new process's rows, but isError stays true. In #40074, LiveObjects stays mounted while a replica is hydrating or restarting. So through an out-of-memory restart loop, the case the tab is meant to diagnose, it shows "Couldn't read which objects … is working on" in place of the live state it actually has.

Details

How the error sticks:

• The controller removes and re-adds a failed replica in one step (rehydrate_replica, src/compute-client/src/controller/instance.rs:1391). remove_replica fails every subscribe that targets the replica (instance.rs:1342). The WebSocket API sends that Error and then ReadyForQuery on the same socket (src/environmentd/src/http/sql.rs:445).
• SubscribeManager.onReadyForQuery sends the subscribe again, and it lands on the re-added replica. The stored error is cleared only when a socket opens (onOpen) or on reset. applySnapshot copies it into the collection status, so status.error stays set while the new rows arrive.
• I ran the hook against mockSubscribeSocket: a snapshot with u20 working, then the server's Error and ReadyForQuery, then the resent subscribe's snapshot and an update. The hook sent the subscribe twice and its objects followed the new process (queued, then working), but isError stayed true after the error.

One fix is to treat a target-replica failure as recoverable in this hook, by restarting the session or by ignoring the error once rows arrive after it. Another is to have SubscribeManager clear the error when it sends the subscribe again after ReadyForQuery.

[devin-ai-integration]

Devin Review found 3 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[devin-ai-integration]

🟡 Later hydration failures disappear after an earlier finish

When a replica crashes while hydrating new objects, findUnfinishedHydrations can discard that failure after an earlier episode finished. Its finished check covers the whole interval since the last restart, not just the hydration active at the crash.

Learn more

A replica can complete several distinct hydration episodes between restarts. This check treats any finished episode after creation or the previous restart as evidence that the replica finished hydrating before the next offline event. A later object installation can start a new episode and remain unfinished at that event, so the earlier finish says nothing about the active episode. The episode collector records only completed episodes; it cannot provide the missing start directly.

Example: A replica comes online at 10:00, finishes its initial hydration at 10:02, begins hydrating a new index at 10:10, then crashes at 10:12. The finish at 10:02 makes finished true, and the 10:10 failure disappears.

Recommended fix: Identify an unfinished attempt after the latest completed episode rather than treating any earlier completion as proof. Use available object installation or hydration state evidence if necessary to distinguish a new episode from an idle replica crash; add a test covering a completed initial hydration followed by a failed new-object hydration.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[leedqin]

Good catch, fixed in c809a93. An attempt now counts as finished only if a hydration finished after the last object created during it, and the restart is dated from that object's creation when the replica had finished before it. Regression test in hydrationModel.test.ts.

[devin-ai-integration]

🟡 Dropped replicas persist beyond the lookback window

As a dropped replica ages out of the selected window, useClusterReplicaHistory continues returning its cached row. The moving start date is fetched once per replica set, and staleTime: Infinity prevents refresh.

Learn more

The replica-history SQL selects replicas whose dropped_at is at or after a start date based on the current time. buildClusterReplicaHistoryQuery therefore returns a different set as time advances even when the current replica IDs do not change. This hook keys only on the duration and replica IDs and marks that result fresh forever. Leaving the tab open across the window boundary, or revisiting it while the query cache is retained, keeps replicas that have aged out.

Example: With a one-hour window, a replica dropped at 10:30 appears at 11:00. If the replica set remains unchanged, it still appears after 11:30 although the SQL would now exclude it.

Recommended fix: Refresh replica history on a cadence that bounds window staleness, or expire it at the nearest dropped replica's window boundary. Keep the query key scoped to the same cluster and selected duration.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[leedqin]

Fixed in c809a93: mergeHydrationReplicas takes the window start and leaves out replicas dropped before it. The hook rounds the start down to the minute, so the filter moves on with the polled history without refetching the replica history.

[leedqin]

Thanks, both are fixed in c809a93:

1. New objects outnumbering the existing ones: an episode now has to cover the objects the cluster had when it finished, so objects created during it raise the threshold as much as its count. There's a regression test with your one-plus-two burst.
2. Live per-object state stuck in error after a restart: fixed in SubscribeManager. When ReadyForQuery follows a subscribe the server ended, the resend starts from a clean snapshot, which drops the old run's rows and error. Listeners keep the held state until the new run updates it, so a statement that fails every time keeps its error without flickering. SubscribeManager.test.ts covers both, and useLiveHydration.test.ts covers your restart case through the hook.

[def-]

QA LLM Review

1. MEDIUM -- A replica created and dropped within five minutes is missing from the hydration data

console/src/platform/clusters/queries.ts:1374

useClusterReplicaHistory keys on the current replica set with staleTime: Infinity. When the set returns to a value it had less than five minutes earlier (react-query's default gcTime), the hook serves the answer fetched before the extra replica existed and does not refetch. So a burst replica, a replica turned on for a scheduled refresh, or a replica from a replication-factor bump that has been dropped again never reaches replicas. That loses its name and size, its status history, and, in #40074, its hydrations, which are exactly what the replica history is for.

Details

Timeline: u1 is current, so key u1 fetches [u1]. Burst replica u2 is created, so key u1,u2 fetches [u1, u2]. u2 is dropped, and key u1 comes back from the cache as [u1]. I ran this sequence through a QueryObserver with the hook's options: the third step returned [u1] with isStale: false after two fetches, so no third fetch ran. With gcTime: 0 added, it fetched again and returned u2 with its droppedAt, and placeholderData kept the previous answer up in the meantime.

Ways to hit it:

• With the default burst linger of 0s (default_hydration_burst_linger), a burst for an OOM restart or a new object is dropped as soon as the steady replica catches up, which is often within minutes.
• A SCHEDULE = ON REFRESH cluster switches between no replicas and one replica. While the tab stays mounted and each refresh finishes within five minutes, the key with no replicas never expires, so every refresh replica since the first load stays missing while the cluster is off.
• Leaving the tab and returning within five minutes after a replica came and went has the same effect.

The console's own useAlterCluster invalidates cluster queries, but controller-driven changes and SQL issued elsewhere don't. mergeHydrationReplicas then drops the replica, useReplicaStatusHistory isn't asked for it, and #40074 filters its hydrations out by replica id.

[def-]

QA LLM Review

1. MEDIUM -- An OOM while hydrating a new object goes unreported on a replica that has been up since before the window

console/src/platform/clusters/ClusterPage/Hydration/hydrationModel.ts:275

findUnfinishedHydrations drops any attempt that began before the window before it checks for objects created during the attempt. So when a replica that has been up since before the window is OOM-killed while hydrating a newly created index or materialized view, nothing is reported. With the 7-day default window in #40074 this covers most long-running replicas, and this restart is the case the new-object rule was added to catch.

Details

The new-object check only needs episodes that finished after the object was created, and those fall inside the window. But line 275 returns before lastCreatedMs is computed.

I ran the model on this timeline: a replica up since 2026-10-18, a window starting at 11:00, an index created at 11:10, an OOM kill at 11:15, back online at 11:16 and rehydrated by 11:25. The output is []. The same timeline with the replica created at 10:30, inside the window, gives [{startedAt: 11:10, endedAt: 11:15, reason: "oom-killed"}]. In a crash loop with OOMs at 11:15, 11:22 and 11:29, only the last two are reported, and the one the index caused is the one that's missing.

Related: objectCreatedAtMs comes from the cluster's current objects (useClusterHydrationData.ts:96). So once the user drops the index that caused the OOM, the restart also disappears on a replica created in the window. The model returns [] once that creation time is left out.

A possible fix: when an object was created in the window during an attempt that began before it, decide with finishedBetween(lastCreatedMs, endMs) and date the attempt from lastCreatedMs instead of skipping it. To cover dropped objects, the creation times could come from mz_object_history, which keeps dropped objects along with their cluster_id.

[def-]

QA LLM Review

1. MEDIUM -- The replica-pinned live subscribe is resent in a tight loop for viewers without USAGE on the cluster

console/src/platform/clusters/ClusterPage/Hydration/useLiveHydration.ts:144

Pinning useLiveObjectHydrations to a replica runs its SUBSCRIBE on the user's cluster, so it fails on every run for a viewer whose role lacks USAGE on that cluster. SubscribeManager resends a subscribe on every ReadyForQuery with no backoff, and the server sends one after every error. Such a viewer therefore sends failing SUBSCRIBEs as fast as the round trip allows for as long as the replica hydrates. In #40074, LiveObjects renders nothing on error, so nobody sees it happen.

Details

• With cluster_replica set, the statement isn't routed to mz_catalog_server (src/adapter/src/coord/catalog_serving.rs:176), and a SUBSCRIBE needs USAGE on its target cluster (src/sql/src/rbac.rs:982). The other collection subscribes run on mz_catalog_server, where PUBLIC has USAGE. A replica with INTROSPECTION INTERVAL = 0 fails the same way on every run, with cannot read log sources of replica with disabled introspection (src/adapter/src/coord/sequencer.rs:998).
• The WebSocket API follows every request with ReadyForQuery, failed or not (src/environmentd/src/http/sql.rs:446). onReadyForQuery then resets and resends (console/src/api/materialize/SubscribeManager.ts:315). Each round makes the coordinator plan a SUBSCRIBE, and the client calls captureException.
• I ran SubscribeManager against mockSubscribeSocket with every run ending in endWithError, which is how the server answers a denied SUBSCRIBE. It sent 500 SUBSCRIBEs on one connection in 168 ms. In a browser the round trip sets the pace, so a single viewer sends tens per second. That lasts for the whole hydration, or, in console: add a hydration tab to the redesigned cluster page #40074's restarting state, for the whole crash loop.
• The new test "keeps reporting the error of a subscribe that fails every time it runs" ends the second run with error, which sends no ReadyForQuery. The test stops after two runs, but the server would keep the loop going.

One fix is to resend only after an error that a retry can clear, such as the target replica failing, and to leave other errors in place until the session restarts. Backing off between resends would also bound the loop.