[leedqin]

Motivation

Part of the cluster details redesign stack, on top of main. The CPU chart shows a replica's CPU as a share of its allocation, averaged over its workers. The CPU troubleshooting guide points out that a replica whose work sits on one worker can peg a core while reporting a modest cpu_percent, and that high CPU with evenly spread work points elsewhere: an expensive object, ad-hoc queries, or an undersized cluster. The Console had no way to tell these apart.

This adds a Workers tab next to Metrics. It answers two questions in one sentence: is a worker overloaded, and which object is behind it. The tab shares the page's ?replica= choice with Metrics, offers every replica whose size it knows, and opens with a short explanation of workers and skew. The CPU chart's Workers button and the Processes drawer link to it on the same replica.

An earlier version of this PR showed live charts of the busiest and the average worker, a busiest-workers card, and per-worker bars on every object row. In review they were hard to read and to act on: the numbers moved every few seconds, the worker count wasn't shown, and terms mixed workers, threads, dataflows and objects. This version reports one fixed minute and keeps the vocabulary to workers and objects.

Description

What it shows. A report of one measured minute that holds still until Measure again. The replica's CPU and the objects' freshness are recorded with the measurement, so a late metrics sample, a refetch on window focus, or coming back to the tab doesn't change a report that's already shown:

• The header: the replica's worker count ("16 workers · 2 processes × 8 workers"), the measured span ("Measured 14:04:10 to 14:05:15") and Measure again, which measures the minute from the click and counts it down ("Measuring a new minute… 42s left"), keeping the current report until then. Someone checking a fix they just deployed sees only data from after it. Coming back to the tab shows the last report as it was, until Measure again.
• One finding naming the worker and the object together. The rules are unchanged: a saturated worker (90% of a core and twice the other workers' average), uneven work (half a core and twice the others), light load, every worker busy, high replica CPU (75% or more), balanced and idle, with the busiest worker compared against the other workers' average, and only a saturated worker outranking high CPU.
  • Skew: "w11 is overloaded, and most of its time goes to orders_enriched". It says how much of the worker's time the object takes, how unevenly the object spreads, and the usual cause: a hot key, or a cross join when nearly all of the object's work is on one worker (from 50 times the other workers' average). It adds the object's freshness, and why replica CPU can read low. A copyable EXPLAIN ANALYZE CPU WITH SKEW, pinned to the replica, finds the step in the plan and its key. It's offered only for an object that leans on the worker. When no object takes most of the worker's time, it says so. On a multi-process replica it places the worker among the processes, as before.
  • Not skew: "Not skew: every worker is busy", or high CPU split into the workers' time and overhead, the work outside the workers' dataflows, each pointing at the object with the most time and the guide's sections. Overhead names what it covers: reading sources and writing sinks, answering SELECTs straight from indexes, queries that started and ended between readings, and scheduling, progress tracking and networking. It links to the cluster's queries that started in those minutes, in Query History.
  • Several workers at their limit, none ahead by 2×: "3 of 4 workers are at their core's limit", naming them, the least busy worker, and the objects taking the most of their time. A finding about one overloaded worker names any other worker at its limit, with the object busying it when that's a different one.
  • Light: "Nothing to act on", still naming an object that leans on a worker.
  • One worker: its work can't be uneven, so the finding is how busy the worker was and the object that took the most of its time. It warns when the worker is busy ("Not skew: r1's only worker is busy") or replica CPU is high, with the share outside its dataflows, and links to the guide's expensive-object and undersized-cluster sections.
  • CPU and process sentences only quote the replica's metrics when they agree with the workers' own time, so a replica whose metrics are missing doesn't read as "0% CPU".
• Workers: one bar per worker for its busy share of a core over the minute, grouped by process, with a dashed line at the other workers' average, the workers at their core's limit in red, and the picked object's part of each bar. The bars use the memory chart's layout, axis, average line and tooltip. Hovering a bar shows the worker's busy share and the picked object's part. Clicking one, or the arrow keys once the chart has focus, selects that worker, and a line under the bars says where its time went: "w7 (process 0) was 64% busy: 97% on orders_enriched, 2% on customers_idx, 1% on everything else." On large replicas, labels thin out to fit, always keeping the busiest and selected workers'.
• Objects: ranked by their time on the selected worker, which starts as the busiest: that time, the share of the replica's CPU, how the object spreads ("Even", "Mostly w11 · 20× the others", "All on w3"), and how far it was behind its inputs at the end of the measured span. Each object's name links to its page. Clicking a row picks the object and shows it on the bars, and the finding's object starts picked. The "Time on wN" value is a toggle button, so a keyboard user can pick a row too. Spread is only colored when the finding is skew, and left out for one worker. Two pooled rows and a footer cover the time no object owns, so the columns add up:
  • Queries and subscribes: the temporary dataflows of sessions' statements, which are SUBSCRIBE, SELECT, COPY TO, and the read of an UPDATE, DELETE or INSERT ... SELECT. A query that finishes inside the span keeps the time the readings saw it use. Two links open Query History on the cluster's statements from every user. "Started in these minutes" lists those that began from a minute before the span to its end, longest first. "Still running" lists those that began up to a day before and haven't finished, oldest first, which is where a long-running SUBSCRIBE shows, since the list filters on when a statement began. Query History filters by cluster, not replica, and lists only the statements that statement logging sampled.
  • Materialize's own dataflows: the introspection-subscribe- and metric-sink- dataflows the coordinator runs on every replica. A finding never names this row.
  • Other dataflows: dataflows that export no object.
• CPU: replica CPU against the workers' time over the last 15 minutes, with the measured span shaded. "Overhead" is the gap between them. Hovering reads the replica CPU sample for the minute that holds the cursor, the workers' time, and the overhead.
• Before anything is read, a first look asks to Start measuring and says what it costs: a read of each worker's time every 5s, from a session on the replica itself. Opening the tab, or picking another replica, reads nothing until that click. Then, until the first minute is measured, the first look shows the lifetime counters, with progress.

What it reads. The tab reads only after Start measuring, while it's open and its browser tab is visible, through one SUBSCRIBE on one session pinned to the replica with cluster and cluster_replica:

• each dataflow's time on each worker: mz_scheduling_elapsed_raw joined to mz_dataflow_addresses_per_worker on (id, worker_id), the key of both indexes, and filtered to root operators (list_length(address) = 1) before count(*). A root's time includes its children's, so summing roots counts each nanosecond once.
• unioned with mz_compute_exports, to name each dataflow. A subscribe's export is tagged with its session's connection id through mz_subscriptions and mz_sessions. A transient (t) export carries its dataflow's name, read per worker on the operators index's key. The coordinator names its own dataflows introspection-subscribe- and metric-sink- (src/adapter/src/coord). Every other transient dataflow runs a session's statement, named subscribe-, oneshot-select-, copy-to- or frontend-read-then-write-subscribe-.

Then:

• Every dataflow's counter grows each second, so elapsed time is rounded down to 10 ms, and an idle dataflow sends nothing.
• Every 5s, timed by the subscribe's upper, the page takes a reading. A reading carries each worker's total, and each row's total per worker, on by each dataflow's growth, so a dataflow that finishes keeps its time up to the reading before it ended. Every 30s it keeps a snapshot of the row totals. A measurement runs from the latest snapshot at least a minute back to the newest, up to 3 minutes, with no pause in between. Counters on a busy replica can advance in steps too long for a snapshot a minute back, and the span then runs longer. Measure again takes a snapshot at the next tick and measures the minute that starts there. If the open session would reach its UP TO before that minute lands, it opens a new session first.
• A dataflow that starts and ends between two readings is never seen. Neither is a SELECT answered straight from an index, which the worker serves outside any dataflow operator (process_peeks, in src/compute/src/server.rs). Both show only as overhead.
• Freshness comes from the query the cluster's index and materialized view lists already use (mz_wallclock_global_lag_recent_history by object id, on mz_catalog_server), read once when a measurement lands and recorded on it.
• The tab leaves its own dataflow out by dataflow id, taken from the export tagged with its session's id (from BackendKeyData). It remembers those ids across its sessions, because a closed session's tag leaves about a second before its dataflow does.
• The subscribe carries UP TO 10 minutes past opening or resuming, so it ends on the server even when the browser stops reading; reaching it reads as the 10-minute pause. Reading also pauses while the browser tab is hidden, and stops when leaving the tab.
• The replica's own CPU comes from the processes drawer's per-process metrics SUBSCRIBE (useReplicaProcessSamples), an index lookup on replica_id over the last 20 minutes. The finding's CPU, and the busiest process, are recorded once the samples covering the measurement have loaded. If the metrics can't be read, the tab says the finding leaves the replica's CPU out.
• Readings and the last measurement are kept for 30 minutes in per-environment TanStack DB collections.

The tab ships with the redesign, so it is behind the same UI preview flag and off in self-managed. A classic-page user following a /workers link lands on the classic overview.

Counter semantics. A dataflow's time leaves the counters when it shuts down (handle_shutdown in src/compute/src/logging/timely.rs). Readings carry totals on by each dataflow's growth, so they don't fall when one finishes. A dataflow new since the last reading counts in full, and one whose counter fell counts as new, as for an id reused after a restart. A measurement never spans a per-dataflow counter fall or a pause.

Supporting changes.

• useObjectLink is shared with the memory by object card. useBuildWorkflowGraphPath returns a stable callback, so the memos built on it hold.
• queryHistoryPath in queryHistoryUtils.ts builds a Query History link from the list's own URL encoding.
• The memory chart's ThresholdLine and TIME_TICK_COUNT are exported for the worker bars and the Workers CPU chart.
• TimeAxis takes a tickFormat, and the Workers chart adds seconds when ticks are under a minute apart.
• SubscribeCollection gains getRows.
• Size allocations include workers.
• GraphTooltip no longer takes pointer events. A tooltip anchored to its mark, as the worker bars' is, could open under the pointer, take it, fire the mark's leave and flicker. No chart puts anything clickable in it.
• SubscribeManager gains getUpper, getConnectionId and isComplete, and its flush takes the buffer before notifying, so a listener can read the upper. buildSubscribeQuery takes upToMs. mockSubscribeSocket hands each connection to the test, with its id, options, advance and complete.
• ReplicaSelect is shared by the Metrics and Workers tabs, and the tab links carry ?replica=. With no replica in the URL, the Workers tab shows the first and leaves the URL alone, so the other tabs keep showing every replica; a replica the cluster doesn't have, or one whose size is unknown, gets a note instead of a different replica's readings. currentReplicaColor is shared with the Metrics tab's colors.
• The Processes drawer takes an initial replica and a link to the Workers tab, and opens from router state when the tab links back; closing it clears that state and returns focus to the Processes button. Its findings are an Alert with a headline and the numbers behind it. processReadingsModel.ts exports its evenness rule (EVEN_RATIO) and otherProcessesLabel.

Cost. Measured on a local replica, the earlier three-subscribe version against this subscribe, one viewer; each row is one replica:

User dataflows (operators)	Workers	CPU per worker, ms/s	Memory per viewer	Rows/s
0 (458)	2	7.1 → 1.5	0.95 → 0.03 MB	3.6 → 0.7
0 (458)	8	8.4 → 1.8	3.6 → 0.08 MB	17.5 → 3.1
200 (20,758)	2	81 → 7.1	29 → 0.13 MB	20 → 12.5
200 (20,758)	8	82 → 7.1	115 → 0.47 MB	98 → 68
500 (51,208)	2	188 → 20.7	72 → 0.30 MB	39 → 29
500 (51,208)	8	226 → 19.6	184 → 1.1 MB	197 → 149

• Counting roots after the indexed join reuses the introspection indexes: about 1.6 ms plus 0.36 µs per operator, per worker per second. Each viewer's read runs on the customer's replica, so viewers add up.
• One session per open: about 23 ms of mz_catalog_server CPU. The freshness query adds one lookup per measurement there.
• After a pause or leaving the tab, environmentd keeps streaming until the browser's WebSocket closing handshake times out, 60s in Chromium, because it doesn't read the Close frame mid-SUBSCRIBE (src/environmentd/src/http/sql.rs ~1175).

Screenshots

From a local Materialize running a demo cluster of 2 processes × 2 workers. A cross-join materialized view keeps one worker busy while orders stream in, and a SUBSCRIBE and keyed SELECTs run alongside it. Local sizes have no CPU limit, so for these screenshots the browser gave each process the host's 16 cores as its allocation. Everything else is the tab's real readings.

Work leaning on w1, with replica CPU reading only 23%. The finding names the materialized view, the table pools the queries and Materialize's own dataflows, and the CPU chart shades the measured span:

Hovering w1:

After clicking w2, the line under the bars and the objects table follow w2:

A single-worker replica:

Verification

• workerCpuModel.test.ts:
  • busy shares between readings;
  • measuring the latest span from a minute up to 3 minutes, longer when counters advance in long steps, across no pause and neither kind of counter fall, needing readings at both ends and starting after a given time;
  • rows and worker totals that keep a finished dataflow's time;
  • per-row totals carried across readings, including for a reused id and workers past the count, and rows remembered for a dataflow whose export left;
  • pooling sessions' SELECT, SUBSCRIBE and read-then-write dataflows into the queries row, and the coordinator's into their own row, which is never the finding's object;
  • spread as even, leaning or all on one worker, including the 50× boundary;
  • the workers each verdict calls out on the bars, every one at its limit;
  • the finding's object for each verdict, and verdicts for saturated, uneven, light, balanced, every worker busy, idle and high CPU;
  • own-dataflow exclusion, and replica CPU per scrape, averaging processes.
• workerCpuSampler.test.ts, through real SubscribeManagers over mockSubscribeSocket:
  • measuring once a minute exists and not before, and the measurement staying fixed as reads arrive;
  • an ad-hoc query that finishes mid-minute keeping its time in the queries row;
  • Measure again measuring the minute from the next reading, keeping the current measurement until it lands, resuming first when paused, and opening a new session when the open one's bound is too close;
  • a recorded annotation restored by the next sampler, and a restored measurement showing as it was, within retention only;
  • row totals carried on from an earlier sampler, and started over for a different worker count;
  • measuring when counters advance in 26 s steps, and reads trimmed to 3 minutes;
  • staying idle until started, through a hidden and shown tab, and opening one session for two starts;
  • the UP TO bound, pauses for the limit, hidden tabs and the user, waiting for counters, cadence under lag, own-subscribe exclusion and continuation across samplers.
• WorkerCpuVerdict.test.tsx:
  • each finding's headline and sentences, hot key against cross join, and the EXPLAIN statement for indexes and materialized views only;
  • freshness and low-CPU sentences, and when they're left out (including a replica whose CPU reads below its workers' time);
  • naming every worker at its limit: as a group when none leads by 2×, with the objects busying them and no skew EXPLAIN for an evenly spread object, and beside an overloaded worker;
  • placing the worker among processes, and high-CPU attributions with the work outside dataflows and the Query History link;
  • the two-worker wording, the one-worker findings, and the queries row's note leaving out "falling behind";
  • the first look's Start measuring, progress, pause, failure and one-worker wording.
• WorkerBars.test.tsx:
  • the summary label, one bar per worker, and the average line with its legend value;
  • the picked object's part never exceeding its bar, and the hover tooltip;
  • selecting by click and arrow keys;
  • the called-out workers' colors and legend item, for one worker or several, and process labels only for multi-process replicas;
  • one worker without an average, and the busiest and selected workers keeping their labels when labels thin out.
• ObjectCpuTable.test.tsx:
  • headers and rankings that follow the given worker, each column's text, and freshness text;
  • the queries row's count and its two Query History links, the own-dataflows row, and the Other dataflows footer;
  • colors only for skew with a real share, and the spread column left out for one worker;
  • picking by row and by keyboard through the pressed toggle, with only the picked row pressed, and the name link opening without picking;
  • pagination and the empty state.
• WorkerCpuCharts.test.tsx: the legend, the measured span's position and clipping, the tooltip rows including the overhead, and reading the replica CPU minute that holds the cursor.
• ClusterWorkers.test.tsx, with fake timers over mockSubscribeSocket:
  • nothing read until Start measuring, and a newly picked replica waiting for it too;
  • a first look while the first minute is measured, then a finding naming the worker and object;
  • the Workers and Objects sections for the measured span, with a picked row changing the bars and a clicked worker re-ranking the objects with its breakdown;
  • the report holding while counters change, and Measure again counting down and then reporting the minute after the click;
  • the finding's CPU holding when a sample from the measured minute lands late;
  • freshness holding when the window regains focus, and read again when the next measurement lands;
  • a failed read shown once, and a note when the CPU metrics can't be read;
  • a single-worker replica's finding;
  • replica selection without writing the fallback into the URL, notes for stale links and unknown sizes, pausing and hidden tabs.
• queryHistoryUtils.test.ts: the Workers tab's Query History links read back, through the list's own URL parsing, as the cluster's statements from every user in the range, and as its running statements, oldest first.
• ClusterDetail.test.tsx: the Workers tab route, the tab strip carrying the replica, and the classic page's redirect.
• ResourceUsage.test.tsx: the CPU chart's Workers link and the Processes drawer's link go to the tab on the selected replica, and the tab's link back opens the Processes drawer.
• workerCpu.test.sql.ts, against a real Materialize:
  • the subscribe's rows match an independent per-operator sum, and the page's own export is tagged with its session;
  • the coordinator's dataflows pool into Materialize's own row;
  • a running SELECT's and INSERT ... SELECT's dataflows pool into the queries row;
  • the subscribe ends at UP TO, and rounding cuts updates.

Follow-ups

• UP TO and the counters' lag both come from the browser clock. A browser clock running 10 or more minutes behind environmentd would fail every open.
• A pending Measure again is lost when the tab remounts before its minute lands.
• On system clusters, s exports aren't read, so their dataflows show as other dataflows.
• A SELECT answered from an index, and a dataflow shorter than a reading, can't be attributed to a row. They only show as overhead.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- Skew verdicts can't fire on 2-worker replicas, which get "spread evenly" instead

console/src/platform/clusters/ClusterPage/WorkerCpu/workerCpuModel.ts:213

isSkewed compares the busiest worker with an average that includes the busiest worker, so the ratio can never exceed the worker count. On a 2-worker replica (M.1-xsmall, the smallest size that gets the Workers button), saturated and uneven can only fire if the other worker is completely idle. A worker pegged at 100% next to one at 20% is therefore reported as "Work is spread evenly", or as "this isn't worker skew" once replica CPU reaches 75%.

Details

With two workers, busiest >= 2 * (busiest + other) / 2 reduces to other <= 0. The introspection logging dataflows run on every worker, so the other worker never reads exactly zero. For [1.0, 0.2], the average is 0.6 and 1.0 < 1.2, so workerVerdict returns balanced. If the replica is at 75% or more, cpuVerdict takes the highCpu branch, and WorkerCpuVerdict.tsx:202 prints "the busiest worker 100% of its core, so this isn't worker skew." Either way, the EXPLAIN ANALYZE ... WITH SKEW suggestion never appears, and BusiestWorkers collapses into a single range. The lifetime check in the first look (WorkerCpuVerdict.tsx:137) has the same ceiling.

This is the case the PR description motivates the drawer with: one worker pegged while cpu_percent looks modest. Four workers are affected too. In [1.0, 0.4, 0.4, 0.4] one worker is pegged at 2.5× each peer with the replica near 55%, and the verdict is still balanced, because the busiest worker has to outweigh all the others combined. The tests only use four-worker inputs with near-idle peers, such as [0.97, 0.14, 0.12, 0.13].

A fix is to compare the busiest worker against the mean of the other workers. That gives 5× for [1.0, 0.2] and 2.5× for the four-worker case. The existing test cases keep their verdicts: [0.21, 0.17, 0.16, 0.18] comes out at 1.24× and stays balanced.

[def-]

QA LLM Review

1. MEDIUM -- High-CPU verdict headline says "spread evenly" while its own body reports uneven work

console/src/platform/clusters/ClusterPage/WorkerCpu/WorkerCpuVerdict.tsx:249

This push turns uneven work under high replica CPU into a note on the highCpu verdict (workerCpuModel.ts:293). The dataflow-dominated branch still uses the headline "mostly running dataflows spread evenly across its workers", so the drawer's top-line finding contradicts the "Work is uneven, w0 at 85% … against 42% for the other workers" sentence directly below it.

Details

This branch is easy to reach. Take a 2-worker replica at one worker per core, with workers [0.85, 0.42] and the replica at 80% CPU. workerVerdict returns uneven, since 0.85 ≥ 2 × 0.42 and 0.85 < 0.9. cpuVerdict then returns highCpu with dataflowFraction 0.635 and outsideFraction 0.165. Because outside ≤ dataflow, Verdict takes the branch at line 246 and renders "r1 is using 80% of its CPU, mostly running dataflows spread evenly across its workers." Four workers at [0.85, 0.42, 0.42, 0.42] hit the same branch.

Before this push the headline was accurate, because an uneven verdict never reached highCpu. No test renders highCpu with uneven set. The model test at workerCpuModel.test.ts:272 uses [0.7, 0.2, 0.2, 0.2] at 90% CPU, which lands in the "mostly outside" branch instead.

One fix is to drop "spread evenly across its workers" from the label when uneven is set.

[def-]

QA LLM Review

1. MEDIUM -- Finished ad-hoc queries keep resetting the verdict window, so replicas loaded by ad-hoc queries never get a verdict

console/src/platform/clusters/ClusterPage/WorkerCpu/workerCpuSampler.ts:262

Any reading where a worker's counter fell is marked afterReset, and verdictWindowStart won't reach back past it. When a slow-path SELECT is running at one reading and has finished by the next, all of its accumulated time leaves the counters. On a replica whose CPU goes mostly to queries that take a few seconds, that happens every few readings. The window never reaches 30s, so the drawer shows "Measuring…" until the 10-minute pause and never gets to the "mostly outside its running dataflows" verdict that links to ad-hoc query troubleshooting.

Details

A worker's counter sums only the dataflows that are still running. A slow-path peek runs in a transient dataflow, which src/adapter/src/coord/peek.rs:1000 drops once the peek is queued, and handle_shutdown then retracts its time. Between two readings 5s apart, a counter moves by the new work minus whatever finished queries had already accumulated at the earlier reading. When maintained dataflows add little, that is often negative, and hasCounterReset has no tolerance for it.

I ran synthetic counters through the PR's hasCounterReset and verdictWindowStart, with the drawer's 30s gate. The load was 10% maintained work plus 70% from back-to-back queries, so the replica sits at 80%. With 2, 3, 4, 6 or 8s queries, 30 to 100 of 121 readings were resets, and no reading in 10 minutes had a verdict. With random 1.5 to 4.5s durations, 45 of 120 readings were resets and none had a verdict. At 20 to 30% maintained work, a verdict showed on roughly 55 to 60% of readings. At 40% or more, there were no resets. The worker chart breaks its line at the same readings, so most of its points fall below the 10s minimum.

workerBusyBetween only compares the window's first and last readings, so falls in between don't distort the verdict. One option is to break the window only when the last reading is below the first. A replica restart, or a dropped dataflow with more time than the window added, still trips that. A query that was running when the window started is undercounted by a few seconds, against minutes of new work.

[leedqin]

Both QA reviews are addressed in 87f9e42:

• Skew on two-worker replicas: the busiest worker is now compared with the other workers' average, in the live verdict and the first look. [1.0, 0.2] and [1.0, 0.4, 0.4, 0.4] both read as saturated, and the earlier cases keep their verdicts.
• Finished ad-hoc queries resetting the window: the verdict window and the chart line now compare only a span's two ends, so a fall in between doesn't break them. A restart, where every earlier reading is higher, still does. The model tests cover an ad-hoc load where every other reading falls.

[def-]

QA LLM Review

1. MEDIUM -- The saturated-worker verdict names a busiest process even when the processes are even

console/src/platform/clusters/ClusterPage/WorkerCpu/WorkerCpuVerdict.tsx:303

The new process sentence names whichever process has the highest mean CPU, with no evenness threshold. A process with many workers barely moves when one of them saturates, so on large multi-process replicas the sentence often credits another process with "the most CPU" by about a point. It then links to the processes drawer, which applies EVEN_RATIO (1.25) and calls the same processes even.

Details

busiestProcess (processReadingsModel.ts:152) is a bare argmax. ProcessesDrawer only calls a process busiest when standsOut holds (ProcessesDrawer.tsx:158). WorkerCpuDrawer.tsx:245 passes the raw result to Verdict, which always prints a sentence for it.

Take 2 processes of 62 workers, with w3 at 100% and every other worker at 20%. w3 adds (1 − 0.2)/62 ≈ 1.3 points to process 0, so process 0 reads about 21% and process 1 about 20%, a ratio of 1.06. Any larger difference between the processes, for example in work outside dataflows, decides which one comes out ahead. If process 1 wins, the verdict for w3 says "Process 1 used the most CPU, 21% against 20% for the other 1." That points away from the worker the verdict is about. "CPU and memory by process" then opens a drawer that says "CPU and memory are spread evenly across the processes." If process 0 wins, "Its process, process 0, used the most CPU" claims a lead the processes drawer doesn't count. The tests only use 61 vs 34 and 70 vs 40, both well past 1.25.

One fix is to share standsOut with the verdict, name a process only when it stands out, and otherwise say the processes used CPU evenly.

[leedqin]

Busiest process named when the processes are even is addressed in 80fef06. standsOut now lives in processReadingsModel.ts, and both the processes drawer and the worker verdict use it. The verdict names a process only when it passes 1.25× the others. Otherwise it says the processes used CPU about evenly and the skew is between workers, not processes, so it agrees with the drawer it links to.

[def-]

QA LLM Review

1. MEDIUM -- The busiest worker's breakdown says "about as much as the other workers" when it spent far less

console/src/platform/clusters/ClusterPage/WorkerCpu/BusiestWorkers.tsx:113

comparedWithOthers only checks the upper side of EVEN_RATIO, so any ratio below 1.25 reads "about as much as the other workers", including 0.2×. The breakdown is new in this push and shows on every verdict. When another worker carries one of the busiest worker's top objects, it calls that object even on this worker, while "Where the CPU went" flags the same object as skewed.

Details

This needs one object hot on the busiest worker and another hot on a different worker, for example two keyed objects whose hot keys hash to different workers, or a single-key reduce. Take 4 workers, a_idx at [20, 2, 2, 2] s and b_mv at [3, 3, 15, 3] s, with w0 the busiest. I rendered BusiestWorkers at the PR head with these rows, and w0's breakdown read:

• "a_idx 87% · 20s, 10.0× the other workers' average"
• "b_mv 13% · 3s, about as much as the other workers"

The other workers averaged 7s on b_mv, so w0 spent 0.43× their average. The table's hottest-worker cell for the same row reads "w2 · 5.0×" in red. Tests cover only ratios of 6× and exactly 1×.

One fix is to treat the band from 1/EVEN_RATIO to EVEN_RATIO as even, and below it say "less than the other workers' average", or give the multiple.

[leedqin]

Breakdown calling a much smaller share even is addressed in bbbd2b7. Even is now the band from 1/1.25 to 1.25. Below it, the line says "less than the other workers" and gives the multiple. When the object is skewed onto another worker, it also names that worker, so it agrees with the table's hottest-worker cell. Your a_idx / b_mv example is the new test: b_mv on w0 now reads "3s, less than the other workers, 0.4× their average, hottest on w2".

[def-]

QA LLM Review

1. MEDIUM -- Measure again is lost when clicked in the last minute of a reading session, and the report keeps the minute before the click

console/src/platform/clusters/ClusterPage/WorkerCpu/workerCpuSampler.ts:493

measure() reuses the open session, and that session's subscribe ends at its UP TO, 10 minutes after it opened. If the click lands less than about a minute before that bound, the sampler pauses for the limit before a new minute is measured. The tab then goes back to "Measured " with the finding from before the click and Measure again enabled, so nothing tells the user that the click did nothing.

Details

measure() sets measureAfterMs and isObjectReadDue and calls resume(), which returns early while reading. measureLatest needs per-object reads 60s apart after the click, but tick calls pauseFor("limit") once nowMs >= read.upToMs or the subscribe completes. isMeasuring stays true, but isMeasuringAgain (WorkerCpuPanel.tsx:209) also requires status === "reading", so the countdown disappears and the old span comes back. Nothing measures again until the user clicks a second time.

I reproduced this in the sampler's own test harness (createSteadyReplica, 2 workers), calling measure() at several times after start and advancing to 700s. Clicks at 520 to 535s each produced a measurement starting at the click. Clicks at 540s and 545s ended paused/limit with isMeasuring: true and the measurement still [0, 60000]. That is the last minute of every 10-minute session, about a tenth of the time the tab reads.

Measure again is the action the description recommends for checking a fix that was just deployed. Someone who clicks late in a session is shown the finding from before the fix. One fix: when less than MEASUREMENT_MS plus one object-read interval is left before live.upToMs, have measure() close the session and open a new one with a fresh UP TO, as it effectively does when paused.

[def-]

QA LLM Review

1. MEDIUM -- A report restored after 20 minutes loses its replica CPU, and its verdict changes

console/src/platform/clusters/ClusterPage/WorkerCpu/workerCpuSampler.ts:266

A restored measurement now stays until Measure again, and it is restored for up to 30 minutes after it ended (READINGS_RETENTION_MS). The panel reads replica CPU only from 20 minutes before it mounted (REPLICA_CPU_WINDOW_MINUTES, WorkerCpuPanel.tsx:87). If someone comes back 20 to 30 minutes after the measurement, the report is re-judged without CPU, and a high-CPU warning can turn into an all-clear for the same minute.

Details

The process-samples subscribe starts at mount time minus 20 minutes (useRollingWindowStartMs, then occurred_at >= since). samplesInMeasurement needs samples from a minute before the measurement's start, so it gets none, replicaCpuFraction is mean([]) = undefined, and cpuVerdict can no longer return highCpu. The PR's own cases show the flip:

• With one worker, [0.6] at 90% replica CPU reads "Not skew: r1 is at 90% of its CPU" with troubleshooting links. Restored, it reads "r1 runs one worker, so its work can't be uneven" (info).
• With four workers at 30% each and 90% replica CPU, the report reads "mostly outside objects". Restored, it reads "Work is spread evenly … Skew isn't slowing r1 down".

The skew verdicts also lose hasCpuMetrics, so the process sentence and the low-CPU note disappear.

Before this push, a restored report was replaced within about a minute. Now it persists, which breaks the sampler's comment that it "shows the last report as it was". samplesInMeasurement already cuts later samples so the finding stays fixed. One fix is to store the replica CPU, and the busiest process, with the Measurement when it is taken. Another is to start the samples window early enough to cover the restored measurement.

[def-]

QA LLM Review

1. MEDIUM -- The queries row's Query History link leaves out statements that began before the measured span, such as a long-running SUBSCRIBE

console/src/platform/clusters/ClusterPage/WorkerCpu/WorkerCpuPanel.tsx:121

The "Queries and subscribes" row counts every session dataflow that ran during the span. Its link (queriesPath, documented in ObjectCpuTable.tsx:296 as "where the statements behind the queries and subscribes row are listed") filters Query History to statements whose began_at falls between a minute before the span and its end. A SUBSCRIBE or long SELECT started earlier can be most of the row's time, and it never appears in the list the row links to.

Details

The list query bounds began_at >= start (queryHistoryList.ts, where("mal.began_at", ">=", startDate)). The range here is startOfMinute(startMs - 1 min) to startOfMinute(endMs + 1 min), so it covers at most about 5 minutes. A session's SUBSCRIBE runs as subscribe-<sink id> (sequencer/inner/subscribe.rs:287). objectRowId puts it in the queries row, and the carried row totals keep adding its time for as long as it runs.

Take an app that holds a SUBSCRIBE open on the cluster, like the one running in the PR's screenshots. Opened 10 minutes before the measurement, it takes most of the queries row's time. Clicking the row lists only the statements that began in the last few minutes, sorted by duration, so the short SELECTs at the top look like the cause.

Statement logging writes a statement's begin record while it runs (begin_statement_execution), so running statements can be listed. One fix is a second link to the cluster's running statements (finishedStatuses: ["running"]) over the list's full lookback. Another is to label the link as statements that started in these minutes.

[leedqin]

The four QA reviews since Oct 22 are addressed:

• Writes labeled as system work, in 580dafd. Only the coordinator's introspection-subscribe- and metric-sink- dataflows count as Materialize's own. Every other transient dataflow pools into Queries and subscribes, frontend-read-then-write-subscribe- included. The SQL tests run an INSERT ... SELECT and check that its dataflow lands there.
• Measure again lost in a session's last minute, in 2d1e942. Measure again now opens a new session first when the open one's UP TO is closer than a minute, plus one read interval and the counters' lag. Your click at 545 s is the new sampler test.
• Restored report losing its replica CPU, in 580dafd. The replica's CPU and busiest process are recorded on the measurement once the samples covering it load, and a restored report keeps them. Freshness is recorded the same way.
• Query History link missing long-running statements, in 2d1e942. The queries row now has two links: "started in these minutes", and "still running", which lists the cluster's running statements from the last day, oldest first, where a long-running SUBSCRIBE shows.

[devin-ai-integration]

Devin Review found 2 potential issues.

[devin-ai-integration]

🟡 Resized replicas retain phantom workers

When a replica changes worker count with no attributed rows, continues accepts the old worker totals. The empty rowTotals passes the length check, leaving removed workers in subsequent readings and producing incorrect CPU findings.

Learn more

The sampler saves its last worker totals in a reading and its per-dataflow state in a continuation. When no running dataflow has an attributed export, rowTotals is empty. Its every check accepts any worker count, even when the saved worker map came from a different size. accumulateWorkerElapsedNs copies that map, so removed workers remain present in later readings. workerBusyBetween includes every worker in the map, including the removed ones.

Example: A two-worker replica runs only unexported dataflows. After its sampler stops, resize it to one worker and reopen the tab. The empty saved row totals pass the check. The resulting measurement still includes w1, although the replica now runs only w0.

Recommended fix: Check the saved reading's worker IDs against workerCount before restoring previous, independently of whether rowTotals contains rows. Also invalidate a stored measurement when its worker count no longer matches the current replica.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[leedqin]

Fixed in 6d0af16. A new sampler now restores readings, object reads and the measurement only when the saved readings hold exactly this size's workers. Every dataflow runs on every worker, so that holds whether or not any rows were saved. The new sampler test is a 2-worker sampler followed by a 4-worker one, with no exported dataflow, and nothing carries over.

[devin-ai-integration]

🟡 Intermediate counter resets distort CPU reports

If a dataflow counter falls then exceeds its starting value within a minute, measureLatest accepts that span. It compares only endpoint rows, so a restart can yield a misleading CPU report.

Learn more

A measurement compares snapshots at its start and end, but the sampler also keeps intermediate object reads. A restarted dataflow can reuse an id, report a lower counter on an intermediate read, then surpass its original value before the end read. growthSince treats the decreased counter as a new dataflow and adds its new lifetime to the carried totals. The endpoint-only reset check misses this, and the measurement combines time from both lifetimes.

Example: A dataflow reads 5 seconds at the start, restarts and reads 1 second halfway through, then reads 30 seconds at the end. The end exceeds the start, so the span is accepted even though the intermediate read shows the reset.

Recommended fix: Check consecutive object reads across each candidate span for counter resets, or retain a reset marker on each read and reject candidates crossing one. Keep the existing endpoint check for a reset not captured by an intermediate read.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[leedqin]

A measurement no longer differences its two end reads. Each 5 s reading carries every row's total on by each dataflow's growth, and a counter below its last value counts as a new dataflow, so a restart inside the span is counted at the reading after it. The endpoint check is an extra guard. Your example is now a sampler test in 6d0af16: the dataflow starts at 5 s, restarts after 25 s and ends at 35 s, and its object gets all 60 s it ran. The two end reads alone would give 30.

[jdonelson]

The one-minute report is much easier to read and act on than the live charts, and thanks for following up on both of my earlier comments. Unit tests pass (758) and typecheck is clean. I didn't run the SQL tests. The main thing is the comment on workerVerdict: several findings contradict the bars or each other at the ladder's edges, and I wonder if the summary needs a different shape. The countdown and the CPU fill are smaller bugs, and the rest are nits.

[jdonelson]

I wonder if one ladder is the right shape for the summary. It answers several separate questions at once: is a worker at its limit, is the work uneven, is the replica out of headroom, and what's to blame. On a real replica two or three of these can be true together, but the ladder reports one, so the others get dropped or contradicted:

1. w0 at 100% with the others at 55% is only 1.8× the others, so it reads "Work is spread evenly… Skew isn't slowing r1 down".
2. 15 workers at 96% and one at 30% reads "every worker is busy… the work is already spread evenly".
3. Idle says "No object did measurable work" while the bars below show an object working, and it runs before the one-worker wording.
4. "Queries and subscribes" is the culprit in a skew finding, with cross join and hot key advice, but is skipped in the busy and high-CPU findings, which then name a smaller object as "the most".
5. The chart calls the busiest worker "Overloaded" whenever work is uneven, while the summary saves that word for 90%+.

The 2× cutoff also flips the story between 1.9× ("spread evenly") and 2.1× (overloaded). Could each question be answered on its own, with the headline from the most urgent and the body still stating the rest? Then a pegged worker always gets mentioned, and the chart and summary could share the same answers.

[leedqin]

Agreed, done in 25e3503. Each verdict now carries the workers at their core's limit, the headline comes from the most urgent finding, and the body still states the rest:

1. A worker at its limit that leads the others by 1.25× to 2× is its own finding: "w0 is at its core's limit… short of the 2× that counts as skew, but work routed to w0 waits on it." So 1.8× no longer reads as even, and 1.9× and 2.1× both name the pegged worker.
2. When workers average 90%+ but one lags, the summary counts the ones at their limit: "15 of 16 workers are at their core's limit, and the least busy, w15, is at 30%." The high CPU path does the same.
3. Idle says how busy the busiest worker was rather than that no object worked, and a one-worker replica gets its wording before the idle check.
4. largestShare counts the queries row like topObjectOnWorker does, so every finding blames the same row. A skew finding on the queries row drops the cross join and hot key advice and links to the queries that ran.
5. The bars take their label from the same verdict (markedWorker): "Overloaded worker" only when saturated, "Worker at its core's limit" when pegged, "Busiest worker" when uneven.

Balanced work between 1.25× and 2× now reads "Work leans a little on w0" rather than "spread evenly". All five of your cases are tests in WorkerCpuVerdict.test.tsx.

[jdonelson]

Thanks, this reads much better. One gap left: the skew and pegged findings don't use atLimit. With w0 and w1 both at 100% on different objects, it names only w0 and orders_idx, averages w1 into "the other 3 workers average 53%", and marks only w0's bar. The QA bot's latest review has the cases. Could these findings state the other workers at their limit too, and mark them on the bars?

[jdonelson]

If the browser tab is hidden (or the user pauses) during Measure again, the countdown reaches "0s left" and stays there about 30s before the report lands. The measurement restarts its minute after the pause, but the countdown still counts from the original click. Could the pause reset the countdown too?

[leedqin]

Fixed in 25e3503. A pause or a failed read now clears the asked-for measurement's start, and the first read after it starts the countdown again, so it counts the minute that will actually be measured. There's a sampler test that pauses partway through and resumes.

[jdonelson]

The Other CPU fill and the Replica CPU line read the same per-minute samples differently. The fill steps across each whole minute, while the line joins each minute's start. In the first screenshot, the fill sits at ~25% from 17:26:30 to 17:27 with no Replica CPU line above it. Maybe draw the line as steps too (curveStepAfter), so the fill always sits between the two lines?

[leedqin]

Done in 25e3503. The Replica CPU line uses curveStepAfter and holds its last sample to the end of that minute, the same minute replicaCpuAt gives the fill. A chart test checks the line's vertices step.

[jdonelson]

With tickFormat's clock times, the evenly split ticks land on odd seconds (17:24:13, 17:24:50, …), and once the window slides they change on every 5s reading. When a tickFormat is passed, could the ticks come from xScale.ticks(TIME_TICK_COUNT), so they land on round times and stay put?

[leedqin]

Done in 25e3503. With a tickFormat, TimeAxis takes xScale.ticks(TIME_TICK_COUNT), so the clock labels land on round times and stay put as the window slides. The relative-time axes keep their even split. There's a test that slides the window 5s.

[jdonelson]

Nit: when the snapshot completes, setState({ snapshotComplete: true }) notifies listeners before currentTimestamp moves to the new progress timestamp, so a listener calling getUpper() then gets the previous upper. The sampler avoids it by deferring, but since this is new on a shared class, maybe set currentTimestamp before that setState?

[leedqin]

Done in 25e3503. currentTimestamp now moves to the progress timestamp before the setState that announces the snapshot, with a test that a listener reads the new upper.

[jdonelson]

Nit: averageNs and extraNs aren't read outside a test. Can we drop them?

[leedqin]

Dropped in 25e3503.

[jdonelson]

Nit: the route parses with queryHistoryListUrlSchema, which also requires REQUIRED_COLUMNS, and strips the whole search when that fails. Parsing with it here (and at line 152) would make the test catch a link that opens the unfiltered list.

[leedqin]

Done in 25e3503. Both tests parse with queryHistoryListUrlSchema, and they pass, so the route keeps the links' filters.

[def-]

QA LLM Review

1. MEDIUM -- When several workers are at their core's limit, the saturated and pegged findings name only w0

console/src/platform/clusters/ClusterPage/WorkerCpu/WorkerCpuVerdict.tsx:489

The skew branch, which now also handles pegged, never reads the new verdict.atLimit, and markedWorker (workerCpuModel.ts:961) marks only one worker. When two or more workers are at their limit, the headline, the body, the focus object, the EXPLAIN suggestion and the bar marker all describe the busiest worker by id tie-break. The other workers at their limit are counted into "the other N workers average X%", so a second hot object goes unmentioned.

Details

I rendered Verdict at the PR head for these cases:

• 4 workers at [1.0, 1.0, 0.3, 0.3], with orders_idx hot on w0 and payments_mv hot on w1. The verdict is pegged with atLimit [0, 1] and reads "w0 is at its core's limit, and most of its time goes to orders_idx … The other 3 workers average 53%, so w0 does 1.9× their work." Neither w1 nor payments_mv appears, and only w0's bar is marked "Worker at its core's limit".
• [1, 1, 1, 0.3], with orders_idx spread evenly over the first three workers. This is pegged too: "w0 does 1.3× their work", followed by the EXPLAIN ANALYZE … WITH SKEW statement. Three workers are equally at their limit, and the real finding is that w3 lags. At an average of 90% or more, the same shape gets "15 of 16 workers are at their limit". At this 82.5% average it doesn't.
• 8 workers at [1, 1, 0.2 × 6] reads "w0 is overloaded". w1 is never named.

Only the highCpu, allBusy and balanced branches call atLimitSentence. A fix is to state the other workers at their limit in this branch when atLimit.length > 1, and to mark every worker in atLimit in the bars.

[jdonelson]

Thanks, this reads much better. One gap left: the skew and pegged findings don't use atLimit. With w0 and w1 both at 100% on different objects, it names only w0 and orders_idx, averages w1 into "the other 3 workers average 53%", and marks only w0's bar. The QA bot's latest review has the cases. Could these findings state the other workers at their limit too, and mark them on the bars?

[leedqin]

Re the QA review: fixed in d39f686.

• Several workers at their limit with none ahead by 2× (the first two cases): the finding is now about all of them, "3 of 4 workers are at their core's limit", with the least busy worker and the objects taking the most of their time. It no longer leads with the busiest worker's tie-break lead, and an evenly spread object gets no skew EXPLAIN.
• An overloaded worker beside another at its limit (the third case): the finding names the other worker, and the object busying it when that's a different one.
• The bars mark every worker at its limit.

[def-]

QA LLM Review

1. MEDIUM -- The multi-worker finding says no worker leads by 2×, then gives the hot-key cause for "that pattern"

console/src/platform/clusters/ClusterPage/WorkerCpu/WorkerCpuVerdict.tsx:709

When group is set, groupSentence replaces the body that said how the busiest worker's top object leans ("orders_idx, which runs 12.5× as much work on w0…"). The cause paragraph still renders whenever that object leans. So the finding states "none leads the others by the 2× that counts as skew", and the next paragraph begins "That pattern usually means a hot key" with no pattern stated. When the workers at their limit share the object, it also says the rows "all land on one worker", which contradicts the headline.

Details

I rendered Verdict at the PR head. With the PR's own test input ([1, 1, 0.3, 0.3], u5: [50, 2, 5, 5], u6: [2, 50, 5, 5]), it reads:

• "w0 and w1 are at their core's limit, and the least busy, w3, is at 30%. orders_idx and payments_mv take the most of their time. Work routed to them waits, though none leads the others by the 2× that counts as skew."
• "That pattern usually means a hot key: one value in a join or GROUP BY key holds most of the rows, so all of them land on one worker."

With u5: [50, 50, 3, 3], the headline is "2 of 4 workers are at their core's limit, and most of their time goes to orders_idx", followed by the same "all of them land on one worker" paragraph. orders_idx's ratio from w0 is 2.7×, so isLeaning holds even though its work sits on two workers. The new test renders the first case but asserts only on the group sentence.

One fix is to state the object's lean in groupSentence before the cause, and not to say "one worker" when the object leans on the group.