Repository navigation
Conversation
e3d6504 to
2c9a170
Compare
e03b526 to
d40a36c
Compare
QA LLM Review1. MEDIUM -- A replica hydrating a new object reports elapsed time since its last boot, often days
DetailsReproduced with the test file's default fixtures: both replicas booted 3 days ago and finished full hydrations, then r1 got one new object (
2. MEDIUM -- A cluster with no replicas shows a spinner forever and never gets its verdict
With no current replicas, DetailsClusters with replication factor 0 are a normal state: paused clusters, and scheduled ( The fix is to treat "no replicas" as complete. Either set |
d40a36c to
61113e6
Compare
|
Thanks, both are fixed in 61113e6:
|
61113e6 to
02a263c
Compare
| note={`${hydration.objectCount} ${hydration.objectCount === 1 ? "object" : "objects"}`} | ||
| /> | ||
| <Stat | ||
| label="Peak memory" |
There was a problem hiding this comment.
It's not clear that disk/swap is included in the memory number. Instead of putting that information in a tooltip, maybe add it in small text under the readings? Or maybe include the RAM number to make it clearer that "Memory" is the sum?
There was a problem hiding this comment.
Agreed. The detail now shows Peak memory, Peak disk and Peak heap (memory plus disk) separately, each as a share of its own limit with the bytes underneath.
02a263c to
2acffa5
Compare
2acffa5 to
8386788
Compare
QA LLM Review1. MEDIUM -- A replica that is offline or still starting reads as hydrated, and the verdict says every replica is hydrated
DetailsBoth cases come from how
In the probe, r1 was at 9 of 9:
A replica that isn't online, per |
8386788 to
b9acf58
Compare
b9acf58 to
0a76db4
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 3 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| {detail.isError ? ( | ||
| <Alert variant="error" message="Object timings couldn't be loaded." /> | ||
| ) : detail.isLoading || !detail.spans || !detail.summary ? ( | ||
| <Flex justifyContent="center" py={6}> | ||
| <Spinner /> | ||
| </Flex> |
There was a problem hiding this comment.
🟡 Catalog failure traps hydration detail in loading
When the object catalog fails, FinishedDetail keeps showing a spinner for a selected hydration. useHydrationDetail never clears isLoading without a catalog snapshot, even when object timings load.
Learn more
The history deliberately falls back to replica starts if the catalog fails, so a finished hydration remains selectable. useHydrationDetail keeps isLoading true until hasObjectNames, but does not read the catalog error. The failed catalog never completes its snapshot, leaving the detail spinner on screen even after the episode timings arrive.
Example: A catalog subscribe fails while the hydration history query succeeds. The history table offers a finished hydration, but selecting it leaves the detail spinner visible indefinitely.
Recommended fix: Read isError from useClusterObjectsLive in useHydrationDetail. On catalog failure, show the object IDs already used as fallback names instead of blocking the timings, or surface a terminal detail error.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in 9db574d: object timings no longer wait on a catalog that failed, and objects go by their ids then. Covered in ClusterHydration.test.tsx.
| lastFull={hydrations.findLast( | ||
| ({ kind }) => kind === "full", | ||
| )} |
There was a problem hiding this comment.
🟡 Latest full hydration uses wrong finish
When replicas hydrate concurrently, findLast can choose a hydration that finished earlier. transformHydrations sorts by start time, so the next-hydration facts use the wrong cutoff.
Learn more
transformHydrations sorts episodes by startedAt. NextHydration counts objects created after the chosen episode's finish, but findLast on that sorted array selects the last start, not the latest finish. Concurrent full hydrations can finish in the opposite order, giving an incorrect creation count and memory reading.
Example: Replica A starts at 10:00 and finishes at 10:08. Replica B starts at 10:01 and finishes at 10:02. An object created at 10:05 is reported as created since the last full hydration, even though A finished at 10:08.
Recommended fix: Choose the full hydration with the maximum finishedAt before passing it to NextHydration, matching the finish-time selection already used in hydrationVerdict.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in c809a93: latestFullHydration picks the full hydration that finished last, and Before the next full hydration uses it.
| {detail.isError ? ( | ||
| <Alert variant="error" message="Object timings couldn't be loaded." /> | ||
| ) : detail.isLoading || !detail.spans || !detail.summary ? ( | ||
| <Flex justifyContent="center" py={6}> | ||
| <Spinner /> | ||
| </Flex> |
There was a problem hiding this comment.
🟡 CPU failure hides available object timings
When the CPU peak request fails, FinishedDetail hides valid object timings and reports that they failed. useHydrationDetail.isError combines both requests, although only CPU failed.
Learn more
The detail runs separate requests for per-object timings and for the highest CPU sample. useHydrationDetail joins their error and loading states. The error branch here then hides the waterfall on a CPU-only error, and the loading branch delays it while the CPU query is pending.
Example: The episode-object query returns two hydrated objects, while the CPU peak query errors. The detail displays “Object timings couldn't be loaded” instead of the two objects and their waterfall.
Recommended fix: Expose separate timing and CPU loading/error states in useHydrationDetail. Gate the waterfall only on the object-timing request, and render a missing CPU sample separately.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in 9db574d: useHydrationDetail reports timings and CPU separately. A CPU failure shows on the CPU stat and leaves the waterfall in place.
c5d1828 to
03c3a78
Compare
03c3a78 to
9db574d
Compare
|
Thanks, fixed in 9db574d. An offline or not-yet-reporting replica reading as hydrated: a replica whose status is offline now reads as offline, whatever its last progress was. A replica that hasn't reported on a cluster with indexes or materialized views reads as starting. Both get a warning instead of a hydrated verdict. A replica with nothing to hydrate, on a cluster without compute objects, still counts as hydrated. Tests in |
QA LLM Review1. MEDIUM -- Selecting a retired replica says the cluster has no replicas
With a retired replica selected, the tab shows "This cluster has no replicas, so nothing on it is hydrated." on a cluster whose replicas are running. The verdict is computed from the narrowed replica list, which then holds no current replica. That also hides the cluster's real verdict, such as a current replica stuck restarting. Details
Reproduced with the scenario of the test "offers a retired replica while its history is in the window": Show the no-replicas verdict only when 2. LOW -- Replicas of a cluster with introspection disabled read as "Starting" forever
DetailsWith A cluster whose only compute objects are Inferring "starting" only until the replica's status turns online, or for a short time after that, keeps the new-replica case and avoids both of these. |
9db574d to
7030b0a
Compare
|
Thanks, both are fixed in 7030b0a:
|
7030b0a to
3941f8f
Compare
jdonelson
left a comment
jdonelson
left a comment
There was a problem hiding this comment.
Thanks for fixing the memory, disk and heap split. The detail card reads much more clearly now. Unit tests pass (898) and typecheck is clean. I haven't seen it in a browser either.
The one I'd fix before merging is the first comment: an offline replica in a restart loop reads as still hydrating, with a timer that keeps ticking. That's the main case this tab should catch. The others are smaller, and the subtitle is a nit.
| const state: ReplicaHydrationState = | ||
| restarts.length > 0 && (isOffline || isHydrating || hasNotReported) | ||
| ? "restarting" | ||
| : isOffline | ||
| ? "offline" | ||
| : hasNotReported | ||
| ? "starting" | ||
| : isHydrating | ||
| ? "hydrating" | ||
| : "hydrated"; | ||
| return { | ||
| replica, | ||
| state, | ||
| progress, | ||
| startedAt: isHydrating | ||
| ? currentHydrationStart( | ||
| bootedAt, | ||
| progress.unhydratedObjectIds, | ||
| createdAtMsById, | ||
| ) | ||
| : undefined, |
There was a problem hiding this comment.
Two ways an offline replica gets misreported:
- With restarts, an offline replica reads as "restarting", and
ClusterHydrationtreats that as working. It keeps showing its last progress (7 of 10) with a timer that keeps ticking, an in-progress mark on the chart, and a live subscribe pinned to the offline replica. Maybe an offline replica shouldn't get a start time or count as working, even when labelled restarting? - The verdict returns on a hydrating replica before it checks for offline ones, so "r1 is hydrating" hides "r2 is offline".
| const unfinished = findUnfinishedHydrations({ | ||
| transitions, | ||
| oomKills: statusHistory.data?.oomKills ?? [], | ||
| oomKills: [ | ||
| ...(currentStatus.data?.oomKills ?? []), | ||
| ...(droppedStatus.data?.oomKills ?? []), | ||
| ], | ||
| hydrations: allHydrations, | ||
| replicas, | ||
| bootTimes, |
There was a problem hiding this comment.
Before 26.43 there's no history, so every restart in the window looks unfinished. A replica that crashed once 1.5 days ago and is now hydrating a new index gets "r1 hasn't finished hydrating since … It restarted once". Maybe skip restart inference when hasHistory is false, since there's nothing to tell a failed restart from a recovered one?
| {peakPercent !== null && ( | ||
| <Fact | ||
| label="Memory used" | ||
| help={HYDRATION_HELP.lastFullMemory} | ||
| value={`${formatPercentage(peakPercent / 100, 0)} of the heap limit`} | ||
| note={`${formatPercentage(Math.max(0, 100 - peakPercent) / 100, 0)} headroom at its peak`} | ||
| /> |
There was a problem hiding this comment.
Two cards describe the same hydration, but name the same number differently:
Before the Next Full Hydration
Memory used 64% of the heap limit
Detail
Peak memory 75%
Peak heap 64% ← the same 64%
"Memory used" is really the heap (memory plus disk), so the reader sees "memory 64%" in one card and "memory 75%" in the other. Could it be labeled "Peak heap", like the detail? Its tooltip also says "at the last full hydration", while the detail's says, correctly, that the peak is recorded since the process started.
Separately, the objects tooltip in the same card says new objects "hydrate during the next one", but they hydrate right away, as the additive hydrations on the chart.
| function currentHydrationStart( | ||
| bootedAt: Date | undefined, | ||
| unhydratedObjectIds: string[], | ||
| createdAtMsById: Map<string, number> | undefined, | ||
| ) { | ||
| if (!createdAtMsById) return undefined; | ||
| const bootMs = bootedAt?.getTime() ?? -Infinity; | ||
| const createdMs = unhydratedObjectIds.map((id) => createdAtMsById.get(id)); | ||
| const isNewObjects = | ||
| createdMs.length > 0 && | ||
| createdMs.every((ms) => ms !== undefined && ms > bootMs); | ||
| if (!isNewObjects) return bootedAt; | ||
| const firstMs = min(createdMs, (ms) => ms); | ||
| return firstMs === undefined ? undefined : new Date(firstMs); | ||
| } |
There was a problem hiding this comment.
The elapsed time can be far off when an unhydrated object's creation time isn't in the lookup. The code then assumes the object is older than the boot and starts the clock at the boot. Two everyday cases:
-
Right after
CREATE INDEX, before the new index's creation time loads. On a replica booted 3 days ago, the alert flashes "r1 is hydrating: 8 of 9 objects, 72h so far", then corrects itself to "30s so far". The chart's y axis briefly stretches to 72h too.
Maybe leave the elapsed time out while a creation time is still loading? -
When the object was created before the selected range, since the lookup only covers that range. With a replica booted 3 days ago and an index created 30 hours ago still loading:
- Last 7 days: "30h so far" (correct)
- Last 24 hours: "72h so far"
startDateis optional inbuildObjectCreationTimesQuery, so maybe fetch creation times without the range? Every range would then give the same answer.
| ]; | ||
| // Replicas often hydrate at the same moment, so each is drawn this far from | ||
| // the next to keep both visible. | ||
| const REPLICA_OFFSET_PX = 4; |
There was a problem hiding this comment.
Replicas that hydrate at the same moment overlap on the chart. That's common with two replicas, since both restart after an upgrade and both hydrate a new index.
- Each replica's dot is offset by
REPLICA_OFFSET_PX = 4, but the dot is 12px wide, so the two read as one. - Each dot also has an invisible 24px click area (
r={12}), so the one drawn on top takes the clicks for the one underneath. With a mouse, only one of the two can be selected.
Maybe offset replicas by at least a dot's width?
| > | ||
| <SectionCard | ||
| title="Hydration History" | ||
| subtitle="From Materialize's hydration history" |
There was a problem hiding this comment.
Nit: the subtitle says "From Materialize's hydration history", but only the finished hydrations come from it:
- Restarts that never finished are inferred from replica status (the legend's tooltip says "Inferred").
- The in-progress dot comes from live progress.
Since the tab labels its inferences everywhere else, maybe something like "Finished hydrations from Materialize's history. Restarts inferred from replica status."?
3941f8f to
9a1e80c
Compare
9a1e80c to
e3965e8
Compare
8c9affa to
b668916
Compare
Second of two PRs adding a Hydration tab to the redesigned cluster page, on top of the one that adds its data. - A Hydration tab after Objects. Its link carries the replica choice, as the Metrics and Workers tabs' do, and the classic page redirects a Hydration link to its overview. - The replica select shared with the other tabs, and a time range of its own: 24 hours, 7 days or 30 days, the status history's retention. - A verdict: every replica hydrated and the last full hydration, a replica hydrating and how far along it is, or a replica that keeps restarting before it finishes, with the troubleshooting guide. - Replicas Now: live objects hydrated per replica, and, for a replica that is hydrating, the objects it is working on and how many wait for a hydration slot, read from the replica itself. - History: a dot per hydration over the window, by replica, filled for a full hydration and hollow for new objects, with restarts that never finished in a lane below and the typical full duration as a line. Each mark selects its hydration. - Before the next full hydration: the typical full duration on this size, objects created since the last one, and its memory peak. - The selected hydration's duration, peaks and sampled CPU, the object that worked longest, and a waterfall of each object's time waiting for a slot and working. - A table of every hydration, filterable by kind. - Tooltips say what each figure is, and which are the Console's inferences rather than the database's records. - Each section renders as soon as its own data arrives, and shows its own error.
b668916 to
b11844a
Compare
QA LLM Review1. MEDIUM -- The history drops every restart without an error when the current replicas' status history is loading or failed
The history's restarts and boot times come from DetailsI reproduced this with the scenario of "calls out a replica that keeps restarting before it finishes", with the current replicas' status query made to fail:
The same partial history shows during the reload after any replica is added or dropped, which happens on every resize or burst, and through the 3 production retries. |
QA LLM Review1. MEDIUM -- A replica that finished hydrating and then crashed is reported as restarting before it finished
The tab treats "no finished hydration recorded" as "didn't finish", but the collector only records a replica's finish if it samples that replica before the process restarts. A replica that hydrates and then crashes a few minutes later, as in an out-of-memory loop under steady-state load, gets "Restarted before finishing" rows and the red "hasn't finished hydrating" alert with a link to the hydration-failure docs. That is the crash-after-hydrating case the description says the history lets the tab tell apart. DetailsThe collector ( Example with 12 online replicas: r1 is created by a resize at 10:00, finishes hydrating at 10:04 and is OOM-killed at 10:09. Its samples were at 10:02, while it was still hydrating, and 10:14, after the restart. While it rehydrates, the tab says "r1 hasn't finished hydrating since 10:00. It restarted once, after running out of memory." The detail says it "ran out of memory and restarted before it finished hydrating." The NOTE on |
Motivation
Tenth PR in the cluster details redesign stack (#39509), on top of #40044, which adds the data. The redesign's prototype has a Hydration report tab. This adds a Hydration tab built on what the database records about hydration, and says which parts the Console infers. Part of CNS-173 and CNS-209. There's no Figma design; the layout follows the prototype and the PM's feedback on it.
Description
?replica=choice like the Metrics and Workers tabs, and the classic page redirects a Hydration link to its overview.Shared pieces: the stack's
ReplicaSelect, which can now mark a replica retired, aHYDRATION_TAB_PATHintabLinks.ts,percentOfexported fromprocessReadingsModel.ts,replicaStatusfrom the configuration drawer, each replica's size allocation fromuseClusterHydrationData, andPaginationControls, split out ofTablePaginationso the waterfall pages the same way tables do.This is larger than the stack's other PRs, about 2,400 lines of components and 1,200 of tests, because it adds the whole tab at once.
Verification
Not yet checked in a real browser. Worth a look: dark mode, chart spacing with many hydrations and a restart loop, the waterfall's pages on a cluster with many objects, the red heap stat, and a replica that is hydrating, for the live object list.
ClusterHydration.test.tsxrenders the tab against MSW handlers and the subscribe socket mock and covers:hydrationView.test.tscovers ordering finished and unfinished hydrations, restarts counted only since a replica last finished, when the current hydration began, offline and starting replicas, and each alert, including none once every replica is hydrated.ClusterDetail.test.tsxcovers the Hydration tab in the tab list, opening it from its link, and the classic page's redirect.🤖 Generated with Claude Code