chore: Simplify and fix Snapshot metrics - #2452
Conversation
There was a problem hiding this comment.
I'm not sure if this is where we would want to do this because there is a lack of actionable information.
What do we now do once we receive this? We need to identify which query this is, but we have no way of doing so..
Perhaps we could explore a timer within the actual snapshot itself, and each snapshot taken automatically gets the caller LoC information embedded?
There was a problem hiding this comment.
I have added a LoC WARN log on long-held StateViews (StateSnapshots are not instantiated per-query, StateViews are).
The per-block log is still important in case we ever get snapshots or views that never end.
There was a problem hiding this comment.
Those two things are essentially synonyms. We should try improve our naming here. I assume its a snapshot because rocksdb calls them snapshots? Perhaps SmtView?
Is there a downside to having just a single one, instead of separate types? I can't imagine a snapshot is expensive to hold temporarily.
There was a problem hiding this comment.
StateViews and StateSnapshots are not synonyms - any number of views (one per API request) can map to a single snapshot. The view is there to enforce the invariants / API appropriate for accessing snapshot + SQL data consistently (block scoped requests). The snapshot is the non-SQL data itself.
There was a problem hiding this comment.
I understand that's what they are in our code. I'm saying the word view and the word snapshot mean almost exactly the same thing, and are therefore not good names for us to use.
If I say StateSnapshot everyone would assume that means a snapshot of our state at a moment in time. If I say StateView everyone would assume that means a view of our state at a moment in time.
There was a problem hiding this comment.
Understood. However, I find the terms view and snapshot to be quite appropriate here (differentiating between the data itself [snapshot] and how it is accessed [view]).
Snapshot (computing):
- A saved state of files or data used for quick backups and recovery.
View (database):
- Virtual table: A saved query that displays data from one or more base tables without duplicating the physical storage.
- Security: Hides confidential columns or rows by only exposing safe subsets of data to specific users.
- Simplicity: Wraps complex multi-table joins into a single, easy-to-query object.
SmtView is not appropriate because StateView is a view into both the trees and the data in SQLite.
Alternatives you might prefer, LMK:
- StateGeneration
- StateReader, ReadTxn
Summary
Closes #2451.
Previously, snapshot-lifetime observability was conditional logging with thresholds baked into the code:
StateView::dropwarning attributing long-held views. Conditional logs make poor metrics — the signal only exists once a hardcoded threshold trips, and alerting can't be tuned without a code change.This PR replaces all of that with unconditional span fields recorded on the
write_blockspan for every applied block:snapshots.lag_blocks— distance in blocks between the chain tip and the oldest still-pinned snapshot generation.snapshots.oldest_superseded_for_ms— how long the oldest pinned generation has been superseded (0 while the oldest pinned generation is still the latest). This fires while the offending reader is still alive, instead of once at release.snapshots.live— number of live snapshot generations.Alerts on slow or leaked readers can now be defined on these fields outside the codebase, with no thresholds in code.
How:
PublishedGenerations:recordstamps the previous latest generation as superseded when its successor is published, andadvancereports the oldest pinned generation's time-since-supersession. The clock still starts at supersession, not publication, so idle chains do not inflate the metric.SnapshotGuardis removed entirely. Its only jobs were maintaining the shared live-generation counter and emitting a release log; the live count is now derived fromPublishedGenerations(theWeakrefcounts are the same ground truth the counter proxied), so theArc<AtomicUsize>plumbing through the writer and lifecycle is gone.StateViewloses itsDropimpl,caller, andcreated_at. Detection of pinned generations comes from the per-block fields above; attribution comes from the instrumented request spans that acquire views, whose durations bound the hold time and identify the endpoint. With#[track_caller]gone,with_viewcollapses back to a plainasync fn.snapshots.oldest_superseded_for_msadded;snapshot.superseded_for_ms,snapshot.lifetime_ms,view.lifetime_ms, andcallerremoved (no emitters remain).Tradeoff: per-view and per-generation release events are gone, so exact view hold time is no longer distinguishable from overall handler duration. In practice views are request-scoped and one-per-handler, so span durations carry the same signal.
Changelog