Repository navigation
Description
Summary
TableHandleCache (crates/omnigraph-core/src/handle_cache.rs) holds up to 64 open Dataset handles keyed by (table_path, table_branch, version, staged_version, e_tag). Every write opens its base table through it (resolve_pinned_for_write → get_or_open in crates/omnigraph/src/db/omnigraph/promotion.rs), and every live-branch read does the same (SnapshotReadCaches in crates/omnigraph-catalog/src/lib.rs). Only a refresh or branch switch clears it.
Under steady writes each commit is a new key, so the cache fills with the newest 64 versions of the hot table. Each handle owns its own decoded Lance manifest, and a manifest lists every fragment. Live memory therefore grows as about 64 × fragment count × ~0.9 KB.
The cap's doc comment says a handle "holds only Arcs (object store + manifest), never table data, so this is cheap". That holds only while the manifest is small; its size is linear in fragment count.
Measurement
Setup:
- the
concurrent-writesbench scenario, one writer,file://, a release build of97f293e8, 64 seed rows; - a 5-second warmup and a 90-second window, two runs per arm;
- live heap read with macOS
heap <pid>at the end of the window.
The two arms differ only in TABLE_HANDLE_CACHE_CAP.
TABLE_HANDLE_CACHE_CAP |
commits (fragments) | live heap | 384-byte blocks | peak RSS | commits/s | p50 |
|---|---|---|---|---|---|---|
| 64 (current) | 2,829 / 2,915 | 197 MB / 197 MB | 186,145 / 193,078 | 376 / 435 MB | 25 / 26 | 38.3 / 36.7 ms |
| 4 | 2,811 / 2,887 | 40 MB / 40 MB | 21,931 / 22,651 | 234 / 263 MB | 26 / 26 | 37.0 / 37.5 ms |
- Live heap. The 60 extra entries hold about 157 MB.
- Block counts. The 384-byte block count is about 66 per fragment at cap 64 and 8 at cap 4: one per cached copy of each fragment's metadata.
- Throughput. It is unchanged, so the extra entries buy nothing for this workload.
- Lance session caches. Read through
Session::metadata_cache_stats/index_cache_statsat the end of the window, they hold 12–13 MB and about 0.1 MB in both arms. This growth does not come from the Lance cache defaults in server: data-plane Lance session caches use Lance defaults (6 GiB + 1 GiB per graph) — long-lived server grows until OOM-killed #702.
Without compaction, the curve looks quadratic in commit count. Each commit adds a fragment and a new cached version, and each of the 64 cached versions holds about as many fragments as the table has. One writer reached 918 MB peak RSS after 3,380 commits (cap 64).
Extrapolation: one hot table at 10,000 fragments makes the 64 entries about 590 MB. There is one cache per graph handle, so this repeats for every graph a server opens.
Why it matters
-
Invariant 11. Hot-path memory must scale with the working set, not accumulated history. Superseded versions are outside the working set:
- the next write stages on the newest pin;
- a live read opens the current snapshot;
- a reader still holding an older
Datasetkeeps its own clone.
Keys are version-exact, so evicting an entry costs at most one reopen and never changes a result.
-
Maintenance dependency. Only an operator running
optimizebounds the fragment count. The cache's memory should not depend on that. -
server: data-plane Lance session caches use Lance defaults (6 GiB + 1 GiB per graph) — long-lived server grows until OOM-killed #702. This is a likely contributor to the climb reported there. A long-lived server with sync writers grows with commit count, and a restart resets it.
Proposed direction
Bound the cache by the working set instead of version count:
- Keep only the newest version, or the newest two, per
(table_path, table_branch). Evict the superseded entry when a newer version of the same cell is inserted. - Keep the overall LRU cap on distinct cells.
- Alternatively, weigh entries by fragment count.
Snapshot-id reads already bypass the cache. #818 (hold the version a write committed) inserts the same keys and keeps the same bound, so the change applies to it as well.
Acceptance:
- Under a sustained single-table write load, the held-handle count per
(table, branch)stays at the chosen bound regardless of commit count. warm_read_cost.rsandwrite_cost.rsrequest counts are unchanged.
Reproduce
Build the scenarios bench at 97f293e8, then again with TABLE_HANDLE_CACHE_CAP set to 4, and run each:
RUSTFLAGS= cargo bench --locked -p omnigraph-engine --bench scenarios --no-run
<scenarios binary> --scenario concurrent-writes --writers 1 --write-branches 1 \
--duration-secs 90 --warmup-secs 5 --rows 64 --dims 8 --runs 1
While the child process runs, heap <child pid> | grep 'nodes (' gives the live heap.
Activity
ragnorc commentedon Oct 17, 2026
Reproduction and assessment
Summary.
TableHandleCachekeeps up to 64 openDatasethandles. Each commit to a table makes a new cache key, so under steady writes the cache fills with the newest 64 versions of the hot table. Each handle owns a full decoded Lance manifest, and a manifest lists every fragment. Memory therefore grows as about 64 × fragment count × 0.9 KB. Only the newest version is in the working set. Evicting an older entry costs at most one reopen and cannot change a result.Verdict: reproduced on
mainat4d7536d68b1db6da34bf9cff22a90d9fdd77bed4, with the same mechanism and the same cost per fragment. The absolute totals are lower than in the report because this VM commits more slowly.Environment, and differences from the report
97f293e84d7536d6(handle_cache.rsis byte-identical between the two)heap <pid>mallinfo2()(uordblks + hblkhd), read in-process at the end of the windowwait4peak/proc/self/statusVmRSS/VmHWMat the end of the windowscenarios --scenario concurrent-writesbenchmarks/deferred/). I used a scratch harness with the same workload: the sameChunk { slug @key, embedding: Vector(8) }schema, 64 seed rows, one closed-loop writer runningSession::mutateinsert onmain, a 5 s warmup and a 90 s window. It has no I/O probes and no readback verification.RUSTFLAGS=), with the cap read from an environment variable in the scratch buildProof 1: minimal failing tests on main
I added two tests in existing test owners. Neither is pushed. A test-only accessor,
TableHandleCache::held(), returns each key with its manifest's fragment count.omnigraph-core, the in-file tests inhandle_cache.rs. This test creates one Lance table with 80 versions (versionvhasvfragments) and callsget_or_openfor each version, as a writer does with its base pin.omnigraph-engine, the in-source tests inruntime_cache.rs. This test runs 80 single-rowSession::mutateinserts onmain, then inspectsread_caches().handles.The totals match the arithmetic for a sliding window of 64 versions (17..80 sums to 3,104; 16..79 sums to 3,040). The engine test is the end-to-end proof; the core test isolates the retention rule. Both can become the fix's regression tests by changing the assertion to the chosen bound.
Test code (engine test; the core test has the same shape)
Proof 2: the bench-shaped workload in a release build
Session::size_bytes()reads 9.5–10.0 MB in all four runs. The growth is not in Lance's caches.Root cause (code at
4d7536d)TableHandleKeyincludesversion,staged_versionande_tag(handle_cache.rsL82-L91), so each commit makes a new key.get_or_open(promotion.rsL90-L120). Live branch reads do the same (omnigraph-catalog/src/lib.rsL711-L740).LruMapevicts only by entry count (L43-L52). The cap is 64, and its doc comment says a handle is "cheap" (L75-L80).Datasetholdsmanifest: Arc<Manifest>(lancedataset.rsL198), andManifestholdsfragments: Arc<Vec<Fragment>>(lance-tablemanifest.rsL55). Each version decodes its own full list, so 64 versions hold about 64 × F fragment records.omnigraph.rsL2208-L2217, L2470-L2485,mutation.rsL1117). A single writer's own publications never reach those paths, so a single writer is the worst case. With several writers, the reprepare path clears the cache as a side effect.(version, e_tag)(get_manifest, L830) and weighs them byDeepSizeOfunder a byte budget (lance-corecache/mod.rsL191). A strongDatasetclone in the handle cache keeps a manifest alive after Lance evicts it, so the handle cache is the holder that has no byte limit.DatasetConsistencyWrapperas the model. That wrapper holds oneDatasetper table and replaces it inupdate()after a write (rust/lancedb/src/table/dataset.rs). OmniGraph keeps 64.This breaks invariant 11: hot-path memory must scale with the working set, not with accumulated history.
Recommended fix (correct by design)
A. Replace the superseded handle at the publication door. After a successful publication, the writer knows the exact pin it staged on and the exact pin it published, for each table it moved. Hold the new handle and remove the base key: a replace, not an add. Do this once, where the publication completes, so every writer gets it:
mutate,load, merge, schema apply, optimize and Blob writes. The cache then holds one version per table per graph branch in use, which is the working set. #818'shold_publishedbecomes the insert half of this replace.Why not "keep the newest N per
(table_path, table_branch)", as the issue proposes:table_branch = Nonebut have different pins.io/commit.rsL1065), andversionis a per-lineage counter. The cache cannot find a lineage from its key.B. Add a byte backstop. Weigh each entry by its manifest size (fragment count, or
Manifest::deep_size_of, as Lance does). Evict least-recently-used entries down to a budget, and always admit the newest entry. This limits the paths that A does not see:Weighing is O(F), but it runs only on a miss, which has already paid an O(F) decode.
C. Update the tests and the doc comment. Remove the "cheap" claim from the doc comment. Keep the two tests above as
issue_859regressions that assert the bound. Keep thewarm_read_cost.rsandwrite_cost.rscounts unchanged. If a labeled DST scenario gains a reopen (for example, after a fork that shares a base pin), updatecrates/omnigraph-dst/cost_table.txtand explain each changed line.Alternatives I set aside:
TableHandleCacheand rely on Lance's session cache. A pinned open whose manifest size is unknown skips Lance's cache (L836-L840) and costs I/O. That breaks the zero-open contract inwarm_read_cost.rs.Tradeoffs and long-term cost. A adds a small amount of code in one place, tied to the publication door that already exists. Its residual cost is that a concurrent reader at the base pin, or a fresh fork that shares that pin, pays one reopen. B adds one weight function and one internal budget constant. The design gives up a small amount of efficiency (an occasional reopen) for robustness (memory that does not grow with commit count). Correctness does not change, because keys stay version-exact and the cache stays derived state (invariant 7).
Related open PRs and issues
hold_published, which inserts each committed version under the same key shape into the same LRU. The bound stays at 64 versions per hot table. The held window moves one version newer, a negligible change. perf(writes): hold the version a write committed in the read-handle cache #818 does not make this issue better or measurably worse. Its saving needs only the newest version, so A fits with it directly: makehold_publishedremove the base key. It is best to land A with or before perf(writes): hold the version a write committed in the read-handle cache #818. A risk: if lance#10037 (below) lands, perf(writes): hold the version a write committed in the read-handle cache #818 would hold written manifests that also keep their encoded fragment bytes, which raises the cost of each held entry.Lance upstream assessment
Released versions do not fix this. OmniGraph pins 11.0.0. Since then, Lance has released v12.0.0 and v13.0.0 (latest), and v14.0.0-beta.19 is out. In v13.0.0 and v14.0.0-beta.19,
Manifeststill holdsfragments: Arc<Vec<Fragment>>, so each version still decodes its full fragment list. A Lance bump does not fix this issue.Fragment metadata trees may help later.
lance.manifest.layout=tree.lance-tableonly. Flat manifests stay the default, and nativeDatasetstill refuses the flag.If versions come to share decoded leaves in memory, the cost per held version could fall. But the feature is experimental and opt-in at table creation, and adopting it would be a storage-format decision for OmniGraph that needs an RFC and compatibility evidence. It does not replace a retention bound.
lance#10037 (open): a written manifest keeps its encoded fragment bytes. This may increase memory for held committed handles (see perf(writes): hold the version a write committed in the read-handle cache #818 above).
Lance's metadata cache is the model for B. It is already byte-weighed with
DeepSizeOf.Limitations
mallinfo2, not macOSheap. The absolute values are not directly comparable to the report; the ratio and the cost per fragment are.wait4peak. I ran two runs per arm, closed loop, onfile://only.blarghmatey commentedon Nov 5, 2026
A production data point for this, on S3 and with many graphs on one server, which the assessment above lists as unmeasured.
We run omnigraph-server v0.14.0 (release binary, glibc, single replica) against S3 in a 2 GiB container. It was OOMKilled four times in 14 days (11-02 14:41Z, 11-03 14:37Z, 11-04 14:17Z, 11-05 12:18Z). The pod restarts nightly, and memory sits at about 100-280 MiB until one client starts a write session. That client commits about once every 3.5s for hours, almost entirely to one long-lived branch of one graph: 6,736 commits to that branch over 11-02..11-04, with its manifest at version 3,831 by 11-04.
On 11-04 the working set went from 125 MiB at 07:00Z to 1,521 MiB at 09:30Z and 1,828 MiB at 10:00Z, then stayed between 1,829 and 1,875 MiB until the kill while the same writes continued (about 2,000 more commits). It was anonymous memory (container RSS 1,816-1,860 MiB, page cache 4-8 MiB). Each kill came when another client's writes landed on top of the plateau. On a day when a different client made 1,748 commits spread over many branches, the daily peak was 585 MiB. Each write from the hot session ran several
FilteredRead(ScalarIndex)plans reading 25-61 MB.What I can't separate from outside is how much of this is the handle cache vs. the Lance data-session caches from #702 vs. allocator retention after those read bursts. The shape (a fast step as the session starts, +741 MiB in 10 minutes on 11-04, then nearly flat) doesn't look like growth linear in commit count alone. We have no heap profile or cache stats from the server. A cache-occupancy metric (#702's fourth ask) would let us tell these apart in production. In the meantime we plan to raise the limit to 4 GiB and pace that client's commits.
fix(engine): evict a table version's cached handle once a write repla…
fix(engine): evict a table version's cached handle once a write repla…