Skip to content

perf(storage): batch index node writes into one storage_store call per flush - #41

Merged
tiensonqin merged 1 commit into
mainfrom
devin/batch-store-writes
Oct 8, 2026
Merged

tiensonqin merged 1 commit into
mainfrom
devin/batch-store-writes

Conversation

@tiensonqin

Copy link
Copy Markdown
Contributor

Summary

Batch index node writes into one storage_store call per store flush, matching cljs *store-buffer* semantics. Fixes the ~50KB/block SQLite write amplification observed in the OCaml db-worker.

Root cause: restoring_node_storage.store_node calls storage.storage_store [single_node] once per stored node (impl/storage.ml). Graph_store.store wraps every call in its own sqlite transaction, so each ~300B node row paid a full commit (~4KB+ WAL). On a restored conn, 200 txs × 5 blocks produced 53,590 storage_store calls and a 58MB file.

Fix: batch_node_writes wraps a Ds.storage record — node-only batches accumulate in a pending buffer that flushes together with the first batch carrying the Storage_root/Storage_tail markers store_to_storage/store_tail append at the end of every write batch. storage_restore/storage_delete consult the pending buffer so deferred reads of just-written nodes still resolve mid-store.

The wrapped record must be the one bound everywhere node storages, db.storage_ref and conn.storage meet, so all per-node writes in a conn lifetime share one buffer:

  • store now prefers the bound storage_ref over a passed ?storage (cljs throws when asked to store to a different IStorage; preferring the bound one keeps that single-storage invariant), and always returns storage_ref = Some <bound>.
  • restore, restore_root_snapshot wrap their storage; empty_db/init_db/create_conn/conn_from_datoms/conn_from_db wrap ?storage in datascript.ml.
  • conn.storage derives from db.storage_ref so store_tail flushes the same buffer (one [tail] call per tx).

Wrapping an already wrapped storage stays correct (inner buffer still flushes on the outer flush call), so the entry-point wraps compose safely.

Measured (restored conn, 200 txs × 5 blocks, branching-factor 32 — bench/restore_probe.exe):

metric before (e439cba pin) after
storage_store calls 53,590 200 (1/tx)
sqlite file 58MB (~58KB/block) 6.4MB (~6.4KB/block)
kvs rows added 53,390 3,996 (same as cljs probe)
node restores 242,817 3,703

Row counts now match the cljs probe exactly (+3,996 live rows for the same workload); the remaining file growth is append-only orphan rows, which is the upstream-canonical trade-off for db-as-value — collect_garbage exists for reclaiming them but is a separate decision.

The db-worker picks this up via its datascript_ocaml dune.lock pin bump on devin/component-migration.

Link to Devin session: https://app.devin.ai/sessions/32c95e46f6504bcfa6e7f632ed797096
Open in Devin Desktop: https://app.devin.ai/desktop/session/32c95e46f6504bcfa6e7f632ed797096?variant=devin
Requested by: @tiensonqin

…r flush

cljs buffers every node write in *store-buffer* and commits a single
-store batch per tx; the restoring node storages called storage_store
once per stored node instead. Graph_store maps each call to one sqlite
transaction, so each ~300B node row paid a full commit (~4KB+ WAL),
producing ~50KB/block of write amplification in the db-worker.

Wrap the backend storage in batch_node_writes: node-only batches
accumulate in a pending buffer that flushes together with the first
batch carrying the root/tail markers store_to_storage/store_tail append
at the end of every write batch. The wrapped record is bound to the
conn/db at every entry point (store, restore, restore_root_snapshot,
empty_db/init_db/create_conn/conn_from_db) so the index node storages,
db.storage_ref and conn.storage all share the same buffer; store now
prefers the bound storage_ref over a passed-in storage, matching cljs's
single-storage-per-conn invariant, and conn.storage derives from it.

Measured on a restored conn (200 txs x 5 blocks, bf=32): storage_store
calls drop 53,590 -> 200 (one per tx) and the sqlite file 58MB -> 6.4MB,
matching the cljs probe's call count and live row count.
@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@tiensonqin
tiensonqin merged commit b346cdd into main Oct 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant