Repository navigation
perf(storage): batch index node writes into one storage_store call per flush - #41
Merged
Merged
Conversation
…r flush cljs buffers every node write in *store-buffer* and commits a single -store batch per tx; the restoring node storages called storage_store once per stored node instead. Graph_store maps each call to one sqlite transaction, so each ~300B node row paid a full commit (~4KB+ WAL), producing ~50KB/block of write amplification in the db-worker. Wrap the backend storage in batch_node_writes: node-only batches accumulate in a pending buffer that flushes together with the first batch carrying the root/tail markers store_to_storage/store_tail append at the end of every write batch. The wrapped record is bound to the conn/db at every entry point (store, restore, restore_root_snapshot, empty_db/init_db/create_conn/conn_from_db) so the index node storages, db.storage_ref and conn.storage all share the same buffer; store now prefers the bound storage_ref over a passed-in storage, matching cljs's single-storage-per-conn invariant, and conn.storage derives from it. Measured on a restored conn (200 txs x 5 blocks, bf=32): storage_store calls drop 53,590 -> 200 (one per tx) and the sqlite file 58MB -> 6.4MB, matching the cljs probe's call count and live row count.
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Batch index node writes into one
storage_storecall per store flush, matching cljs*store-buffer*semantics. Fixes the ~50KB/block SQLite write amplification observed in the OCaml db-worker.Root cause:
restoring_node_storage.store_nodecallsstorage.storage_store [single_node]once per stored node (impl/storage.ml).Graph_store.storewraps every call in its own sqlite transaction, so each ~300B node row paid a full commit (~4KB+ WAL). On a restored conn, 200 txs × 5 blocks produced 53,590storage_storecalls and a 58MB file.Fix:
batch_node_writeswraps aDs.storagerecord — node-only batches accumulate in apendingbuffer that flushes together with the first batch carrying theStorage_root/Storage_tailmarkersstore_to_storage/store_tailappend at the end of every write batch.storage_restore/storage_deleteconsult the pending buffer so deferred reads of just-written nodes still resolve mid-store.The wrapped record must be the one bound everywhere node storages,
db.storage_refandconn.storagemeet, so all per-node writes in a conn lifetime share one buffer:storenow prefers the boundstorage_refover a passed?storage(cljs throws when asked to store to a differentIStorage; preferring the bound one keeps that single-storage invariant), and always returnsstorage_ref = Some <bound>.restore,restore_root_snapshotwrap their storage;empty_db/init_db/create_conn/conn_from_datoms/conn_from_dbwrap?storagein datascript.ml.conn.storagederives fromdb.storage_refsostore_tailflushes the same buffer (one[tail]call per tx).Wrapping an already wrapped storage stays correct (inner buffer still flushes on the outer flush call), so the entry-point wraps compose safely.
Measured (restored conn, 200 txs × 5 blocks, branching-factor 32 —
bench/restore_probe.exe):storage_storecallsRow counts now match the cljs probe exactly (+3,996 live rows for the same workload); the remaining file growth is append-only orphan rows, which is the upstream-canonical trade-off for db-as-value —
collect_garbageexists for reclaiming them but is a separate decision.The db-worker picks this up via its
datascript_ocamldune.lock pin bump ondevin/component-migration.Link to Devin session: https://app.devin.ai/sessions/32c95e46f6504bcfa6e7f632ed797096
Open in Devin Desktop: https://app.devin.ai/desktop/session/32c95e46f6504bcfa6e7f632ed797096?variant=devin
Requested by: @tiensonqin