Repository navigation
Description
Problem
Most of a write's object-store requests re-read state that, under the v0.12 server's deployment model, the writing process itself produced.
Server runtime and online deployment designates one mutation-capable serving process per cluster root, with other writers excluded by the operator. Every production write to a graph therefore goes through one long-lived handle in one process. The writer still captures and revalidates as if another process may have published at any moment.
One single-row insert on main (97f293e8, after #834 moved the schema contract into __manifest and removed the schema-apply sentinel), measured with omnigraph-gqt --measure (DST, in-memory store, unit cost model), makes 13 serial requests:
| # | Requests | What |
|---|---|---|
| 6 | puts | table data file, transaction and detached manifest; __manifest data file, transaction and the version (the publish CAS) |
| 1 | get, table data | the key-existence check |
| 2 | head + get, table manifest | the table open (#818 serves it from the handle on a warm write) |
| 3 | lists of __manifest/_versions |
latest-version probes: at capture, in revalidation, at publisher load |
| 1 | get, __manifest manifest |
the publisher reloading the catalog state it then overwrites |
Requests 6 to 13 run after mutation.post_stage_pre_effect_gate, under the branch gate, so 8 serial requests bound one branch's commit rate however many writers it has. (Before #834 the same insert made 20 requests, 12 of them gated; six _refs/branches listings and a __manifest data read are gone.) The server RFC's resource-bounds section notes that catalog publication cost grows with history; the per-write __manifest reload is part of that.
Proposed change
Direction, not a design. Capture a write from the state this process last published (schema, catalog, branch head, table pins), as an immutable snapshot it already holds. Keep the publication CAS (_versions/{v+1}.manifest written only if absent) as the cross-process fence. A foreign publication then fails the CAS, and the write re-prepares from a fresh load, as a lost CAS does today.
Expected effect on the insert above: about 9 serial requests, 7 with #818 (the 6 puts and the key check), and the gated section from 8 to about 5. It helps a single writer, which group commit (#785) does not, and it multiplies with group commit under concurrency.
What it has to keep:
- Invariant 3 (one coherent attempt). The captured state must be one immutable snapshot, and a retry starts fresh.
- Invariant 12 (mutable-tip caches are hints). The held state is a hint; the CAS stays the commit authority.
- Branch identity. The CAS on a version file does not by itself show that the branch still has the same lifetime. A deleted and recreated branch is what the identity check catches today. In perf(merge): hold a merge's source only until capture, so source writes stop waiting #822's recreation test, it is
ReadSetChangedonbranch_identifier:t. One identity check before acknowledgement probably stays. - Schema apply from another process. Since refactor: inline the schema contract in the catalog #834 the schema contract publishes in the same
__manifestCAS, so a foreign apply moves the version and fails a stale writer's CAS; the separate sentinel reads are already gone. - Embedded and CLI writers. These remain correct but may pay a CAS loss and a re-prepare after a foreign publish.
Affected areas
- Engine write path:
open_write_txn,revalidate_write_txn,commit_alland the publisher's state load inomnigraph-catalog. - Mutation and load entry points.
- Invariants 3 and 12.
- The group-commit RFC (docs(rfc): group commit, with the Lance composition guard #785), which names this as "the next lever" and would build on it.
- The
concurrent-writesbenchmark and bench: concurrent-merges overlap diagnostic #835'sconcurrent-mergesfor measurement, and the DST cost golden.
Alternatives considered
- Do nothing. A write keeps paying the 4
__manifestreads and the 8-request gated section. - Group commit only (docs(rfc): group commit, with the Lance composition guard #785). It amortizes the gated section across writers but leaves a single writer's latency unchanged.
Not started. This should be measured on its own, separately from group commit, before any RFC is written.
Activity
rfc(group commit): recheck after the schema contract moved into __man…
ragnorc commentedon Oct 18, 2026
Summary
A warm single-row insert still reads
__manifestagain for state this process just published. The cross-process fence remains the create-if-absent put of_versions/{v+1}.manifest. That put usesCommitBuilder::with_max_retries(0)incrates/omnigraph-catalog/src/commit.rs.On current
mainthe comparable insert makes 14 object-store requests and a makespan of 12. The report at97f293e8was 13 serial requests. The extra request is the shape of the version probe after #962. The gated section is still 8 requests. The proposed capture still removes that gated catalog work.Reproduction
Verdict: reproduced, with a different request count.
0a62bb7b2aa5b99b012f413807c216277f7f9596(main).omnigraph-gqt --measure --model unit.omnigraph-engine-dst, storagein-memory-object-store, seed 0. Replay matched.Personnode withnameas key, empty seed, then three inserts (alice,bob,carol) on one handle.omnigraph-gqt insert_one_row.gqt --measure --model unit.The second insert is the same shape as the filed measurement: the table already has a version, and the catalog row cache is warm. The tool reports 14 requests, 1 repeat, makespan 12, span 12.
__manifest/_versionsforvandv+1__manifestvandv+1mutation.post_stage_pre_effect_gate__manifest/_versionspublish.load_state__manifestdata, transaction,_versions/{v+1}.manifestRequests 7–14 run after the gate. That is 8 gated requests. Makespan of the step is 12 because each probe pair is one tick.
The third insert on the same handle makes 16 requests, makespan 14, 2 repeats.
fork.before_classifygrows from 2 requests to 4: the detached commit heads and gets the same table manifest the stage just opened (the get is a 4096-byte range; the manifest put by the second insert is 33921 bytes). Those two requests are inside the gate, so the gated count becomes 10. The second insert does not do this re-read (its base manifest is 996 bytes and is read in full at the open).The first insert makes 27 requests, makespan 19. It is a different operation. It writes the full-text declaration (
omnigraph_fts_compat.jsonand a second detached manifest; #909, after97f293e8) and cold-loads catalog fragments inpublish.load_state(11 requests: one list, one manifest get, nine data-range gets).What changed since
97f293e8#962 replaced the capture list and the revalidation list with two HEAD pairs (
latest_attached_versionincrates/omnigraph-catalog/src/lib.rs). Each pair is one makespan tick. That change turns two serial lists into four HEAD requests.On the warm path the publisher no longer gets the
__manifestmanifest body.GraphNamespacePublisher::datasetreturns the dataset frompublished_rowswhencurrent_control_datasetreports that dataset is still latest (crates/omnigraph-catalog/src/publisher.rs,crates/omnigraph-core/src/instrumentation/control_dataset.rs).load_publish_statethen usescached_rows. The remaining catalog read is the listing inside Lanceresolve_latest_location→current_manifest_path.Net against the filed table: the manifest-body get is gone, and two lists became four heads plus the publisher list that remains. The step is 14 requests and makespan 12, against 13 serial requests. The gated count on this second insert stays 8.
The DST cost golden row for
InsertVis 6 puts, 1 list, and 3 gets (crates/omnigraph-dst/cost_table.txt). That row does not record these probe heads. It is not this trace.Where each
__manifestread comes fromopen_write_txncallswrite_authority_for_known_branch(..., fresh: true)(crates/omnigraph/src/db/omnigraph.rs). For the bound branch,probe_latest_incarnationcallslatest_attached_version, which headsvandv+1together. A warm match returns the coordinator snapshot and does not open a new coordinator.commit_alltakes the schema, branch, and table gates, then callsrevalidate_write_txn(crates/omnigraph/src/exec/staging.rs). For the bound branch that function callswrite_authority_for_known_branchagain, so the same two heads run under the gate. The unbound fast path (manifest_probe.is_current) is not the path this handle takes.publish_with_preconditioncallsload_publish_state. On a warm main branch,dataset()callscurrent_control_dataset, and Lance lists_versionsto resolve the latest location. The row batch is then the cached batch. The strict overwrite (max_retries0) does not list again: Lance skipsload_and_sort_new_transactionsfor that mode.open_write_txnandcommit_all, and both publish throughpublish_with_precondition.Correctness constraints
Invariant 3 requires one coherent accepted view, and a retry must start a new attempt. A held publication can be that view only when the attempt discards it after a lost CAS or a
ReadSetChangedand loads again. The publish loop already does that reload (publish_with_precondition).Invariant 12 says a mutable-tip cache is a hint and the commit authority stays
__manifest. The held snapshot must not replace the CAS.commit::overwritealready puts versionv+1only when that name is absent.Branch lifetime is separate from the version number.
revalidate_write_txnreturnsReadSetChangedonbranch_identifier:<branch>when the identifier changes. For a named branch,probe_latest_incarnationreads the live identifier (get_live_manifest_branch_contents) because a delete and recreate can repeat version, e-tag, and timestamp. #822's recreation case is that check. A version-file CAS does not carry the identifier. The capture can skip the version probe and must keep one identifier comparison before detached table commits on a named branch. Main's probe usesBranchIdentifier::main()and does not do that read.A foreign schema apply publishes in the same
__manifestCAS since #834. A stale writer then fails that CAS. The retry loads the new contract. Skipping the version probe does not remove that fence. It does lengthen the interval in which this attempt can write detached table files that the CAS then rejects. Those files stay unreferenced (invariant 5). The same interval already exists between today's revalidation and the CAS.Embedded and CLI writers in one process share the hold and skip the same reads. Two processes do not share it. The loser pays one CAS failure and one fresh prepare. That is the current conflict path.
Recommended fix
Keep the snapshot this process last published: catalog version, branch identifier, schema identity, graph head, and the published row batch. For the next attempt on that same branch, use that snapshot as the capture.
Skip the capture HEAD pair, the revalidation HEAD pair, and the publisher
_versionslisting when the process-local record still names that publication. Keep the create-if-absent put of_versions/{v+1}.manifestas the only commit authority. On CAS conflict orReadSetChanged, drop the snapshot and start a new attempt from a storage load.Keep the named-branch identifier comparison before any detached table commit.
On the second-insert shape above, that removes four heads and one list: 9 requests, makespan about 9, and 5 gated requests (the two revalidation heads and the list leave the gated section). #818 removes the table-manifest head and get on a warm open, which brings the same shape to about 7 requests. The third insert's extra table-manifest head and get sit inside Lance's detached commit. #818 does not by itself remove that pair.
This is the single-writer lever. Group commit (#785) amortizes one publication across several writers and leaves one writer's prepare cost in place. The group-commit RFC names this capture as the next lever and leaves it to #837. The two changes multiply under concurrency.
Alternatives
Leave the probes in place. A single-writer server keeps paying 5 catalog requests per insert, 3 of them under the branch gate.
Ship group commit only. Concurrent writers share one CAS. One writer still pays the full prepare.
Trust a Lance version hint for this skip. The hint is already in the pinned Lance 11.0.0 (
lance-format/lance#6752).uses_version_hintis false when the store's listing is lexically ordered. S3 Standard, GCS, Azure, and the in-memory store in this measure do not use the hint. The hint also does not know that this process is the writer ofv.Liability
The new state is one process-local snapshot and the rule that clears it. The rule clears it on CAS conflict,
ReadSetChanged, a branch change, and process start. A stale snapshot that also skipped the CAS could publish over a foreign schema apply or a recreated branch. This design keeps the CAS, so a missed clear becomes one failed attempt and unreachable detached files. The software liability added is that invalidation rule. The liability removed is four catalog reads, and one list, on every write that follows this process's own publication.Lance
Pinned crate:
lance11.0.0 (Cargo.lock).Lance 11.0.0 already contains the version hint and
CommitHandler::version_exists(HEAD of a named manifest).current_manifest_pathstill lists on a lexically ordered store. Strict overwrite already skips the conflict listing. Neither change removes the publisher list measured here.Lance 12.0.0 (2026-09-17) includes
lance-format/lance#8800:put_if_predecessorfor an external manifest store. The PR states that no built-in store implements it and that unconditioned commits do not change. OmniGraph publishes through the default object-store handler.Lance 13 includes
lance-format/lance#8729: on a lexically ordered store, latest-version resolution stops after the first manifest instead of scanning a long window. The list request remains. That PR is not in the pin.A search of open
lance-format/lancepull requests for manifest commit and latest-version resolution did not return a change that lets a writer skip latest-version resolution when it already holds the version it will commit withmax_retries0. An upstream Lance release does not close #837.Related work
__manifest. The filed 13-request baseline is after this.ReadSetChangedonbranch_identifier.concurrent-mergesoverlap diagnostic.docs/rfcs/2026-09-29-server-runtime-and-online-deployment.md. One mutation-capable process per cluster root is the deployment this capture matches.No open issue supersedes #837. This comment does not close it.
Limitations
The
97f293e8binary was not run again. The comparison uses the table in the issue.concurrent-writesandconcurrent-mergeswere not run. The reproduction is the single-writer measure the issue specifies. A second process still probes.The DST golden was not regenerated. Its
InsertVrow does not count the probe heads.The third insert's extra table-manifest read is measured. The Lance metadata-cache reason is not isolated beyond the manifest sizes above.
ragnorc commentedon Nov 13, 2026
A smaller step under this issue, found in the 2026-11-13 storage audit (finding ID L-publisher-load-probe), at
9fb5b48e:GraphNamespacePublisher::dataset(publisher.rs:604-626) →current_control_dataset(control_dataset.rs:71-104) → Lanceresolve_latest_location. On S3, GCS and Azure that is a LIST page of_versions/on every publish, by every process, not only a writer-epoch holder.latest_attached_version, lib.rs:999-1018: HEADvand HEADv + 1together) answers the same question. The HEAD ofvalso returns the ETag and size thatsame_locationcompares, so the warm dataset can be kept._versions/{v+1}.manifeststays the authority. The ETag form must match the listing's (fence "Local manifest identity").InsertV,KeyedInsert*andOverwriteWriter*incrates/omnigraph-dst/cost_table.txtgo from 1l.listto 0, at the price of two HEADs. Code inspection only; I did not attribute that LIST by a trace in this run.Report: underused-primitives-20261113.md