Skip to content

performance: capture a write from the process's own last publication instead of re-reading __manifest #837

Open

Description

@ragnorc

Problem

Most of a write's object-store requests re-read state that, under the v0.12 server's deployment model, the writing process itself produced.

Server runtime and online deployment designates one mutation-capable serving process per cluster root, with other writers excluded by the operator. Every production write to a graph therefore goes through one long-lived handle in one process. The writer still captures and revalidates as if another process may have published at any moment.

One single-row insert on main (97f293e8, after #834 moved the schema contract into __manifest and removed the schema-apply sentinel), measured with omnigraph-gqt --measure (DST, in-memory store, unit cost model), makes 13 serial requests:

# Requests What
6 puts table data file, transaction and detached manifest; __manifest data file, transaction and the version (the publish CAS)
1 get, table data the key-existence check
2 head + get, table manifest the table open (#818 serves it from the handle on a warm write)
3 lists of __manifest/_versions latest-version probes: at capture, in revalidation, at publisher load
1 get, __manifest manifest the publisher reloading the catalog state it then overwrites

Requests 6 to 13 run after mutation.post_stage_pre_effect_gate, under the branch gate, so 8 serial requests bound one branch's commit rate however many writers it has. (Before #834 the same insert made 20 requests, 12 of them gated; six _refs/branches listings and a __manifest data read are gone.) The server RFC's resource-bounds section notes that catalog publication cost grows with history; the per-write __manifest reload is part of that.

Proposed change

Direction, not a design. Capture a write from the state this process last published (schema, catalog, branch head, table pins), as an immutable snapshot it already holds. Keep the publication CAS (_versions/{v+1}.manifest written only if absent) as the cross-process fence. A foreign publication then fails the CAS, and the write re-prepares from a fresh load, as a lost CAS does today.

Expected effect on the insert above: about 9 serial requests, 7 with #818 (the 6 puts and the key check), and the gated section from 8 to about 5. It helps a single writer, which group commit (#785) does not, and it multiplies with group commit under concurrency.

What it has to keep:

  • Invariant 3 (one coherent attempt). The captured state must be one immutable snapshot, and a retry starts fresh.
  • Invariant 12 (mutable-tip caches are hints). The held state is a hint; the CAS stays the commit authority.
  • Branch identity. The CAS on a version file does not by itself show that the branch still has the same lifetime. A deleted and recreated branch is what the identity check catches today. In perf(merge): hold a merge's source only until capture, so source writes stop waiting #822's recreation test, it is ReadSetChanged on branch_identifier:t. One identity check before acknowledgement probably stays.
  • Schema apply from another process. Since refactor: inline the schema contract in the catalog #834 the schema contract publishes in the same __manifest CAS, so a foreign apply moves the version and fails a stale writer's CAS; the separate sentinel reads are already gone.
  • Embedded and CLI writers. These remain correct but may pay a CAS loss and a re-prepare after a foreign publish.

Affected areas

Alternatives considered

Not started. This should be measured on its own, separately from group commit, before any RFC is written.

Activity

  1. added
    acceptedTriaged and validated; open for a PR
    performanceCorrect result, wrong cost: work scales with the table, not with the delta or the query
    P-highHigh priority
    and removed
    needs-triageAwaiting maintainer triage
    on Oct 1, 2026
  2. ragnorc commented on Oct 18, 2026

    @ragnorc
    ContributorAuthor

    Summary

    A warm single-row insert still reads __manifest again for state this process just published. The cross-process fence remains the create-if-absent put of _versions/{v+1}.manifest. That put uses CommitBuilder::with_max_retries(0) in crates/omnigraph-catalog/src/commit.rs.

    On current main the comparable insert makes 14 object-store requests and a makespan of 12. The report at 97f293e8 was 13 serial requests. The extra request is the shape of the version probe after #962. The gated section is still 8 requests. The proposed capture still removes that gated catalog work.

    Reproduction

    Verdict: reproduced, with a different request count.

    • Commit: 0a62bb7b2aa5b99b012f413807c216277f7f9596 (main).
    • Tool: omnigraph-gqt --measure --model unit.
    • Environment: DST target omnigraph-engine-dst, storage in-memory-object-store, seed 0. Replay matched.
    • Case: a Person node with name as key, empty seed, then three inserts (alice, bob, carol) on one handle.
    • Command: omnigraph-gqt insert_one_row.gqt --measure --model unit.

    The second insert is the same shape as the filed measurement: the table already has a version, and the catalog row cache is warm. The tool reports 14 requests, 1 repeat, makespan 12, span 12.

    # Verb Object Role
    1–2 head, head (absent) __manifest/_versions for v and v+1 capture probe, in parallel (1 tick)
    3–4 head + get detached table manifest table open
    5 get table data file key-existence check
    6 put table data file staged data
    7–8 head, head (absent) __manifest v and v+1 revalidation, after mutation.post_stage_pre_effect_gate
    9–10 put, put table transaction, detached table manifest detached commit
    11 list __manifest/_versions publisher freshness check, publish.load_state
    12–14 put, put, put __manifest data, transaction, _versions/{v+1}.manifest publish, last put is the CAS

    Requests 7–14 run after the gate. That is 8 gated requests. Makespan of the step is 12 because each probe pair is one tick.

    The third insert on the same handle makes 16 requests, makespan 14, 2 repeats. fork.before_classify grows from 2 requests to 4: the detached commit heads and gets the same table manifest the stage just opened (the get is a 4096-byte range; the manifest put by the second insert is 33921 bytes). Those two requests are inside the gate, so the gated count becomes 10. The second insert does not do this re-read (its base manifest is 996 bytes and is read in full at the open).

    The first insert makes 27 requests, makespan 19. It is a different operation. It writes the full-text declaration (omnigraph_fts_compat.json and a second detached manifest; #909, after 97f293e8) and cold-loads catalog fragments in publish.load_state (11 requests: one list, one manifest get, nine data-range gets).

    What changed since 97f293e8

    #962 replaced the capture list and the revalidation list with two HEAD pairs (latest_attached_version in crates/omnigraph-catalog/src/lib.rs). Each pair is one makespan tick. That change turns two serial lists into four HEAD requests.

    On the warm path the publisher no longer gets the __manifest manifest body. GraphNamespacePublisher::dataset returns the dataset from published_rows when current_control_dataset reports that dataset is still latest (crates/omnigraph-catalog/src/publisher.rs, crates/omnigraph-core/src/instrumentation/control_dataset.rs). load_publish_state then uses cached_rows. The remaining catalog read is the listing inside Lance resolve_latest_location → current_manifest_path.

    Net against the filed table: the manifest-body get is gone, and two lists became four heads plus the publisher list that remains. The step is 14 requests and makespan 12, against 13 serial requests. The gated count on this second insert stays 8.

    The DST cost golden row for InsertV is 6 puts, 1 list, and 3 gets (crates/omnigraph-dst/cost_table.txt). That row does not record these probe heads. It is not this trace.

    Where each __manifest read comes from

    1. Capture. open_write_txn calls write_authority_for_known_branch(..., fresh: true) (crates/omnigraph/src/db/omnigraph.rs). For the bound branch, probe_latest_incarnation calls latest_attached_version, which heads v and v+1 together. A warm match returns the coordinator snapshot and does not open a new coordinator.
    2. Revalidation. commit_all takes the schema, branch, and table gates, then calls revalidate_write_txn (crates/omnigraph/src/exec/staging.rs). For the bound branch that function calls write_authority_for_known_branch again, so the same two heads run under the gate. The unbound fast path (manifest_probe.is_current) is not the path this handle takes.
    3. Publisher load. publish_with_precondition calls load_publish_state. On a warm main branch, dataset() calls current_control_dataset, and Lance lists _versions to resolve the latest location. The row batch is then the cached batch. The strict overwrite (max_retries 0) does not list again: Lance skips load_and_sort_new_transactions for that mode.
    4. The same three sites serve mutation and load. Both enter through open_write_txn and commit_all, and both publish through publish_with_precondition.

    Correctness constraints

    Invariant 3 requires one coherent accepted view, and a retry must start a new attempt. A held publication can be that view only when the attempt discards it after a lost CAS or a ReadSetChanged and loads again. The publish loop already does that reload (publish_with_precondition).

    Invariant 12 says a mutable-tip cache is a hint and the commit authority stays __manifest. The held snapshot must not replace the CAS. commit::overwrite already puts version v+1 only when that name is absent.

    Branch lifetime is separate from the version number. revalidate_write_txn returns ReadSetChanged on branch_identifier:<branch> when the identifier changes. For a named branch, probe_latest_incarnation reads the live identifier (get_live_manifest_branch_contents) because a delete and recreate can repeat version, e-tag, and timestamp. #822's recreation case is that check. A version-file CAS does not carry the identifier. The capture can skip the version probe and must keep one identifier comparison before detached table commits on a named branch. Main's probe uses BranchIdentifier::main() and does not do that read.

    A foreign schema apply publishes in the same __manifest CAS since #834. A stale writer then fails that CAS. The retry loads the new contract. Skipping the version probe does not remove that fence. It does lengthen the interval in which this attempt can write detached table files that the CAS then rejects. Those files stay unreferenced (invariant 5). The same interval already exists between today's revalidation and the CAS.

    Embedded and CLI writers in one process share the hold and skip the same reads. Two processes do not share it. The loser pays one CAS failure and one fresh prepare. That is the current conflict path.

    Recommended fix

    Keep the snapshot this process last published: catalog version, branch identifier, schema identity, graph head, and the published row batch. For the next attempt on that same branch, use that snapshot as the capture.

    Skip the capture HEAD pair, the revalidation HEAD pair, and the publisher _versions listing when the process-local record still names that publication. Keep the create-if-absent put of _versions/{v+1}.manifest as the only commit authority. On CAS conflict or ReadSetChanged, drop the snapshot and start a new attempt from a storage load.

    Keep the named-branch identifier comparison before any detached table commit.

    On the second-insert shape above, that removes four heads and one list: 9 requests, makespan about 9, and 5 gated requests (the two revalidation heads and the list leave the gated section). #818 removes the table-manifest head and get on a warm open, which brings the same shape to about 7 requests. The third insert's extra table-manifest head and get sit inside Lance's detached commit. #818 does not by itself remove that pair.

    This is the single-writer lever. Group commit (#785) amortizes one publication across several writers and leaves one writer's prepare cost in place. The group-commit RFC names this capture as the next lever and leaves it to #837. The two changes multiply under concurrency.

    Alternatives

    Leave the probes in place. A single-writer server keeps paying 5 catalog requests per insert, 3 of them under the branch gate.

    Ship group commit only. Concurrent writers share one CAS. One writer still pays the full prepare.

    Trust a Lance version hint for this skip. The hint is already in the pinned Lance 11.0.0 (lance-format/lance#6752). uses_version_hint is false when the store's listing is lexically ordered. S3 Standard, GCS, Azure, and the in-memory store in this measure do not use the hint. The hint also does not know that this process is the writer of v.

    Liability

    The new state is one process-local snapshot and the rule that clears it. The rule clears it on CAS conflict, ReadSetChanged, a branch change, and process start. A stale snapshot that also skipped the CAS could publish over a foreign schema apply or a recreated branch. This design keeps the CAS, so a missed clear becomes one failed attempt and unreachable detached files. The software liability added is that invalidation rule. The liability removed is four catalog reads, and one list, on every write that follows this process's own publication.

    Lance

    Pinned crate: lance 11.0.0 (Cargo.lock).

    Lance 11.0.0 already contains the version hint and CommitHandler::version_exists (HEAD of a named manifest). current_manifest_path still lists on a lexically ordered store. Strict overwrite already skips the conflict listing. Neither change removes the publisher list measured here.

    Lance 12.0.0 (2026-09-17) includes lance-format/lance#8800: put_if_predecessor for an external manifest store. The PR states that no built-in store implements it and that unconditioned commits do not change. OmniGraph publishes through the default object-store handler.

    Lance 13 includes lance-format/lance#8729: on a lexically ordered store, latest-version resolution stops after the first manifest instead of scanning a long window. The list request remains. That PR is not in the pin.

    A search of open lance-format/lance pull requests for manifest commit and latest-version resolution did not return a change that lets a writer skip latest-version resolution when it already holds the version it will commit with max_retries 0. An upstream Lance release does not close #837.

    Related work

    No open issue supersedes #837. This comment does not close it.

    Limitations

    The 97f293e8 binary was not run again. The comparison uses the table in the issue.

    concurrent-writes and concurrent-merges were not run. The reproduction is the single-writer measure the issue specifies. A second process still probes.

    The DST golden was not regenerated. Its InsertV row does not count the probe heads.

    The third insert's extra table-manifest read is measured. The Lance metadata-cache reason is not isolated beyond the manifest sizes above.

  3. ragnorc commented on Nov 13, 2026

    @ragnorc
    ContributorAuthor

    A smaller step under this issue, found in the 2026-11-13 storage audit (finding ID L-publisher-load-probe), at 9fb5b48e:

    • The listing left in the publisher (the "remaining catalog read" above) comes from GraphNamespacePublisher::dataset (publisher.rs:604-626) → current_control_dataset (control_dataset.rs:71-104) → Lance resolve_latest_location. On S3, GCS and Azure that is a LIST page of _versions/ on every publish, by every process, not only a writer-epoch holder.
    • The by-name probe perf(catalog): probe the next __manifest version by name instead of listing #962 built for reads (latest_attached_version, lib.rs:999-1018: HEAD v and HEAD v + 1 together) answers the same question. The HEAD of v also returns the ETag and size that same_location compares, so the warm dataset can be kept.
    • Keep the listing as the fallback for handlers with an identity, for a gap of more than one version, and on any error, as the freshness probe does. The version CAS on _versions/{v+1}.manifest stays the authority. The ETag form must match the listing's (fence "Local manifest identity").
    • Expected effect: publishing rows such as InsertV, KeyedInsert* and OverwriteWriter* in crates/omnigraph-dst/cost_table.txt go from 1 l.list to 0, at the price of two HEADs. Code inspection only; I did not attribute that LIST by a trace in this run.

    Report: underused-primitives-20261113.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P-highHigh priorityacceptedTriaged and validated; open for a PRfeatureFeature proposalperformanceCorrect result, wrong cost: work scales with the table, not with the delta or the query

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions