Repository navigation
Open
Labels
P-mediumMedium priorityMedium priorityacceptedTriaged and validated; open for a PRTriaged and validated; open for a PRfeatureFeature proposalFeature proposalperformanceCorrect result, wrong cost: work scales with the table, not with the delta or the queryCorrect result, wrong cost: work scales with the table, not with the delta or the query
Description
Measurement
| Call | Wall time, 3 runs |
|---|---|
GET /healthz (round trip only) |
0.10 s, 0.11 s, 0.18 s |
omnigraph snapshot --json (metadata only) |
0.66 s, 0.58 s, 0.57 s |
query -e 'match { $p: Person } return { $p.slug } limit 1' |
0.36 s, 0.34 s, 0.36 s |
query -e 'match { $p: Person } return { count($p) as n }' (4,678 rows) |
0.45 s, 0.35 s, 0.35 s |
keyed lookup Person { slug: $s } |
0.42 s, 0.39 s, 0.38 s |
omnigraph version (CLI startup) |
0.01 s |
So a trivial served read costs about 0.35 s, of which 0.10–0.15 s is the round trip. The remaining ~0.2 s is server-side and unattributed. It is not manifest reloading: the warm-read contract (crates/omnigraph/tests/warm_read_cost.rs, since v0.7.1 #268) makes a warm same-branch read zero __manifest reads and one version probe. snapshot being slower than a query is itself unexplained.
For the agent workload (many small calls per task) this fixed floor outranks any plan choice; RFC 0047 (#791) makes this rollout step 7.
Ask
- Attribute the floor per phase on the server: auth/token verification, settings resolution, version probe (object-store round trips and their latency from the serving region), compile-cache hit, table open by location+version, the scan itself, result encoding. The existing
helpers::costobject-store counters andwarm_read_cost.rsgive the request counts; wall time per phase needs a tracing span or the additiveusageobject RFC 0047 (rfc: RFC 0047, search plan truth on engine v2 #791) proposes, which reports time per phase (authentication, target resolution, compilation, planning, execution, serialization). - Explain why
snapshotcosts ~0.6 s when a query costs ~0.35 s. - Cut what the attribution shows, in order of size. Caching of immutable version-pinned state is invariant-12-clean if that is where the time is; a second version probe or a per-request policy/token round trip would be a different fix.
Acceptance
A trivial served read's server-side time is reported per phase in the response (or a tracing export), and a cost test pins the object-store request count of that read at its current value so the floor cannot regress silently.
Metadata
Metadata
Assignees
Labels
P-mediumMedium priorityMedium priorityacceptedTriaged and validated; open for a PRTriaged and validated; open for a PRfeatureFeature proposalFeature proposalperformanceCorrect result, wrong cost: work scales with the table, not with the delta or the queryCorrect result, wrong cost: work scales with the table, not with the delta or the query
Activity
rfc: replace Phase A's caching item with attribution of the measured …
rfc: replace Phase A's caching item with attribution of the measured …
rfc: replace Phase A's caching item with attribution of the measured …
ragnorc commentedon Sep 25, 2026
Attributed on main (
80406fee) against a local RustFS, with and without toxiproxy adding 15 ms each way (~30 ms per request). The fixture is a 4,678-rowPersongraph, served withomnigraph-server --cluster s3://…. Times are medians of 7.GET /healthzlimit 1)countsnapshotWith fast storage the server-side floor is about 8 ms, so the floor is serial object-store round trips, not CPU.
Requests per warm call. I put a logging TCP proxy in front of RustFS; three runs per call were identical.
GET _schema.pgHEADandGET _schema.ir.jsonHEADandGET __schema_state.jsonLIST __manifest/_versions/snapshot, 14: the schema check and the_versions/LIST twice, then one more LIST and aGETof the manifest. That explains whysnapshotcosts about twice a query (question 2).Where the unattributed time goes. The 5 schema requests (about 3 serial round trips) come from
ensure_schema_state_valid, which re-validates the full schema contract on every operation so a long-lived handle catches external drift of the source, IR or state (see its comment incrates/omnigraph/src/db/omnigraph.rs). The warm-read contract inwarm_read_cost.rspins__manifestreads, so these schema-file reads are outside what it guards. At S3 latency they cost more than the manifest probe itself. Separately, thelimit 1read plans a full-fragment range (4,678 rows) rather than a single row.A cheap direction consistent with invariant 12: a conditional check (ETag or
If-None-Match) of the three contract objects in one parallel round trip, or folding a contract digest into the manifest the version probe already reads. Either would keep drift detection without two sequential round trips per request.ragnorc commentedon Oct 2, 2026
Re-measured after #834 (the schema contract moved into the catalog), with the setup of the 2026-09-25 comment: RustFS 1.0.0-beta.12, toxiproxy adding 15 ms each way, a logging proxy counting S3 requests, a 4,678-row
Persongraph served withomnigraph-server --cluster s3://…. Medians of 7 after two warm-up calls. One difference: this build is a debug build ofmain97f293e plus unrelated planner changes (RFC 0047 steps 2–3, which add plan acceptance on the CPU and no storage requests for these queries), so the fast-storage column carries debug-build CPU.GET /healthzlimit 1)countsnapshotThe five schema-contract requests per call are gone (
warm_read_cost.rs::warm_query_validates_schema_contract_oncenow pins zero), which was most of the floor. Every remaining request is one serial round trip:__manifest/_versions/once: the warm-read contract's one version probe.snapshotstill lists__manifest/_versions/three times before one manifest GET.server_snapshot(crates/omnigraph-server/src/handlers.rs) callssnapshot_of(target), which resolves the branch (one listing), theninternal_schema_version_of(target), which resolves it again and callsinternal_schema_stamp_at, which opens__manifestat its newest version (a listing and the manifest GET). So the storage-format stamp in the response comes from whatever version is newest at that moment, not from the snapshot it is returned beside. Resolving once and reading the stamp at the snapshot's own manifest version would make the response coherent and drop at least two of the four requests.What remains of this issue is the per-phase
usageobject (RFC 0047 step 7), plus thesnapshotdouble resolution above.ragnorc commentedon Oct 2, 2026
Proposal: the
usageobject (RFC 0047 step 7)RFC 0047 specifies "time per phase (authentication, target resolution, compilation, planning, execution, serialization) in an additive
usageobject". The re-measurement above shows the floor is now serial object-store round trips, so the object should report requests as well as time. Proposed shape onReadOutput(POST /graphs/{id}/query):total_msis that whole span, so it also covers anything between phases.compileis near zero on a compiled-query cache hit; the phase stays present.storage_requestscounts object-store requests issued while serving this read (the version probe and data reads), from the same wrapper the cost tests use today; it is omitted if counting is unavailable for a backend.application/vnd.apache.arrow.stream) cannot carry a body field before the rows, so they get the standardServer-Timingheader with the phases known before the first byte (authenticate,resolve,compile,plan). JSON responses carry the same header too, so proxies and browser tools see it.snapshotare out of scope for this step; the shape extends to them unchanged.To agree on:
storage_requestsin the object (it is the number that explains the floor).Server-Timingfor Arrow streams, and on JSON as well.ragnorc commentedon Oct 17, 2026
More data from a 0.13.0 server reached over HTTPS from a laptop, with the release CLI (10 ms start-up). The graph has 653 tables and about 4,900 commits.
/healthz, reused connectionlimit 0)@id, at the branchsnapshot --json(653 tables, 163 KB)schema show(199 KB)changes poll, caught upThe pinned-read surcharge has its own cause (snapshot-id reads skip the read caches) and its own issue, #1052. Clients that need a node with its relationships make one request per relationship group today, so this floor multiplies: a node with 92 relationship groups pays it about 185 times.