Skip to content

rfc(gq3): the language, its search contracts, one break and the programme - #1302

Open
ragnorc wants to merge 43 commits into
ModernRelay:mainfrom
ragnorc:rfc/gq-3
Open

ragnorc wants to merge 43 commits into
ModernRelay:mainfrom
ragnorc:rfc/gq-3

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

What this is

One RFC for GQ 3, replacing four open drafts that specified one surface from four documents: the GQ 3 grammar (#1202), the GQ 3 programme (#1213), analyzed lexical search (#792) and RFC 0048 search contracts (#793). Those four close with this PR; #1244 is answered in the Alternatives table and on its thread.

docs/rfcs/2026-11-07-gq-3.md states, in layers that do not move:

  • Axioms (eight): totality with a declared cost, one snapshot, the plan as the truth, closure, typed semantics, one spelling per term, reads free of effects, quality declared not chosen.
  • Core: twelve operators over typed relations, three terminal write sinks, Call as compile-time inlining; every surface form desugars to exactly one core term, with the closure table and the envelope (first-order with counting and ordering plus two declared-quality operators; below Datalog, above relational algebra).
  • Surface: let and from inside match; arithmetic; retrieval as relation functions l2, nearest(…, k:, recall:), bm25, with rrf(…, k:, weights:) and union; one declaration keyword query, the body's shape deciding read or write; a write body that takes a read's clauses before insert, update $x, delete $x.
  • Search contracts: @index(p, fulltext, analyzer = …, scorer = …) with three immutable profiles and bm25_v1 over the field corpus; terms(…) consumed by match_terms and bm25; Vector(n, distance = …); embedding recipes; the exact scan as oracle under every index state; the eleven laws and four populations of RFC 0048, with its reviewed default-window inconsistency resolved by construction (k is required).
  • The break: edition 3 under gq 3;, seven edition-2 spellings rewritten by queries migrate with a typed-IR proof (six) or rows-equality (rrf); a one-release edition-2 window rewritten at boot; queries fmt.
  • The programme: stage edges, slot owners, verdicts on every open language, search and access RFC, and the rules every later change answers.

Assets

  • assets/2026-10-28-gq-3-grammar-evaluation.md: the whole-language evaluation (outcome paragraph amended for closure).
  • assets/2026-11-07-gq-3-grammar-comparison.md: every GQ 3 production against ISO GQL, Cypher, SQL and Datalog.
  • assets/2026-11-07-gq-3-before-and-after.md: forty-two kinds of query in edition 2 and edition 3, "Not supported" where edition 2 cannot express one.

Checks

typos, python3 scripts/check-docs.py (473 files OK), git diff --check.

…ramme

One RFC replaces four open drafts: the GQ 3 grammar (PR 1202), the GQ 3
programme (PR 1213), analyzed lexical search (PR 792) and RFC 0048 (PR 793).
It states eight axioms, a twelve-operator core with three write sinks, the
surface (let, from, arithmetic, relation functions, one query keyword),
the search contracts restated on that surface, the seven-row break with its
rewriter, and the stage programme with verdicts on every open language,
search and access RFC. Three evidence assets accompany it: the whole-language
evaluation, a construct-by-construct grammar comparison against GQL, Cypher,
SQL and Datalog, and an edition 2 against edition 3 corpus by kind of query.
@ragnorc ragnorc changed the title rfc: GQ 3, the language, its search contracts, one break and the programme rfc(gq3): the language, its search contracts, one break and the programme Nov 8, 2026

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of d626e288 (RFC: GQ 3)

Why. GQ grew one feature at a time. The four drafts that tried to fix this (#1202, #1213, #792, #793) drifted from each other. This RFC puts them in one document with four layers: axioms, a core algebra, the surface syntax, and the search contracts. Then it plans one breaking release (edition 3) and a sequence of additive stages.

What I checked. I read the whole RFC and spot-checked the two assets. I compared the claims about today's behaviour with the code at the PR base ea4a16d5 (the RFC cites 03e0b623, which is on main) and with crates.io Lance 13.0.0.

Outcome: revise before acceptance. Consolidating the drafts removes a real liability: there are fewer documents to drift and one decision log. The search section is accurate about today's behaviour. But the write half of the design changes two engine rules that the RFC says do not change. Also, three acceptance gates in blocked_on cannot pass as the text is written now. The four blocking items below are about the text. None of them needs a new mechanism.

Blocking

B1. The write examples break the D₂ rule, which the RFC says is "unchanged". close_company (line 515) and retire (line 1189) each delete $k and update $p in one body. Today the engine refuses this before any I/O (enforce_no_mixed_destructive_constructive, crates/omnigraph/src/exec/mutation.rs:757-795). This is not only a parser rule. Other code depends on it: the delete-count deduplication (mutation.rs:726-731, "If D₂ is ever relaxed, this dedup must be revisited") and the commit-path change discovery (changes/candidate_scan.rs:23-26, "one engine-authored transaction is either a row-set-preserving insert/update or a delete, never both"). Invariant 4 also names the split. Choose one of two options:

  • Keep D₂. Make the examples legal by splitting them, and state that a mixed body is a typed refusal.
  • Lift D₂. Then say so in Invariants and Compatibility, name the invariant-4 change, and add the deduplication and change-discovery proofs to Evidence.

The examples are the RFC's main argument for writes fed by reads, so this choice changes the design.

B2. One shared clause relation per write body changes the meaning of multi-statement mutations, so the rewriter's IR-equality gate cannot pass. The grammar allows clauses only before all statements (mutation_body = (!stmt_keyword ~ clause)* ~ mutation_stmt+). Each statement then "runs once per row of the clauses' relation". Edition 2 bodies often have several statements, and each one has its own where. Here are three ways the rewrite goes wrong:

  • update Person … where name = "alice" and update Marker … where name = "initial" (cases/detached_multitable_update_failure_after_table_2.gqt, w(): four updates on four types). The rewrite gives one product of four bindings. If any one predicate matches nothing, the product is empty and no update runs.
  • A clause-less insert beside a rewritten update … where. The insert now runs once per clause row, not once.
  • Edition 2 has read-your-writes inside a mutation: an update … where scans the earlier statements' pending rows (scan_with_pending, mutation.rs:1809-1825). The RFC says "a later statement cannot read an earlier statement's effect". The two IRs can be equal and still run differently.

A simple scan found 17 declarations in crates/omnigraph-gqt/cases/ with two or more statements, at least one of them update/delete … where. These declarations cannot meet "zero hand edits … typed-IR equality". To fix this, give each statement its own scope (for example, a statement-local clause block, or one clause relation per statement). Or make the rewriter refuse these bodies and remove them from the gate. In both cases, state the read-your-writes decision explicitly.

B3. The rules for ranking populations contradict each other, and the rewriter writes text that the type checker refuses. §5 says a ranking's population "must carry exactly one node column". But:

  • The §6 example ranks { … return { $r, $p } }, which has two node columns.
  • The rewriter's nearest, bm25 and rrf rows write a population that "returns every variable the pattern binds". Every ranked edition-2 query with a traversal therefore gets two or more node columns. The user guide documents this shape as supported ("Ranking a traversal-reached binding").
  • Fuse returns R[$x, s: F64] (§2), so the other bindings are gone after from rrf(…). Law 4 and §4 say "fan-out rows of one target carry its score", and today's fused order keeps every fan-out row (docs/user/search/index.md, "an entity that fans out to several rows contributes each of them").
  • ApproxTopK says "at most k rows", but law 4 implies at most k targets, plus their fan-out rows.

One rule fixes all four: a population may carry any columns. The ranked column is named by the field argument (for example bm25(rel, $p.text, q) instead of a bare text). The output is every population row of the selected targets, so k bounds targets and not rows. Fuse keeps the arms' columns, or the rewriter joins them back.

B4. Axiom A6 is stated as absolute, but the RFC keeps several double spellings, so the "desugaring-uniqueness test" in blocked_on fails by construction. A6 says that two surface forms with one core term "are a violation". The RFC keeps all of these pairs:

  • exists { p } and count { p } > 0
  • not { p } and count { p } = 0
  • the property braces and the equivalent filter (§6 says they "are one Filter")
  • both triple orders
  • { tags: "rust" } and "rust" in $d.tags (both lower to CompOp::Contains: ir/lower.rs:898-905, ir/coerce.rs:371-387)
  • search, match_text, fuzzy and match_terms(…, terms(…))

fmt and lint handle all of these. That is a good policy, but it is not the policy that A6 states. Restate A6 as "one canonical spelling per core term; every other accepted spelling is listed in an alias table that fmt rewrites and lint names". Then make the test check that table. After that change, the Alternatives rows that refuse a form only because of A6 (a scalar l2, @approximate, where on statements) need a second reason, or a note that they could become alias rows.

Should fix

  • The rrf rewrite has no proof for stored user text. The rewriter "refuses to write when the edited text does not compile to a typed IR equal to the original's", but the rrf row is not IR-equal by design. Its rows also differ when a traversal removes some of an arm's window. Today a nearest arm in rrf is a top-k window taken before a postfilter that the rrf_plan gate chooses (docs/user/search/index.md:40-42, 86-88). The rewrite ranks inside the population. That change is a fix, but "proved rows-equal on the corpus" proves nothing about a user's data. Classify the row as a change in meaning: migrate reports it and does not apply it silently, and the boot window treats it the same way.
  • The edition window covers only sources held in the registry. A file without a gq line is read as edition 3 (line 927), so -e, --query files, lint and a cluster apply of an unmarked source directory refuse edition-2 constructs at once. This contradicts "edition 2 compiles for one release". Define how the window classifies unmarked text. Also name the format owner of the edition that the registry newly records for each source.
  • Unsigned integers and integer-to-float widening. The value table (line 197) has no U32 or U64, but both are in the grammars (query.pest:144, schema.pest:47). The arithmetic rules do not define U64 ± I64. "An integer with a float is F64" silently loses precision above 2^53. Invariant 9 asks for an explicit Cast in that case. Add the unsigned types and make the widening a visible Cast, or refuse it.
  • Fan-out in writes. An Update whose relation has two rows for one entity, with different assigned values (update $p set { city: $c.city } with two $c), is not defined. An unkeyed edge insert creates one edge per fan-out row. Make conflicting assignments a typed error (invariant 8) and state the cardinality of an insert.
  • Diagnostic codes. T17 currently means "nearest ordering requires a limit", and the RFC reuses it for "nearest without k". codes.rs says a published meaning is frozen, so allocate a new code. §5 lets update take "a bound node or edge variable", but T16 refuses an update of an edge today. Say that T16 stays.
  • Grammar consistency.
    • The weighted example in the before-and-after asset writes { from l2(…) return { … } }, with no match. The §7 grammar refuses that (inline_read = "{" ~ read_body ~ "}", and read_body starts with match_clause).
    • §3 says a write body has "zero or more match clauses", but the grammar and every example use bare clauses. Choose one form and pin it in a parser test.
  • Line 760. order { $distance, $x.id }: $x.id is a user property, and the identity is $x.@id. Order already breaks ties by identity, so the second key can be removed.

Optional

  • k: means the window size on nearest and the RRF constant on rrf, often in one expression (rrf(nearest(…, k: n), …, k: 60)). A different option name for the constant would remove a likely agent error.
  • english_v1 copies Lance's filter order: stemming comes before stop-word removal (lance-index 13.0.0 tokenizer.rs:1082-1095). A profile is immutable, so state that this order is a deliberate choice.
  • The registry row for 2026-11-07 is above the 2026-10-29 row. docs/rfcs/README.md says rows are "by date".

Verified as correct

  • GQ_LANGUAGE_VERSION = (2, 3) (omnigraph-compiler/src/lib.rs:48) and EXPLAIN_VERSION = 8 (omnigraph-planner/src/explain.rs:34).
  • The Lance 13 default analysis is simple, a 40-byte token filter, lowercase, stem, stop words and ASCII folding (lance-index 13.0.0 tokenizer.rs:634-657), so english_v1 differs from it only by the token filter and NFC, as the RFC says.
  • search and match_text build FullTextSearchQuery::new, which is Operator::Or (query.rs:391-398), and fuzzy with no budget is 2 edits (engine/scan.rs:961-965).
  • The list-contains to in rewrite is IR-equal.
  • #1202, #1213, #792 and #793 are closed and not merged.

Liability

The core adds Correlate with a choice for the planner, a plan budget, and cardinality estimates across a correlation. These are real long-term costs, and the RFC names them. The grammar-pin rule ("no new rule without an amendment") is a good control for what the language looks like after five more changes. The large unnamed liability is in B1 and B2: two rules of write execution change, but the RFC calls them unchanged. The engine comments say that this is exactly the complexity D₂ prevents.

Validation

  • Local runs in a detached worktree at d626e288:
    • python3 scripts/check-docs.py with the pinned docs requirements: "Documentation OK (473 Markdown files checked)".
    • bash scripts/check-agents-md.sh: OK.
    • git diff --check origin/main...HEAD: clean.
    • typos was not run, because the binary is not on this host.
  • CI at d626e288 (04:31Z): 25 pass, including PR Title, GQT (dst), the DST pinned suite, RustFS S3 and Azurite; 4 skipping; 2 pending (Test Workspace, Storage Upgrade Compatibility).
  • Regression proof does not apply: the PR changes only RFC text and contains no code or query change.
  • The corpus count in B2 comes from a source scan of .gqt files, not from running a rewriter.
  • I did not build Lance and did not run any search instrument.

Comment thread docs/rfcs/2026-11-07-gq-3.md Outdated
plan. No semantics pass through strings, settings, transport flags or side
tables. Every refusal is a typed code with the fix that makes the text legal.

**A6. One spelling per term.** Two surface forms that desugar to the same

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B4: as stated, A6 is violated by forms this RFC keeps (exists/count > 0, not/count = 0, braces and the filter, both triple orders, list braces and in, the three lexical rows and match_terms). Restate it as one canonical spelling plus an alias table that fmt rewrites, and make the uniqueness test check that table.

Comment thread docs/rfcs/2026-11-07-gq-3.md Outdated
endpoint assignment being a key literal or a bound reference. There is no
`where` on a statement: the clauses are the selection, and a second
predicate on the sink would be a second spelling of a filter (A6). Each
statement runs once per row of the clauses' relation; its assignment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B2: one clause relation shared by all statements makes an edition-2 multi-statement body a product. If one statement's where matches nothing, the other statements do not run, and a clause-less insert runs once per row. It also removes today's in-mutation read-your-writes (scan_with_pending). There are 17 such declarations in cases/, for example detached_multitable_update_failure_after_table_2.gqt w().

query close_company($slug: String) {
$c: Company { slug: $slug }
$p $k:worksAt $c
delete $k

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B1: delete $k with update $p in one body is refused today by D₂ (exec/mutation.rs:757-795), which the RFC says is unchanged. The delete deduplication (mutation.rs:726-731) and commit-path change discovery (changes/candidate_scan.rs:23-26) depend on D₂. Keep D₂ and split the example, or lift D₂ explicitly with those proofs.

Comment thread docs/rfcs/2026-11-07-gq-3.md Outdated
match {
from bm25(
{ match { $r: SourceRevision { matter_number: $matter } $r hasPassage $p contains($p.text, "tariff") }
return { $r, $p } },

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B3: this population has two node columns ($r, $p), and §5 refuses that ("exactly one node column"). The rewriter rows write every bound variable, so all traversal-shaped ranked reads hit this. Also Fuse returns only [$x, s], which drops $r after rrf. Suggest: any columns, the target named by the field ($p.text), and k bounding targets, not rows.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 73f6ee9c (delta from d626e288)

This review covers only the new commit. The review of d626e288 still applies. This commit does not change B1 to B4 or the should-fix items, so they stay open.

Why. The previous revision had a batch declaration: several reads under one snapshot, with @brief and @phase. This commit removes it. The argument is that one snapshot and one round trip are properties of a request, not of the language.

What. batch_decl leaves the grammar and the T6x list. @brief and @phase leave the annotation slots and become request fields. @budget stays a per-declaration annotation. Stage N now describes the feature as a request shape. The asset example becomes query bounded() @budget(...), and a new Alternatives row and a decision-log entry give the reason.

Outcome. The direction is correct. The new text is accurate, except for one claim about what exists today. The overall recommendation is unchanged: revise before acceptance, because of B1 to B4.

Assessment

The change removes a primitive. It removes a declaration kind, two reserved annotations, one freeze-list slot and one grammar rule. batch { q1 q2 } and a file with two declarations would have compiled to the same thing, so the RFC's own axiom A6 argues against keeping both. @brief and @phase change no plan, so they belong to the request. Five more changes of the same kind would add request fields, not grammar, which costs less to keep stable.

The new text has no stale batch, @brief or @phase references. No other open RFC (#1244, #1197, #1149) depends on batch.

Should fix

S1. "A request shape the inline door already has" is too strong (line 1236 and the decision log at line 1593). Today the inline door compiles a file of several declarations, but it runs one of them:

  • QueryRequest.name selects one declaration "when query declares multiple" (crates/omnigraph-api-types/src/lib.rs:1435-1458). There is one params object, and the struct is deny_unknown_fields.
  • ReadOutput is one result with one query_name (lib.rs:651). The Arrow IPC response carries that name in a single Omnigraph-Query-Name header (lib.rs:120), so one stream holds one result.
  • A stored query is invoked by one name in the path, /graphs/{graph_id}/queries/{query_name}. "A request listing names" has no route today.

So "several declarations in one request, one result each" is new HTTP contract work. It is a smaller cost than a language construct, but stage N must decide these points:

  • how params map to each declaration (one shared namespace, or one object per declaration);
  • the response envelope, and what happens to the Arrow IPC media type;
  • whether one failing declaration fails the whole request or gives a per-result error;
  • the operation for a list of stored names, and its invoke_query gate for each name;
  • the inventory, OpenAPI and convention-test rows (docs/dev/http-api.md).

Suggested text: the request already pins one snapshot and already compiles a file of declarations, and stage N adds running all of them with one result each, additively under the HTTP conventions. W1 already uses this wording for its transport work.

S2. The freeze list does not say that the request shape is not available yet (line 975). The "Declaration kinds" row puts "a request may carry several declarations … one result each" in the Edition 3 column. The "In this release" cell does not mention it, and the Owner cell names only the one-file loader. A reader cannot tell that this is stage N work. Add "running several declarations in one request is stage N" to the release cell, and add N to the owners.

Optional

  • W2 still depends on "W1, N" (line 1365). The N dependency came from "the batch's write annotations", which this commit removes. Remove N, or say what W2 still needs from N.
  • The evidence asset 2026-10-28-gq-3-grammar-evaluation.md still recommends batch (lines 2845, 2939-2967, 3033, 3646, 3654). Its outcome paragraph already says the RFC wins where they differ. But that paragraph lists the positions brought into line, and batch is not in the list. Add it there, so a reader of line 3033 ("the three closure forms plus batch cover the measured demand") knows the demand is now met by a request shape.

Validation

  • Local runs in a clean detached worktree at 73f6ee9c:
    • python3 scripts/check-docs.py with the pinned docs requirements (Python 3.12): "Documentation OK (473 Markdown files checked)".
    • bash scripts/check-agents-md.sh: OK.
    • git diff --check origin/main...HEAD: clean.
    • typos was not run, because the binary is not on this host.
  • CI at 73f6ee9c: 28 pass, 4 skipping, none failing or pending.
  • The S1 statements come from reading the code at this head. I did not send requests to a server.
  • A regression test does not apply: the commit changes only RFC text.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 05d913e4 (delta from 73f6ee9c)

This review covers only the new commit. The reviews of d626e288 and 73f6ee9c still apply. This commit does not touch B1 to B4, and it does not address S1 or S2 of the last review, so they stay open.

Why. The previous revision reserved every future form as a "slot": it parsed, it was refused with a T6x code naming its owner, and a doc check kept a slot-owner table in sync. This commit removes that machinery. Its argument: a form that a later stage adds starts at a token that edition 3 refuses, so adding it later is additive and needs no reservation.

What. The T6x list, the slot-owner table, the slot-owner check, the graph_commits reservation, the federation and TC stage rows, the sequence picture, the rollout restatement, the coverage request setting and several verdict rows go. The freeze list gets a "Later" column. Unknown annotations become metadata, as in .pg. Only @if_commit and @valid are refused. The Lance BM25 evidence moves from section 4 to section 10, and the orthogonality and precedent tables move to the assets. Net: −74 lines.

Outcome. The direction is correct, with one exception. Removing the slot machinery removes real liability: a refusal path per future form, an owner table that can drift, and a doc-check rule. The argument "a refused token is free" holds for grammar productions. It does not hold for annotations, because this same commit makes every unknown annotation accepted. The overall recommendation is unchanged: revise before acceptance.

Blocking

B5. "Every later form is additive" is false for annotations, and the asset's own @budget example shows the break (line 882, line 943, and asset line 637).

  • Today .gq refuses an unknown query annotation. query_annotation admits only @description and @instruction (crates/omnigraph-compiler/src/query/query.pest:118). I checked this: query q() @foo("x") { … } fails omnigraph lint, and the same @foo("x") on a .pg node passes. So the line 882 rule ("retained as metadata, as .pg does today") is a change for queries, not a carry-over.
  • With that rule, query bounded() @budget(rows = 1000, on_exceed = refuse) (asset line 654) compiles at B and runs without a bound. When stage N types @budget, the same stored text starts to refuse, or with on_exceed = partial to return fewer rows. That is a change in meaning for accepted text, not an additive change. Rule 5, "Cases first", does not catch it: no refusal exists at B, so N has no expect error case to flip.
  • The reason the RFC gives for refusing @if_commit and @valid ("retaining them would mislead a writer") applies the same way to @budget and @example. It also applies to a typo such as @descripton. Today that typo is an error. Under line 882 it is silently kept.

Choose one of two options:

  • Keep today's .gq rule. An unknown query annotation is a type error. Then every annotation a later stage types starts from a refusal, and the additivity argument holds without exceptions. Metadata for tools can use a single typed annotation (for example @meta(key = …)), which A6 would accept.
  • Keep open annotations. Then refuse by name every annotation the RFC already plans (@budget, @example, @if_commit, @valid), give each one an expect error case at B, and state that a stage can type only a name that B refuses.

Should fix

  • S3. coverage = exact stands (line 1360) points to a setting this commit removes. The RFC 0048 verdict row still says it stands, but the coverage request setting paragraph is gone, and the decision log lists it as removed. Remove it from the verdict row, or say that it moves to a later stage.
  • S4. The asset now disagrees with the RFC (asset lines 637-639).
    • "Each is a parse error today": @budget(…) is accepted as metadata under line 882 (see B5). $t: @Type is a type error per the freeze-list row "an unknown @ name is a type error", not a parse error. Only crm::Person is a parse error.
    • "@if_commit alone is refused by name": line 882 and the Annotations row refuse both @if_commit and @valid.
  • S1 and S2 from the last review are still open. The decision log (line 1515) still says the multi-declaration request is a shape "the inline door already has". The Declaration kinds row (line 949) now has a "Later" column, but it names only the one-file loader. Running several declarations in one request is stage N work, and the row should say so.

Optional

  • The BM25 guards are not in the repository (line 1310). native_fuzzy_bm25_rewards_a_rare_expansion and native_bm25_idf_can_round_a_common_term_to_zero are not on main or at this head (git grep finds only this RFC). They are at 1e0bed40, the head of #751, which was closed so that each guard "returns in the implementation PR that first depends on it". Section 10 is headed "Existing owners", so the bullet reads as present evidence. Say that the guards are at 1e0bed40, and name the stage that restores them (B's exact lexical scan, or L).
  • The refusal of @if_commit and @valid has no code. The removed T6x bullet gave reserved constructs a code. The new bullet (line 882) gives none. codes.rs freezes a published meaning, so name the code in the error list.
  • @valid has two descriptions. The Schema roles row (line 965) says @valid is "accepted and refused at use" in .pg. The Annotations row (line 961) says it is "refused". State that the refusal at line 882 is for query declarations, and that the schema role behaves as row 965 says.
  • W2 still depends on "W1, N" (line 1343). The last review raised this.

Liability

The commit removes more than it adds. The slot table and its doc check were a second source of truth beside the freeze list, and that kind of state drifts. The replacement rule ("a refused token is free") needs no maintenance, but only if edition 3 refuses everything that a later stage will define. B5 is the one place where the commit adds liability: every unknown annotation becomes accepted surface. That surface either constrains each later stage or changes meaning when the stage lands. Five more annotation-typed stages would each meet this problem.

Validation

  • Local runs in a clean detached worktree at 05d913e4:
    • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt (Python 3.12): "Documentation OK (473 Markdown files checked)".
    • bash scripts/check-agents-md.sh: OK (67 links, 63 docs).
    • git diff --check origin/main...HEAD: clean.
    • typos was not run, because the binary is not on this host.
  • Annotation behaviour: omnigraph lint (CLI 0.14.1-edge+269299b) on a one-node schema with @foo("x") and @budget(5) passes. A query with @foo("x") fails. The grammar at this head (query.pest:118) matches this result.
  • git grep for the two guard names at this head and on main finds only the RFC. gh pr view 751 shows the PR closed and not merged.
  • CI at 05d913e4: 28 pass, 4 skipping, none failing or pending.
  • A regression test does not apply: the commit changes only RFC text.

Comment thread docs/rfcs/2026-11-07-gq-3.md Outdated
order to schema order, an equality between a bound variable's property
and a value written as a filter into that binding's braces, a `search` or
`match_text` or `fuzzy` call to `match_terms` once that lands).
- An unknown annotation is retained as metadata, as `.pg` does today; a

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B5: today .gq refuses an unknown query annotation (query.pest:118 admits only @description and @instruction). With this rule, @budget(rows = 1000, on_exceed = refuse) compiles at B and runs without a bound, and then starts to refuse when stage N types it. That is a change in meaning, not an additive change. Keep unknown query annotations a type error, or refuse every planned name (@budget, @example, @if_commit, @valid) at B.

```

**Added by later stages.** Edition 2: **Not supported** and not parseable.
Each is a parse error today and additive when its stage lands; `@if_commit`

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S4: under RFC line 882, @budget(…) below is kept as metadata, not a parse error. $t: @Type is a type error per the freeze list. Also, the RFC refuses both @if_commit and @valid, not @if_commit alone.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of ab74f629 (delta from 05d913e4)

This review covers only the new commit. The reviews of d626e288, 73f6ee9c and 05d913e4 still apply. This commit does not touch B1 to B5 or S1 to S4, so they stay open.

Why. The previous text refused recursion (A1) and named an open upper bound on Expand as the one later change for unbounded reachability. After a comparison with CozoDB, this commit admits recursion.

What. Fix becomes the thirteenth core operator: the least fixpoint of a declared read that calls itself. The read may not negate or count its own rows. It may aggregate them only with min or max. A computed column needs min or max and @budget(rounds = n). A1 is restated. A call cycle is no longer always T70: a monotone cycle compiles to Fix. A new stage, Rec, ships the evaluator. Until then a self-call is T70. Graph algorithms join stage R as relation functions. Two examples (reachable, distance), test rows, three Alternatives rows and open question 11 are added. Net: +95 lines (136 added, 41 removed).

Outcome. The direction is sound. One operator replaces a special case, and the finite active domain bounds it. But the admission rule is not correct yet. It is a list of refused spellings (not, count, other aggregates), not a property of the operators. So it accepts some bodies whose fixpoint is not defined, and it refuses one monotone form. The overall recommendation stays the same: revise before acceptance.

Blocking

B6. The monotonicity rule accepts non-monotone bodies and refuses a monotone one (lines 231, 496-500, 583, 884).

  • Cut (order … limit), ApproxTopK (nearest(…, k:)) and Fuse (rrf) are not not, count or an aggregate, so the rule accepts them over the recursive relation. None of them is monotone: more input rows can remove an output row.
  • Example. Take reachable (line 505) and add order { $r.handle } limit 1 to the step arm. Use the graph a→b, a→c, b→g, c→e, e→f.
    • Naive evaluation (the step reads the whole relation each round) gives {b, c, e}.
    • Semi-naive evaluation (the cost shape at line 231: the step reads only the rows the last round added) gives {b, c, e, f}.
    • Line 231 describes both, but they give different answers. So "the least fixpoint" has no single meaning for this body.
  • In the other direction, exists { p } desugars to count { p } > 0 (line 298). If the rule applies to the core term, "may not count its own rows" refuses exists { from reachable($from) … }. That form is monotone.
  • With min or max, the rule does not say what the body may do with the aggregated column. Cozo's aggregation chapter says that recursive semi-lattice aggregation has "soundness constraints on what can be done on the bindings coming from the auto-recursive parts", and that Cozo cannot check them. The RFC needs to state these constraints or refuse the uses it does not check.

Fix: define monotonicity once, on the core operators. Add one column to the operator table (line 217) that says which inputs each operator is monotone in:

  • Monotone: Scan, Filter over reference or stored columns, Project, Union, Expand, Correlate with a monotone inner term, and Group with no aggregates.
  • Not monotone: Cut, ApproxTopK, Fuse, Filter(¬ …), and Group with count, sum or avg.

A cycle compiles to Fix only when every operator between the recursive reference and the head is monotone in that input. Otherwise T70 names the operator. The type checker then owns one rule. Each later operator (rerank, norm and the stage R algorithms) fills in one cell and does not need its own exception. Add T70 cases for limit, nearest and rrf in a step arm, and an accepted exists case, to the list at line 1293.

Should fix

  • S5. Fix is not specified as a set operator, but Project and Union keep duplicates (lines 222, 225, 231). "Until none is new" has a meaning only for sets. If reachable returns { $q } and not distinct { $q }, two readings are possible. In the first, each round adds the same rows again, and the read does not end on a cycle. In the second, the head is silently deduplicated, which contradicts the Project row. State that the recursive relation is a set: either Fix removes duplicates, or the head must be distinct or a keyed min/max group, and T70 otherwise. The pinned DataFusion 54.0.0 RecursiveQueryExec has exactly this switch (is_distinct). Its doc comment says that it applies no check for infinite recursion and leaves that to the planner (recursive_query.rs lines 48-62).

  • S6. The RFC does not say which arguments a recursive call may pass (lines 256-262, 482, 496-502). Call is inlined at compile time with its scalar arguments substituted. Both examples pass $from unchanged. But a relation expression may read any variable in scope (line 482), so a step can write from reachable($r.handle). That is one relation for each argument value, not one f: R → R, and compile-time inlining does not end. Do one of the two:

    • Require each call on a cycle to pass the cycle's parameters unchanged, and refuse other calls with T70.
    • Define the parameterized fixpoint: the arguments become columns, which are finite because they come from stored values. Then give its bound.
  • S7. reachable is not the unbounded case of Expand (lines 525, 1276, 1323). Expand uses the shortest distance from the start node, so the start node comes back only through its own self-loop, never through a cycle (docs/user/queries/traversal.md lines 39-42). The Fix step from reachable($from) $q knows $r is a plain triple. I tested this with the installed CLI on a→b, b→a, b→c:

    • $p knows{1,5} $f from a returns {b, c}.
    • The two-triple chain $p knows $x $x knows $f returns {a, c}.

    So reachable("a") returns {a, b, c}, and knows{1,n} returns {b, c}. The test at line 1323 compares the two on a cyclic graph and will fail. Line 1276 calls one a special case of the other. Say where they differ (the start node), and make the test compare them with that difference. Another option is to exclude the start node in the example.

  • S8. Two statements are now stale or not correct.

    • The Expand row (line 219) still says "an open upper bound is the one later change this operator admits". Line 1276 and the decision log say that Fix replaces it.
    • The envelope (lines 369-374) says that the degree of the polynomial is "the nesting depth of Correlate, Expand and Fix". A Fix bound depends on the width of the recursive relation, not on nesting. A two-column transitive closure (all pairs) at nesting depth 1 has up to |Person|² rows. In the Fix row (line 231), "≤ |domain| of its reference columns" must be the product of the column domains (with min or max, the number of keys). "Or the budget" bounds rounds, not rows. Suggested text: rows ≤ the product of the key-column domains; rounds ≤ rows, or the budget.
  • S9. The freeze list has no row for @budget(rounds = n), and two places decide its stage (lines 1011, 1411, 1546). The Annotations row (line 1011) lists only @budget(rows, ms, on_exceed). The Rec row says "N for @budget". Open question 11 asks whether Rec or N types it. Let one place decide. This also adds to B5: at B, line 932 keeps @budget(rounds = 50) as metadata on any read. When a later stage types rounds, the same accepted text changes meaning.

Optional

  • "The rules are Datalog's" (line 496). Stratified Datalog does not allow aggregation through recursion. min and max in recursion is the semi-lattice extension (Cozo's term). Say so, so that a reader does not look for it in Datalog.
  • Line 356 says that Cozo "warns that it can be expensive". Cozo's queries chapter says that such queries "may produce infinite relations" and that it cannot reject all of them. Line 1275 already says this correctly. Use the same words.
  • Cost shape at line 231. With min or max, the delta must hold the rows whose aggregate improved, not only the rows that were added.
  • The #1244 verdict (lines 1240 and 1435) says that #1244 converges with the core. #1244 says "Direct and indirect recursion refuse during compilation". After this commit, recursion is a new difference. Add it to the Alternatives row.
  • The PR description still says "twelve operators".

Liability

The commit adds one operator, one evaluator stage, a monotonicity check and a round budget. It removes the planned open-bound Expand, so shortest distance and reachability get one owner. That is fewer primitives, and the choice is correct. The liability is in the admission rule. With a list of refused spellings, every new operator brings a new question: is it safe inside a recursion? Stage R alone adds rerank, norm and three graph algorithms, so five more operators give five more exceptions to remember. A monotonicity column on the operator table makes each operator answer that question once, in the one table that already owns signatures and bounds. If stage Rec builds on DataFusion, RecursiveQueryExec with is_distinct already gives set semi-naive evaluation for recursion over references. It has no lattice aggregation, so distance needs its own owner. Name that in the Rec row, so that the design keeps one evaluator and does not get two.

Validation

  • Local runs in a clean detached worktree at ab74f629:
    • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt (Python 3.12): "Documentation OK (473 Markdown files checked)".
    • bash scripts/check-agents-md.sh: OK (67 links, 63 docs).
    • git diff --check 05d913e4 ab74f629: clean.
    • typos was not run, because the binary is not on this host.
  • Expand start-node behaviour: omnigraph CLI 0.14.1-edge+269299b, local graph node Person { handle: String @key }, edge Knows: Person -> Person, edges a→b, b→a, b→c. knows{1,5} from a: {b, c}. knows{2,2}: {c}. Two triples: {a, c}.
  • B6 example: no Fix evaluator exists yet (stage Rec). I computed the trace with a short script that runs the step arm both ways (whole relation each round, and last-round rows only) over the graph above. This is a hand model of line 231, not engine output.
  • Read: Cozo's queries chapter (Recursion, Aggregation) and aggregations chapter (semi-lattice soundness); DataFusion datafusion-physical-plan 54.0.0 recursive_query.rs (the version in Cargo.lock); the recursion section of #1244.
  • CI at ab74f629: 28 pass, 4 skipping, none failing or pending.
  • A regression test does not apply: the commit changes only RFC text.

Comment thread docs/rfcs/2026-11-07-gq-3.md Outdated
| **Score**(R, $x, kind, field, q) | `R` → `R + [s: F32]` | an exact score per row: `l2` is the squared L2 distance of `field` to `q`; `bm25` is the BM25 of `field` against the indexed table's statistics, the population only filters; a null field value drops the row *[as today]*; a non-finite input is a typed error | `≤ \|R\|` | reads every row's field |
| **ApproxTopK**(R, $x, kind, field, q, k, recall) | `R` → `R + [s: F32]`, at most `k` rows | the best `k` of the population under `kind`, approximate with declared `recall`, reported exhaustive when the engine searched everything (A8); the overfetch ladder and probe policy are its plan policy *[as today]* | `≤ k` | index probes, widened by the ladder |
| **Fuse**(R1…Rn, method, params) | ordered scored relations sharing one node column → `R[$x, s: F64]` | `rrf`: `Σ 1 / (k + rank_i)` over the arms in which `$x` appears; a missing arm contributes nothing, not zero; joined by node identity | `≤ Σ \|Ri\|` | linear in the arms |
| **Fix**(f) | `f: R → R` monotone → `R` | the least fixpoint of a declared read over itself: rows added round by round until none is new; `f` may not negate or count its own rows, and may aggregate them only with `min` and `max`; a column it computes rather than reads is admitted only under one of those and a declared round budget | `≤ \|domain\|` of its reference columns, or the budget | semi-naive: each round joins only the rows the last round added |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B6. This rule names refused spellings (not, count, other aggregates), not monotone operators. It accepts Cut, ApproxTopK and Fuse over the recursive relation. None of them is monotone. With order { $r.handle } limit 1 on reachable's step arm over a→b, a→c, b→g, c→e, e→f, naive evaluation gives {b, c, e} and the semi-naive cost shape in this row gives {b, c, e, f}. It also refuses exists, which desugars to count { p } > 0 (line 298) and is monotone. Suggest a per-operator monotonicity column in this table, so that T70 names the first non-monotone operator on the cycle.

Comment thread docs/rfcs/2026-11-07-gq-3.md Outdated

| Operator | Signature | Semantics | Cardinality | Cost shape |
|---|---|---|---|---|
| **Scan**(T) | → `R[$x: N<T>]` | every node of type T in the snapshot | `\|T\|` | linear in `\|T\|`, or index-bounded by a pushed Filter |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S8. This row still says "an open upper bound is the one later change this operator admits". Line 1276 and the decision log now replace the open bound with Fix. Also, reachable and knows{1,n} differ at the start node on a cycle (see S7), so the replacement is not exact.

…ix over sets, bound Call, named ranked columns, write groups, the certified-margin fetch, the parser and red-team decisions

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of d79b6ea7 (delta from ab74f629)

This review covers only the two new commits (f7a5ca07, d79b6ea7). The earlier reviews of d626e288, 73f6ee9c, 05d913e4 and ab74f629 still apply where the status list below says "open".

Why. Five spikes (parser, rewriter, a reference interpreter of the core, Lance lexical guards, a red-team corpus) tested the RFC. This commit folds their findings back into the text.

What. The core gets a total tie rule for Order, an edge selector on Expand, an edge Scan ($k: E), the not-distinct equality, Fix over sets with a fuller monotonicity rule, and Call as binding with pass-through recursion. l2 becomes distance, which reads the field's declared distance. A ranking names its column ($d.embedding). rrf returns only the target and score. A write becomes a sequence of groups, and a later group sees the earlier groups' effects. Unknown annotations are refused except four metadata names. The WAND fast path becomes a certified candidate generator with an F64 rescore and a margin ε. The grammar gets the full edition-3 rule set and the prototype's decisions. Net: +302 lines in the RFC (628 added, 326 removed).

Outcome. Most of the earlier blocking findings are now fixed, and the spike evidence makes the text much more concrete. Two new statements are not correct, and the engine shows it: the write-conflict rule contradicts the group rule, and limit with no order is marked [as today] but is not what the engine does today. The recommendation stays: revise before acceptance.

Blocking

B7. The new conflict rule refuses the sequential writes that the group rule keeps (lines 267-270, 610-612, 1177).

  • Line 267: "Two different values for one entity and property … or two different upserts of one key within one write are a typed refusal."
  • Line 610: "a later group sees the effects of the groups before it [as today], so a body that inserts a document and then updates it by key still works". That body writes two different values for one entity and property in one write, so line 267 refuses it.
  • The rewriter writes "one group per statement, in order" (line 1177). So every edition-2 body that writes one property twice becomes a refusal after the rewrite. The Compatibility section (line 1749) does not list this break, and the "rewritten corpus green" gate in blocked_on would fail on any such case.
  • Today all three of these bodies succeed with one commit, and the last write wins (installed CLI, local graph):
    • insert Person { handle: "d", name: "first" } update Person set { name: "second" } where handle = "d" gives second.
    • insert Person { handle: "e", name: "x" } insert Person { handle: "e", name: "y" } gives y.
    • two update Person set { name: … } where handle = "a" statements give the second value.

Fix: scope the conflict rule to one group, which is the unit that shares a snapshot. Inside a group, two different effects on one entity, property or key have no order, so a refusal is correct. Between groups, a later group overwrites as today. Also remove "an update and a delete of one entity" from the list. The per-write D₂ split (line 271) already makes that case impossible, so the clause is dead text.

B8. limit with no order is not deterministic today, and the new rule changes its rows and its cost (lines 236, 242).

  • The Cut row says that over an unordered input it "orders by the tie rule alone first [as today]". Line 242 says that limit without order "therefore stays legal and deterministic".
  • Today the engine does not order there. On a local graph with Persons a to e, match { $p: Person } return { $p.handle } limit 2 returns {b, c}. explain shows LimitExec over ScanExec with ordering: null. The shipped skill says the same (skills/omnigraph/SKILL.md line 95): "Without order, limit n may return any n valid rows (the subset can change between releases)".
  • Under the new rule the same read returns {a, b}. That is a legal answer under today's contract, so it is not a break for users. But the [as today] tag is false, the rewriter's typed-IR equality cannot hold for this shape (edition 2 lowers to a bare limit, edition 3 to an order and a cut), and the cost changes.
  • The cost column of the Cut row says k. With an order by identity first, the cut must read the whole input: |R| log k, not k. For a paging-style limit 50 over a large type, that is a full scan in place of an early stop. A1 makes the cost column part of the contract, so it must be correct.

Fix: mark the rule [choice] and give the reason (the interpreter's 138 of 12,799 result). State the cost of a Cut over an unordered input as |R| log k. Name it in the rewriter section as a shape whose rows can change, with the rows-equality exception that the rrf row already has. If an early stop is still wanted for an unordered limit, that is a different operator with a declared nondeterminism, and A6 and A8 would need to say so. Do not keep both meanings under one Cut.

Should fix

  • S10. The new second bound on Expand is false when $k binds (line 229). The row adds "≤ |R| · |T′|". With $k bound, one row comes out for each edge, and parallel edges stay distinct. Test: five unkeyed Knows edges a→b, with three Person nodes. $p $k:knows $f from a returns 5 rows, more than |R| · |T′| = 3. $p knows $f returns 1 row. Apply the |T′| bound only when $k is not bound. With $k, the bound is ≤ |R| · |E<e>|, or the fan-out bound alone.
  • S11. The fuzzy budget claim does not match Lance 13 (line 834). The text says the expansion "keeps the fifty lexicographically smallest expansions per term". In lance-index 13.0.0 the budget is one budget for the whole query, used in query-position order (src/scalar/inverted.rs lines 127-166; the default of 50 is at src/scalar/inverted/query.rs line 83). Lance's own test says "max_expansions should cap the whole fuzzy query, not each token" (src/scalar/inverted/index/tests/query.rs line 475). So a multi-term fuzzy query can give a later term no expansion at all, not even its exact form. That is a wider break than the text says. Correct the sentence. Add a multi-term case to the gq3_native_fuzzy_expansion_truncates_membership_silently guard. Also check that stage L's "complete fuzzy expansion through with_max_expansions within the automaton's limit" sizes the budget for all terms together.
  • S12. Two new type and name changes are not in the rewriter's fixed map (lines 237, 1179, 1187).
    • bm25 becomes F64. Today every ranking score except rrf is F32 (crates/omnigraph-compiler/src/query/typecheck.rs lines 416-423). Line 1187 says that a result column "keeps its name and type", so the rewrite of bm25(…) as s changes a column type that the text says it keeps.
    • The fixed map has one rename, d._distance to _distance. Today an unaliased bm25(…) is d._score and an unaliased rrf(…) is d._rrf (docs/user/search/index.md lines 131-135). Edition 3 names them score.
    • List the F32 to F64 change and both renames in the fixed map and in Compatibility. Or keep bm25 at F32 in edition 3 and move the F64 score to stage L, where the rescore needs it.
  • S13. The spike evidence is not pinned, and two new rules depend on code that is not in this repository (lines 52, 1342, 1528, 1859-1861). The RFC names branches (spike/gq3-core and the others "on the fork"). The reports under docs/spikes/ are on those branches, not in this PR. Branch names move, so the counts (2,801 recursions, 3,231 declarations, 395 red-team cases) cannot be checked against the exact code that produced them. Two statements give that code a lasting role: the interpreter is "kept beside the RFC … and a change to the core changes it first" (line 1528), and "the parser spike's corpus walk is the regression owner" (line 1342). Nothing in docs/dev/testing.md owns either, and CI does not run them. Link each spike at its commit (ragnorc/omnigraph at c5ba069a1b core, 0162a33005 parser, f7b4d934a5 rewriter, 12e64ab51b lexical, f882e85bf3 red team). Then either move the reports into assets/ and give the interpreter an owner when stage B lands, or call them evidence only. Otherwise the RFC has a second executable copy of section 2 that can drift (invariant 12).
  • S14. Stale or incomplete text after the fold.
    • Line 1681, the RFC 0048 verdict row: "exact: true is l2" must be distance. This is the last stale use of the old name. The same row still says "coverage = exact stands" (S3, open).
    • Line 405, the envelope: "degree is the nesting depth of Correlate, Expand and Fix". The Fix row (line 240) now bounds rows by the key-column domains, so the degree of a Fix bound is the number of key columns, not the nesting depth (second half of S8, open). Also write "the product of the key-column domains" in the Fix row: "|domain| of its key columns" can be read as one domain.
    • Line 1660, the Rec row still says "B; N for @budget". Line 1150 now types @budget(plan = n, rounds = n) at B, so Rec does not wait for N. Also say what B does with @budget(rows = …), which the asset uses at line 677. If B ignores an argument that it does not know, B5 comes back at the argument level. If B refuses it with stage N named, say so in the Annotations row.
    • The Unresolved list is numbered 1, 11, 2, 4, 3, 4, 5, …. Markdown renumbers it, but the source is hard to cite. Item 11 is not an open question of this RFC. It is a defect in the shipped engine: fuzzy($d.title, $q, 1) can miss the document that holds the exact term. No issue exists for it (#748 is about the analyzer, not the budget). File it now, with S11's multi-term case, and link it from the RFC.

Optional

  • The monotonicity rule in the Fix row (line 240) is now complete for the thirteen operators. It is still a list of surface forms. Stage R adds rerank(model, text, k), a top-k, and the list does not name it, so as written a rerank over the recursive rows would pass. Add one sentence to the stage R row: each new relation function declares whether it is monotone in its input, and T70 is the default.
  • "no not over its own rows" (line 240): say "no not whose pattern reads the recursive relation". A not { $r blocked $_ } that is correlated on a recursive row but reads only stored edges is stratified and monotone. A reader can take the current words to refuse it.
  • The #1244 row (line 1438) still does not list recursion as a difference. #1244 refuses recursion at compile time, and this RFC now admits it.
  • S1 and S2 are still open (the decision log at line 1845, "inline door already has", and the Declaration kinds row at line 1138, which does not name stage N).

Status of earlier findings

  • Fixed: B1 (the examples are split and D₂ stays per write), B2 (groups with sequential visibility), B3 (a ranking names its column, and the rest rides as payload), B4 (A6 now has the formatter's complete list), B5 (unknown annotations are refused; this also matches today's .gq grammar, which admits only @description and @instruction, query.pest lines 118-120), B6 (sets, the full rule, exists admitted, the 74-of-817 evidence), S5, S6, S7, and the Fix row of S8.
  • Partly fixed: S8 (the envelope) and S9 (the Rec row). See S14.
  • Open: S1, S2, S3 (see S14), and S4: asset lines 661-662 still say "Each is a parse error today" and "@if_commit alone is refused by name". At this head @budget is typed at B, $t: @Type is a type error, and @valid is refused too.

Liability

The fold removes more than it adds. One Expand with a selector replaces two Union desugarings. distance gives one exact spelling on every vector field. The group rule keeps edition 2's sequential writes, so the break does not get an eighth spelling. The red-team rule that every fix is a byte-range suggestion gives the refusal texts one owner and a test. The new mechanism with lasting cost is the certified-margin fetch. It needs an ε table per term count, an F64 rescorer outside Lance and five plan-eligibility conditions (no deletion files, no row policy, one table, prefilters only, full coverage). All five are conditions on choosing a fast path, with the exact scan as the fallback, so invariant 7 holds. Write them as one eligibility predicate in RFC 0047's validator, not as checks spread across the scan code. The two risks for later changes are B7 and S13. If the conflict rule and the group rule stay two separate statements, each later write feature (returning, assert) must decide which one applies. If the interpreter stays outside the repository, each core change must update a second copy that CI does not check.

Validation

  • Clean detached worktree at d79b6ea7:
    • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt (Python 3.12.16): "Documentation OK (473 Markdown files checked)".
    • bash scripts/check-agents-md.sh: OK (67 links, 63 docs).
    • git diff --check ab74f629 d79b6ea7: clean.
    • typos was not run, because the binary is not on this host.
  • Engine behaviour, omnigraph CLI 0.14.1-edge+269299b, local graph node Person { handle: String @key name: String age: I32? }, edge Knows: Person -> Person:
    • S10: five unkeyed a→b edges, three Persons. $p $k:knows $f from a: 5 rows. $p knows $f: 1 row.
    • B7: the three bodies above each published one commit. Final values second, y, u2.
    • B8: limit 2 with no order returned {b, c}. explain: Page / LimitExec over ScanExec, ordering: null.
    • Null order: asc puts nulls first and desc puts them last, which agrees with the RFC's fixed value order and docs/user/queries/index.md line 233.
    • .gq annotations: @foo and @owner(…) are refused by lint (Q000).
  • Code read: lance-index 13.0.0 src/scalar/inverted.rs (expansion budget), src/scalar/inverted/query.rs (default 50), src/scalar/inverted/index/tests/query.rs (whole-query cap); omnigraph-compiler typecheck.rs score_type; query.pest annotation rules; docs/user/search/index.md; skills/omnigraph/SKILL.md.
  • Not checked: the spike counts, which I did not rerun (I only confirmed that the five branches and their heads exist); the ε value; the WAND and NFC claims of section 4.
  • CI at d79b6ea7: 28 pass, 4 skipping, none failing or pending.
  • A regression test does not apply: the commits change only RFC text.

…atements, recursive types, the rrf population rewrite, the grammar decisions and the restated gate

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of d0155e50 (delta from d79b6ea7)

This review covers only the new commit d0155e50. The earlier reviews of d626e288, 73f6ee9c, 05d913e4, ab74f629 and d79b6ea7 still apply where the status list below says "open".

Why. Four more spikes (type checker, executor, second grammar round, competence instrument) tested the RFC. This commit folds their results back into the text.

What.

  • Writes: the statements of a group now run in order, and each keyed effect sees the statements before it. The conflict refusal now covers only the rows of one statement (a run-time E error). Across statements, the later effect wins. Whole-write validation runs once, at publication.
  • Core: three leaves (Unit, Env, RecRef). A block correlates when it names an outer variable again, and the inline-read T68 no longer applies to blocks. A recursive read types to Fix in release B, and the planner refuses it with P006. The monotonicity rule is stated on the term.
  • Surface: a triple with no bound end is now legal. It starts from a Scan of the edge's source type. A filter moves to just after the binder of its last variable. Arithmetic gets rules for literal types, mixed signedness, % and Date.
  • Diagnostics: codes T71 to T80, P006 and an E namespace.
  • Rewriter: the normalisation N1 to N7. The rrf row now emits a "type-population" form, because the population form changes rows on 3 of 21 cases.
  • Gate: the competence gate now uses the full front end and compares correctness with edition 2.
  • Asset: six examples corrected.
  • Net: +321 and −153 lines in three files.

Outcome. B7 is fixed, and the write rule now matches what the engine does. Most of the new text is careful, and I confirmed six of its [as today] claims on the engine. Two new problems need a fix before acceptance. (1) The RFC no longer states today's rule that a block must read an outer variable. With the new unanchored triple, a text that is refused today gets a new meaning with no warning. (2) The acceptance gate and the rewriter tally do not count the three corpus cases that the new grouped-order rule refuses. B8 is still open. The recommendation stays: revise before acceptance.

Blocking

B9. A block that reads no outer variable is no longer refused, and the new triple rule makes this a silent change (lines 333, 337-339, 562-567, 1859-1862).

  • Today a block must read an outer-bound variable. not { $q: Person { name: "zed" } } is T9 ("a negation block is uncorrelated"), and an exists block is T39 (crates/omnigraph-compiler/src/query/codes.rs lines 35 and 57). The RFC does not state this rule. The derived-forms table (lines 337-339) lowers every block to Correlate, which accepts an inner term that does not read Env.
  • This commit also makes two other rules explicit: a block sees only the text before it (lines 562-566), and a triple with no bound end is legal (line 333).
  • Together they give a new meaning to a text that is refused today. match { not { $p knows $_ } $p: Person } return { $p.handle } is T5 today. Under the new rules, $p in the block is a fresh name, the triple starts from a Scan of Person, and the block becomes a constant test: it removes every row if any person knows anyone. The writer wanted "persons who know nobody". The formatter keeps the order of blocks (lines 566-567), so it does not repair this.
  • Compatibility (lines 1859-1862) lists the texts that edition 3 now accepts. This one is not in the list.

Fix: keep today's rule as a typing rule: a block must read a variable bound before it. Keep T9 and T39, and print the fix "move the block after the binder of $p". State it once, in the Correlate record or in the scope section. If dropping the rule is a decision, list it in Compatibility and add a lint warning for a block that names a variable bound after it.

B10. The gate and the rewriter tally do not count the three cases that the new grouped-order rule refuses (lines 15, 269, 1254-1258, 1856-1859).

  • Compatibility now says that three texts edition 2 accepts are refused: "an order key over grouped rows that is a meta-field neither returned nor a key (3 stored cases)". In the corpus I find these three: issue_955_aggregate_comparison.gqt (aggregate_metadata_name_still_orders, constant_edge_type_still_orders_groups) and meta_fields_resolve_by_role.gqt (aggregate_identity_order). All three expect unordered rows.
  • The first blocked_on item requires "no hand edit to query text beyond the seventeen fusion cases" and "the rewritten corpus green". The rewriter table has no row for this shape, so the gate cannot pass as written.
  • The tally at lines 1254-1258 is not updated for this fold. It counts "28 multi-statement bodies that the group rule rewrites" among the 50 texts that "carry a refusal", but the new group rule keeps those bodies (line 269: 52 reproduce). It does not count the three grouped-order cases.

Fix: add a rewriter row. All three cases expect unordered rows, so the rewriter can drop an order key over groups that reads neither a key nor an aggregate; the rows stay equal as a bag. Where a limit follows, refuse with the fix printed, because the rows can change. Or count the three as hand edits in blocked_on and in Compatibility. Then redo the tally under the group rule.

B8 (open, not changed). The Cut row (line 241) still says that an unordered input is ordered by the tie rule [as today], with cost k. Line 247 still says that limit without order "stays legal and deterministic". The engine does not do this today (see the previous review for the evidence).

Should fix

S15. The scope of a write group is not stated (lines 266-272, 636-641, 1236).

  • "A later group sees the groups before it" can mean their effects or their variables. The text does not say whether a second group can read $p from the first group, or bind $p again. The rewriter turns two edition-2 statements into two groups, each with "a name the declaration does not use" (line 1236), so it needs this rule too.
  • An assignment reads the group's rows, which are evaluated once before the statements run. So once arithmetic exists, the group boundary changes the result. $p: Person { handle: "a" } update $p set { age: $p.age + 1 } update $p set { age: $p.age + 1 } is one group and adds 1. The same two statements, each after its own binding, are two groups and add 2.

Fix: state that a group's variables are local to the group, that a later group binds again (under any name), and that $x.prop in an assignment reads the group's rows, not the effects of earlier statements in the same group. Add the example above to the Writes tests.

S16. A7 disagrees with the new write rule and with the wrapper refusal (lines 159, 167, 186-190, 270, 641, 1068, 1204, 1407-1409).

  • A7 says that a write "evaluates them and its effects on one snapshot (A2)". With several groups, and with statements that see earlier keyed effects, a write reads several states. A2 (line 159) is about reads.
  • A7 says that a body opening with match { } "is a read when return follows and a write when a statement does". Lines 1068 and 1204 make that body a parse error whose fix removes the wrapper, and lines 1407-1409 say that the kind is decided there only to print the fix.
  • "publishes once (A4)" appears three times (lines 188, 270, 641). A4 is Closure (line 167). No axiom says that a write publishes once.

Fix: in A7, say that each group's clauses read the state that the earlier groups left, and that the write publishes once. Say that a statement after match { } is edition 2's wrapper, refused with the fix. Cite the correct axiom for publish-once, or put that rule into A7.

S17. The IR-equality claim for rrf does not fit the form the rewriter emits (lines 15, 1243, 1265-1281).

  • The rewriter now emits the type-population form for rrf: let $all = { match { $d: T } return { $d } }, then the fused rows, then "the rest of P, correlated on $d". N1 ("the read's only match clause is the from of a ranking") and N2 ("the population is the whole match") do not hold for that form.
  • N2 maps a population to the ranked scan with the pattern's filters as a prefilter. For rrf, line 1243 says that edition 2 windows the whole type and filters afterwards. So N2 can make two terms with different rows IR-equal. The numbers show it: the population form is rows-equal on only 18 of 21 cases.
  • So "27 of 28 rrf rewrites are IR-equal" (line 1275) is either about the form that is not emitted, or it is not evidence that the rows are equal.

Fix: limit N2 to nearest and bm25. In blocked_on, write "typed-IR equality on every declaration except the rrf row", as the previous version did, and keep the rows-equal check for rrf. Remove the 27-of-28 count, or say which form it measured.

S18. The "in full" grammar uses rules that it does not define (lines 1325-1375).

  • The block uses 17 rule names that it does not define and that query.pest and schema.pest on main do not define either: agg_name, annotation_value, call_name, edge_sel3, kw_exists, kw_false, kw_let, kw_match, kw_optional, kw_order, kw_query, kw_return, kw_true, or_expr3, type_ref_or_kind, undirected_edge3 and where_expr.
  • where_expr is new in this commit. It carries one of this fold's decisions (it "admits bare names", lines 1099-1102 and 1373), so its definition is part of what the rule-set pin must pin.

Fix: define where_expr and the missing keywords, or say which rules come unchanged from a named grammar file. Otherwise "In full" is not true, and the rule-set pin of section 11 has no exact set to pin.

S13 (open, now larger). This commit adds four more spike branches and gives them lasting roles: "the type checker spike (spike/gq3-typeck, 556 tests) owns these today" (lines 1597-1598); "(they are, on spike/gq3-typeck)" and "6 remain on spike/gq3-grammar2" (line 17); "the type checker produces it today and the planner answers P006" (line 18), which is true of the spike, not of main. Pin each one at its commit, ragnorc/omnigraph 2f393bb0ca (gq3-typeck), 0032c2b808 (gq3-exec), 9fef4cf3a9 (gq3-grammar2) and 45d11951ce (gq3-instrument), together with the five from the previous review. Write "on the spike" where the text says "today".

Optional

  • L codes exist already: L201 warns on a nullable property (crates/omnigraph-compiler/src/query/lint.rs line 12, docs/user/queries/index.md line 326). Line 1120 says "The lint prefix L is added". Write "new L codes", numbered after L201.
  • T76 is a good change. Today a body that mixes delete with insert passes lint and fails at run time with an uncoded bad_request. Name this move in "What a reader sees changed". The GQT corpus has no case for the D₂ refusal today, so add one.
  • Unresolved 12 is answered by the RFC's own rule: an order over groups reads keys and aggregates only (lines 1858-1859), and return distinct is a Group with no aggregates (line 336). Only a rank key is left, and the rewriter row at line 1241 already drops it. Unresolved 13 repeats item 11 for fuzzy. Both edition-2 defects are real and still have no issue. I confirmed the second one: bm25($d.title, $s) > 0.0 in a filter passes lint and fails at run time with "unsupported expression in filter" (bad_request, no code). File both and link them.
  • The unanchored triple (line 333): say whether "the smaller-named end for <e>" means the variable name. Say what happens for an edge selection whose members have different source types, for example $a (knows | likes) $b with neither end bound. Keeping T5 there is the smallest rule.
  • The rrf rewrite keeps edition 2's "window the type, then filter" in every migrated text, and line 1243 says that this is against law 1. List those texts in the rewriter's report, so that their owners can move each one to the population form on purpose.
  • The gate compares "semantic correctness on the shared tasks" with edition 2, but the text reports edition 3 over all 45 tasks (38 to 39). Report edition 3 on the 26 shared tasks, and say which number decides when runs differ (edition 2 gave 24 and 25).

Status of earlier findings

  • Fixed: B7. The conflict rule now covers the rows of one statement, across statements the later effect wins, and the dead "an update and a delete of one entity" clause is gone.
  • Open:
    • B8 (see above).
    • S10: the Expand row (line 234) still has ≤ |R| · |T′| with no condition on $k.
    • S11: lines 866-867 still say "the fifty lexicographically smallest expansions per term".
    • S12: the fixed map still has no d._score or d._rrf rename and no F32 to F64 change for bm25.
    • S13 (see above).
    • S14: line 1783 still says "exact: true is l2"; the envelope (line 420) and the Rec row (line 1762) did not change; the Unresolved list is now numbered 1, 11, 2, 4, 3, 4, …, 10, 12, 13.
    • S1, S2, S3, and S4 (asset lines 685-686 still say "Each is a parse error today" and "@if_commit alone is refused by name").

Liability

The fold removes liability in the write path. One rule (statements run in order, the later effect wins, the conflicts of one statement's rows are refused) replaces two rules that disagreed, and it matches the engine, so the 52 stored multi-statement bodies need no new spelling. Moving the D₂ refusal to a type code (T76) removes a refusal that only run time finds. The new liability is in three places. (1) Rules that are true only on spike branches: the type checker that owns tests, the planner that answers P006, the red-team counts. Until they are pinned and moved into this repository, each core change must also update a second copy that CI does not run. (2) The rrf migration keeps, in every migrated text and with no flag, a behaviour that the RFC calls wrong. (3) The unanchored triple interacts with the scope rules (B9). Each later feature that adds a binder (a polymorphic Scan, optional in writes) must check that interaction again, unless the block rule is stated once and owned by the type checker.

Validation

  • Clean detached worktree at d0155e50:
    • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt (Python 3.12.16): "Documentation OK (473 Markdown files checked)".
    • bash scripts/check-agents-md.sh: OK (67 links, 63 docs).
    • git diff --check d79b6ea7 d0155e50: clean.
    • typos was not run, because the binary is not on this host.
  • Engine behaviour, omnigraph CLI 0.14.1-edge+269299b, local graphs (Person, Company, Knows, WorksAt; Item with an I64 key):
    • Block re-mention: $p: Person not { $p: Person { name: "bob" } } returns every person except bob. The [as today] claim holds.
    • Uncorrelated blocks (B9): not { $q: Person { name: "zed" } } is T9; exists { $q: Person { name: "bob" } } is T39; a not block before the binder of its variable is T9; not { $p knows $_ } before $p: Person is T5.
    • A triple with no bound end, $a knows $b alone, is T5. The claim holds.
    • @index(title, fulltext) and title: String @index(fulltext) are refused: "@index on Doc references unknown property 'fulltext'". The claim holds.
    • An edge insert with from: "5" into an I64 @key endpoint succeeds. The claim holds.
    • insert Tag { … } delete Tag where …: lint OK; at run time bad_request with no code.
    • bm25($d.title, $s) > 0.0 as a filter: lint OK; at run time "unsupported expression in filter" (bad_request, no code).
  • Code read: codes.rs (T5, T9, T39, T42, T58; P001 to P005; no T60 or higher on main), lint.rs (L201), the rule names in query.pest and schema.pest on main, and the three GQT cases in B10. Other open PRs: none uses T71 to T80, P006 or an E namespace (#1101 adds T60, which this RFC does not use).
  • Not checked: the spike counts (I did not rerun them; I confirmed that the four new branches exist on the fork, at the commits above), the instrument results, and ε.
  • CI at d0155e50: 28 pass, 4 skipping, none failing or pending.
  • A regression test does not apply: the commit changes only RFC text and its assets.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of a8c332ff (delta from d0155e50)

This review covers only the two new commits, 1f43be0f and a8c332ff. The review of d0155e50 and the earlier ones still apply where the status list below says "open".

Why. The type checker now exists on the front-end branch, and the competence instrument has run through it. The RFC must say how the compiler holds two editions without two pipelines, and it must record what the instrument found.

What.

  • New section "One IR" (lines 1778-1797). Both editions lower into the typed core IR, and the planner reads only the core. Edition 2 goes through the same surface tree and type checker as edition 3. Today's pipeline IR and the frozen v1 reference engine are deleted. No module name carries an edition number.
  • New blocker (line 18): one IR, with every edition-2 GQT case keeping its rows, shapes, plans and codes.
  • M0 (line 1757) and the red-team blocker (line 17): the instrument, the 30-task corpus and the 395-case red-team corpus stay outside the repository.
  • Line 16 records the full-front-end run of 2026-11-09.
  • Net: +29 and −5 lines in one file.

Outcome. The direction is good. One IR, one checker and no parallel edition modules remove a lot of liability, and deleting v1 together with the IR it depends on is a coherent choice. But there are two new blocking problems. (1) The set of places where edition 2 and edition 3 type differently is named wrong and is too small. As written, the new "keeps its codes" blocker cannot pass. (2) The run recorded on line 16 fails the gate on its own numbers, and the measurement that could pass the gate needs stage B, which the gate controls. B8, B9 and B10 are still open. My recommendation stays the same: revise before acceptance.

Blocking

B11. The edition differences in the shared checker are not the "seven freeze-list rows", and the "keeps its codes" blocker conflicts with Compatibility (lines 18, 1781-1789, 1191-1215, 1870-1882).

  • Line 1782 says that "the seven freeze-list rows" are "the only places the edition changes what the checker accepts". The freeze list (lines 1191-1215) has 17 rows, and it lists edition-3 surfaces, not differences. The seven are the rewriter's spellings (lines 68 and 1870).
  • Compatibility (lines 1870-1882) lists more differences than seven. Edition 2 accepts, and edition 3 refuses, 17 rrf shapes (identical arms, arms over two bindings) and 3 grouped-order texts. Edition 2 refuses, and edition 3 accepts, nine texts (T40, T5, T42, T9, T58).
  • The GQT corpus pins some of those edition-2 refusals today. Two examples: issue_566_order_key_binds_to_return_item.gqt boolean_key_unbound (order { $p.age > 30 }, T42, line 143), and issue_764_relationship_count_filter.gqt outer_property_as_the_bound_is_refused (count { … } >= $n.followers, T40, line 187).
  • Suppose edition 2 goes through the edition-3 checker and only seven rules are gated by edition. Then those cases are accepted under gq 2;, and the 20 edition-2 texts above are refused. Line 18 says that every edition-2 case keeps its codes. Lines 1788-1789 say that "a code that moves is one the fixed map of section 6 names". But the fixed map is the rewriter's map for edition-3 output. The two sentences do not agree, and neither one matches Compatibility.

Fix: make the edition-gated rules one table, owned by the type checker, and make that table equal to Compatibility: the seven spellings, the two rrf shapes with no rewrite, the grouped-order rule, and the five lifted refusals. Add one test per row under gq 2; and under gq 3;. Or decide that edition 2 takes edition 3's typing on some of those rows. In that case, list each GQT case and code that moves in the line-18 blocker. Do not call the table the freeze list. Each row is a branch in the checker that someone must maintain until the edition window closes. The exact count is the liability this section is meant to keep small.

B12. The 2026-11-09 run fails the gate, and the measure that would credit edition 3 needs the stage the gate controls (lines 16, 18, 1757-1758).

  • The gate: "first-try acceptance and semantic correctness on the shared tasks at or above edition 2's". The new run gives acceptance of 44 to 45 of 45, against 43 for edition 2, so acceptance passes. It gives correctness of 23 to 25 of 45 by typed-IR equality, against 28 of 45 by the same measure for edition 2. On the only correctness measure taken for both editions, edition 3 is lower.
  • Line 16 explains the gap ("a hand reading of the misses found most of them the right answer in another spelling"). But it does not say that the gate failed, and the item stays a blocker with no verdict. The hand reading is now the deciding evidence, and it is not recorded: which tasks, which verdict per task, and who read them.
  • Line 16 says that typed-IR equality "cannot credit [those spellings] until edition 3 executes". Edition 3 executes when the planner lowers the core (line 18), which is stage B work. Line 1758 says that B "ships after M0's instrument". So as written, M0 cannot reach the measure that would let it pass until B ships.

Fix: choose a measure that M0 can take before B and that credits equal meanings. Two options use what the RFC already has. (a) Put "One IR" before the M0 correctness run. When the planner reads the core, the edition-3 forms that lower into today's operators execute, so you can compare rows on the tasks that use only those forms. (b) Normalise both IRs with N1 to N7, plus a rule for each equivalent spelling the hand reading found, and report both editions on that one measure. In both cases, state this run's result (correctness by typed-IR: failed) and what changes before the next run. Publish the per-task hand verdicts beside the numbers. The seeded-repair rate is only "reported". It fell from 89-94% (16 to 17 of 18) to 79-83% (23 to 24 of 29). Say whether that drop is acceptable.

Should fix

S19. The gate evidence is outside the repository with no pinned commit, and two sections still name it as an in-tree owner (lines 16-17, 1757, 1585-1589, 1860-1861).

  • M0 says that the instrument and the 30-task corpus stay "outside the repository for now (branch gq3/phase-0-instrument on the fork)". That branch head is ragnorc/omnigraph cb0f4634 (committed 2026-11-09T09:04Z). It is the head of #1317, which was closed at 23:44Z, just before this push. A branch can move. Gate evidence needs a commit.
  • Section 10 (lines 1585-1589) still lists the 395-case red-team corpus under "Existing owners" as a compiler parser test whose "mismatch set is pinned and must shrink to zero". Invariant 13 (lines 1860-1861) says that "the instrument owns agent competence". Both now name owners that are not in the repository and that CI does not run. This is S13 again, and larger.

Fix: pin cb0f4634 (or the exact commit the 2026-11-09 run used), and pin the model and card hashes beside the numbers on line 16. In section 10, make the in-tree owner what the changelog line says: one parser or checker test for each red-team case that pins a distinct behaviour. Make the 395-case run spike evidence at a pinned commit. Change invariant 13 to match.

S20. The reason given for deleting v1 is not exact (lines 1791-1795).

  • v1 does not read today's pipeline IR. It reads a separate, frozen omnigraph_compiler::ir::untyped::QueryIR, which QueryIR::erase makes only at the test reference door (crates/omnigraph-compiler/src/ir/erase.rs, 825 lines; ir/untyped.rs; docs/dev/architecture.md line 105). So deleting the pipeline IR does not force deleting v1. The other option is a new erasure from the core IR to untyped. That is the real cost of keeping v1, and the RFC should name it.
  • The corpus has 34 --- expect same as v1 steps in 16 cases (15 under cases/, 1 under cases_slow/), not fifteen. Every one of those steps also pins its rows (4 of them pin zero rows), so "sits beside pinned rows" is true.
  • Some differential signal goes with v1. cases_slow/v2/wide_column_scan_limit_answers.gqt (line 2) records a red run where v2 hit the memory limit and v1 returned the rows.
  • "Today's typed expression nodes", with Cast (IRExpr::Cast, BlockAggregateExpr::Cast, mirrored in omnigraph-planner/src/mirror.rs), live in ir/mod.rs beside the IROp pipeline that is deleted. I confirmed the claim. But the text does not say where the boundary between "kept" and "deleted" runs.

Fix: say that v1 reads ir::untyped through erase.rs, and that the choice is between deleting v1 and writing a core-to-untyped erasure. Give the cost of each choice. Write "34 comparisons in 16 cases". List what goes with v1: the omnigraph-reference-engine crate, ir/untyped.rs, ir/erase.rs, tests/frozen.rs, the forbidden_apis.rs guard, the GQT section "Reference comparison", and the v1 text in docs/dev/testing.md, execution.md, architecture.md and .github/CODEOWNERS. Name the types that stay (IRExpr, BlockAggregateExpr, IRProjection?) and the types that go (IROp, QueryIR).

S21. The open implementation PR builds the parallel structure that this section removes.

  • #1316 (draft, head 75cb05fc, not updated since this commit) adds ir/core3.rs, query/ast3.rs, parser3.rs, typeck3.rs beside typecheck.rs, habits3.rs, print3.rs and planner edition3.rs. It also adds 397 red-team corpus files under query/testdata/gq3-redteam/.
  • The new text says that no module carries an edition number, that one checker serves both editions, and that the red-team corpus stays out of tree. #1316 does the opposite in each case.

Fix: in the stage table or in "One IR", say that #1316 converges before it merges: rename the modules, merge the two checkers, and move the corpus out of the tree, keeping one test per distinct behaviour. Otherwise the first implementation PR ships the two-pipeline structure that this RFC rules out, and a later PR must remove it.

Optional

  • Line 16 puts edition 3's typed-IR count (23 to 25) and edition 2's rows count (39 to 40) in one sentence. A small table would be clearer: one row per metric, one column per edition, with the denominator and the three runs. Edition 2's acceptance moved from 44 of 45 (2026-11-08) to 43 (2026-11-09). Say whether the cause is the front end, the card or the run-to-run spread.
  • The earlier optional point about the shared tasks is now handled: both editions are reported over the same 45 tasks.

Status of earlier findings

These commits do not touch the earlier findings. Still open: B8, B9, B10; S10, S11, S12, S14, S15, S16, S17, S18; S13 (now part of S19); and S1 to S4.

Liability

The One IR decision removes liability by design. One IR, one checker and one planner input replace two pipelines. Deleting v1 together with the IR it reads removes the frozen 6,584-line engine crate, untyped.rs (315 lines), erase.rs (825 lines) and their guards, with no second erasure left behind. Five more editions under this rule add rows to one gated table, not five module trees. The new liability is in three places. (1) Every edition-gated rule is a branch in the shared checker until the window closes, so the table must be exact (B11). (2) Without v1, the GQT corpus loses its only differential check. Pinned rows remain, but a case like the memory-limit one would no longer get a second answer. (3) The gate evidence (the instrument, the agent corpus, the red-team corpus) is outside the repository, so CI does not run it and nothing stops it from drifting from the type checker (S19).

Validation

  • Clean detached worktree at a8c332ff:
    • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt (Python 3.12.16): "Documentation OK (473 Markdown files checked)".
    • bash scripts/check-agents-md.sh: OK (67 links, 63 docs).
    • git diff --check d0155e50 a8c332ff: clean.
    • typos was not run, because the binary is not on this host.
  • Code read at a8c332ff: ir/erase.rs (QueryIR::erase → untyped::QueryIR), ir/untyped.rs, ir/mod.rs (IRExpr::Cast, BlockAggregateExpr::Cast), omnigraph-planner/src/mirror.rs (the Cast mirror), docs/dev/architecture.md line 105, and the GQT README section "Reference comparison".
  • Corpus counts, from a script over crates/omnigraph-gqt/cases*: 34 exact --- expect same as v1 lines in 16 files. Every one of those steps has a rows expectation (4 are empty). The edition-2 error expectations include T5 ×4, T40 ×2, T42 ×5 and T9 ×1. I read the T42 and T40 cases cited in B11.
  • Other PRs: #1316's file list through the GitHub API (head 75cb05fc). #1317 is closed (2026-11-09T23:44:17Z), and its head cb0f4634 is the head of ragnorc/omnigraph gq3/phase-0-instrument.
  • Not checked: the instrument numbers on line 16 (I did not rerun the instrument), and the hand reading.
  • CI at a8c332ff when I wrote this: 19 pass, 4 skipping, 7 pending, none failing.
  • A regression test does not apply: the commits change only RFC text.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the delta 7d57cec..e56a86e (one commit, "agents rewrite text from the respelling map; no rewriter is built"). The review on 7d57cec and the reviews before it cover the earlier commits.

Why, what, how, impact. The RFC no longer builds the one-shot rewriter. Agents rewrite the repository's text in the break, and later every stored source, from the "respelling map" (the old rewriter table) and the fix that each refusal prints. The proof for the repository changes from a per-edit check (typed IR, else plan, else rows) to one corpus check: the GQT corpus runs green, every case keeps its rows, shapes and plan lines except where "the fixed map" names a move, and the reference interpreter agrees. The commit also removes the stale T38 from the match_terms row (:920) and sets updated: 2026-11-13. Only docs/rfcs/2026-11-07-gq-3.md and its before-and-after asset change. There is no code.

Outcome: 2 blocking gaps in the new proof rule. The direction is sound. Not building a tool that is used once and then deleted is a real saving: no edition-2-to-core lowering, no fragment mode, no idempotence and layout tests. A corpus that keeps its expectations is a simple check that anyone can rerun. The two gaps below are about what the new check does not reach, and about the one artifact it depends on.

Blocking

B1. Text outside the GQT corpus now has no check at all.

  • :1422-1431 says that agents rewrite "the GQT corpus, user guide, skill, examples and benchmarks", and then: "For the repository the check is the GQT corpus". :15 and :1849-1851 give the same gate. The user guide and the skill are not in the corpus.
  • At e56a86e there are 21 gq fences in docs/user, 33 in skills/omnigraph, and 3 .gq files outside the corpus (docs/user/queries/, crates/omnigraph/tests/fixtures/). scripts/check-docs.py checks links and fences, not query text, and no test parses these blocks. (I counted the fences with grep -c '^```gq' and searched scripts/ and the crate tests for a parser over doc fences. I found none.)
  • So a skill example that an agent leaves in edition 2, or rewrites with a change of meaning, ships green. This matters more than for most docs: the skill is the text that agents copy, and this RFC now depends on agents to write edition 3 correctly. This gap replaces B3 of the review on 7d57cec. That one was a rows level that passed when no case ran the text. Now nothing runs the text.
  • The violated invariant: "No edition-2 text compiles after the break" (Compatibility) is checked only where the compiler sees the text. Docs and the skill are text that the compiler never sees.
  • Fix by design, at the layer that owns the text: add one test that parses every gq fence in docs/user and skills/omnigraph, and every .gq file, under edition 3, and type-checks each fence against a schema the fence names. A block that is not a whole declaration is marked as a fragment and is only parsed. Name this test in section 10 beside the GQT corpus line, and add it to the blocked_on gate at :15.
  • Cost: one small test and a fence convention. It also removes a liability that exists today: a docs example can drift from the language after any later change, not only after the break.

B2. "The fixed map" names one move, but five gates depend on it naming all of them.

  • :15, :18, :1430, :1851 and :2153 say that a case keeps its rows, shapes, plans, codes and column names "except where the fixed map names a move". The only move that the text gives to "the fixed map" is the _distance rename (:1453, :1502). There is no table of moves.
  • The RFC itself already knows of more moves in the corpus:
    • Compatibility (:2240-2248) accepts texts that edition 2 refuses. The corpus has expect error cases for T38 (8), T5 (4), T40 (2), T42 (5) and T9 (1). Each one whose text edition 3 accepts becomes a case with rows.
    • The 3 order keys over groups move the other way, from rows to a refusal.
    • The 17 fusion cases are "rewritten by hand" (:2237-2240).
    • The 7 reads with limit and no order change plan under the tie rule (B2 of the review on 7d57cec, still open).
  • With the rewriter gone, this list is the only thing that bounds which expectations an agent may edit. If the list has one row, the gate cannot pass, or agents edit expectations outside it. A changed expectation that is written to match the new output proves nothing.
  • Fix: make the moves one table in section 6, one row for each moved code, column name or plan class, with its corpus cases. Say that the check is mechanical: the --- expect blocks of the edition-3 corpus are byte-identical to edition 2's, except in the cases that the table lists, and the reference interpreter must agree with each new expectation in the listed cases. Use one name for one thing. Today "respelling map" (the text rows) and "fixed map" (the moves) appear side by side, and :1430 uses both in one sentence.
  • Cost: one table, which the earlier reviews already asked for in parts. No code.

Still open from earlier reviews

These parts of the text are not in this delta, so I checked them by reading the head text:

  • B1 of the review on 7d57cec: a let that can fail still stops all pushdown of where (:517-523).
  • B2 of the review on 7d57cec: the Cut row (:245) still declares cost k over an unordered input that it first orders by the tie rule.
  • B2 of the review on b4f32a4: T68 beside other elements still has no row (the not { } row, :1442).
  • B1 of the review on 76feb8a: alpha-renaming of block-local names still has no row.
  • B2 of the review on 76feb8a: blocked_on (:17) still says that the closure table is green on spike/gq3-typeck. The front-matter date is fixed now.
  • O1 of the review on 76feb8a: the rrf row (:1456) still prints <the rest of P, correlated on $d>. Now that agents write this row by hand, they need the real from more than a tool did.
  • B3 of the review on 9853010: the **I**, **Poly** row is still a paragraph and not part of the table.

Optional

  • O1. The per-row checks are now claims, not checks. :1427-1429 says that "each row states the strongest of three checks its meaning meets". Nothing in the break computes typed IR equality any more, because no edition-2-to-core lowering is built. The IR and plan claims rest on the 2026-11-08 spike (clause-list surface) and on the 98 converted ground truths. That is acceptable, but say so: give each row its evidence and its date, and say that only rows are checked in the break. Rows on a seed are a weaker check. For example, the not coalesce(…, false) row changes rows only when the seed has a null in that property. A reader should know which rows the corpus can actually see.
  • O2. Stale wording. :1386-1387 says that agents rewrite stored text "in the break". Stored sources outside the repository are rewritten later, when their owner meets the refusal (:1425). The decision log entries at :2407, :2425-2426 and :2448 still describe the rewriter. That is correct as history, but the entry of 2026-11-13 could say that it supersedes them. The section "The spike: rerun on the new rewriter rows" was removed, so the counts at :1465-1473 (600 rewritten, 50 refused, 28 multi-statement bodies) are now of the old surface with no owner that measures them again. Say this next to the counts.

Tradeoffs and liability

  • Removed. A tool with its own parser bridge, edition-2-to-core lowering, fragment mode and test list, which the project would write, test and delete. That is a real saving in work, even if it is not a lasting liability. No upgrade command still ships, so no compatibility path remains.
  • Added. The proof is now weaker and more spread out. Before: one tool refused any edit it could not prove. Now: agents follow a map, and the corpus is the only check. The corpus has 1,529 query steps, and only 385 of them have expect plan lines, so most rows are checked on one seed only. The risk moves into B1 (text the corpus never runs) and B2 (expectations that change with no list). Both fixes are small and lasting: a doc-fence test that also stops later drift, and one table of moves.
  • After five more changes of this kind. Each later language change will again need agents to rewrite docs and the skill. With B1's test, that is a CI failure that points to each block. Without it, it is a review task that nobody owns.

Validation

  • Exact head e56a86e, previous reviewed head 7d57cec. I reviewed in a clean detached worktree. The delta is docs/rfcs/2026-11-07-gq-3.md (+96 -100) and docs/rfcs/assets/2026-11-07-gq-3-before-and-after.md (+12 -12), with no code.
  • python3 scripts/check-docs.py with the pinned packages from scripts/requirements-docs.txt: "Documentation OK (473 Markdown files checked)". git diff --check 7d57cec1 e56a86e4: clean. typos is not installed on this host, so I did not run it.
  • B1: counted the gq fences and .gq files outside crates/omnigraph-gqt/cases/, and searched scripts/ and the crate tests for a check that parses doc fences. I found none.
  • B2: grep for "fixed map" in the head text (9 lines, one named move), and counts of --- expect error for each code over crates/omnigraph-gqt/cases/. I did not decide which of those cases edition 3 accepts. That is the table B2 asks for.
  • Corpus size: 1,529 --- query steps, 385 --- expect plan, 393 --- expect error, 34 --- expect same as v1.
  • No Decision needed comments on this PR.
  • GQT regression rule: this delta changes no behavior, so no GQT regression applies.
  • CI on the exact head: 27 checks passed, 4 skipped.
  • Not done: I did not compare the delta with #1316's branch, and I did not run the spike corpora, which are outside the repository.

Lance's fuzzy expansion budget belongs to the whole query; its default analysis drops tokens of 40 bytes or more; an IVF index over a cosine field stores normalized vectors, so a cosine search re-scores its candidates. Spell the schema form @index(fulltext, ...) on the property everywhere, apply row-level read policy to the field corpus once it exists, qualify repeated analyzed terms in stage L, list the search capabilities section 4 leaves out and why, and compare recall:, distances and fusion with LanceDB in Precedent.
english_v1 keeps Lance's stem-then-stop order, the only order Lance's index builder produces, so a plain stop list indexed 13 stop words in stemmed form. The profile's checked-in list holds each of NLTK's 198 English stop words and its Snowball stem, and the profile records the list's digest and the stemmer's identity. Dropping a word whose stem is a stop word and folding after stemming stay part of the profile; a stop-first profile waits for a configurable filter order upstream.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the delta e56a86e..f6038ef (11 commits, bdacbb7 to f6038ef). The review on e56a86e and the reviews before it cover the earlier commits.

Why, what, how, impact. This delta closes plan-dependent answers and corrects the Lance facts. It changes only docs/rfcs/2026-11-07-gq-3.md (+274 -65). There is no code. The changes:

  • A1 gets a read work budget, charged by the rows that each expansion produces (:162-165).
  • The plan budget is now charged while the tree is built (:301-304).
  • optional { p } writes R once (:371).
  • and and or are left-to-right guards, and a where filters its conjuncts in written order (:325-330).
  • Comparison treats -0.0 and 0.0 as equal, but the value order keeps them apart (:351-355).
  • A whole-number float literal stays F64, and a negated integer literal takes the other operand's type.
  • Negative numbers are allowed in list literals, explain can follow declarations, and three new recognizers print a fix.
  • Section 4 gets the Lance 13 corrections, an english_v1 stop list that holds each stop word and its stem (:891), a "Left out" table (:1208) and a LanceDB comparison (:844).

Outcome: 2 blocking gaps. The rest of the delta is correct, and most of it removes liability. Each new rule names a real plan-dependent answer and fixes it with one stated rule, not a special case. Both blocking items are new rules whose reach stops short of the class they target.

Blocking

B1. The read work budget misses the other operator that multiplies rows.

  • A1 (:162-165) says that "each expansion charges the rows it produces". The decision log (:2684-2698) says that this "bounds every hop". The motivating case is a chain of Expands that streams with flat memory.
  • But Expand is not the only operator whose output can be larger than its input. The cardinality column of the operator table shows two: Expand (≤ |R| · Σ fanout^h) and Correlate (Σ_r |f(r)|). Section 3 (:571) says that "from is the one element that multiplies rows by a relation", and from is a Correlate. So let $r = { … }, from $r, from $r, from $r multiplies |r| three times and charges nothing. It is the same failure as the twenty-hop chain: it stops in principle, it streams with flat memory, and no budget fires. A key-pinned product, an inline read and an optional are Correlates too.
  • #1316 shows the gap in code. At eebe693, charge_produced is called only in engine/operators/expand_stream.rs:509 and the lexical scan (engine/lexical_scan.rs:301-329). CorrelateExec (operators/correlate.rs:229) and CrossJoinExec (operators/cross_join/mod.rs:99) reserve memory, but they do not charge produced rows, and the cross join streams its pairs.
  • The violated invariant: "every read's work is bounded by one budget" (A1). Today it holds for one of the two row-multiplying operators.
  • Fix by design, at the layer that owns the contract: charge the rows produced by every operator, or by every operator whose cardinality bound can be larger than its input. The operator table already lists these, so the rule follows from it and needs no second list. A1 then reads "each operator charges the rows it produces". A new operator gets the charge by construction, and nobody has to decide each time. Add a GQT case beside the chain case: a triple from over one named relation that exhausts a small read_work_limit with the typed error.
  • Cost: one sentence in A1, and one charge call in the shared operator path in #1316. It removes an open class instead of one path.

B2. The new conjunct order breaks the key-pinned product rule.

  • :325-330 says that the planner "pushes and reorders the conjuncts before the first one that can fail, and none after it".
  • :372 admits a product of two components only when every component but one is key-pinned, by "a top-level conjunct" that equates its key with a literal or a parameter. That rule exists so that the product costs one component, not |A| · |B|.
  • The two rules now conflict when the pin comes after a conjunct that can fail. Example: match { $a: Person, $b: Company } where $a.score / $a.n > 2 and $b.slug = "acme". Under :329 the pin cannot move into $b's scan. If no company acme exists, evaluating the pin first gives zero rows, but written order fails on any person with n = 0. So the planner must build |Person| · |Company| rows before it filters. T81 does not fire, so the read is accepted, and its cost is the product that T81 exists to refuse.
  • The violated invariant: a rule that admits a term because of its cost must hold under the evaluation order the RFC fixes.
  • Fix: a conjunct pins a component only when it comes before the first conjunct that can fail. This is the same boundary as the pushdown rule, so there is one rule, not two. A pin after that point is T81, and its printed fix moves the pin first. That keeps "the answer is a function of the text" and makes the cost bound true. It needs no new state. Add the T81 case to the corpus.

Optional

  • O1. The cosine sentence gives the wrong reason (:1097-1102). The text says that Lance "stores the vectors normalized and searches them under L2, so an index-served value is not the field's cosine distance". In Lance 13.0.0, IVF_FLAT normalizes the stored vectors (lance-index vector/ivf.rs:153-160) and the query (lance io/exec/knn.rs:185-196). It finds partitions under L2 (index/vector/ivf/v2.rs:2197-2205). But it scores inside a partition with the cosine metric: v2.rs:2146-2152 changes the sub-index metric to L2 only for PQ. So the value is the cosine distance of the normalized copies. That is the field's cosine distance in exact arithmetic, and it differs only by f32 rounding. The conclusion still holds: GQ 3 wants the oracle's exact value, so it re-scores. Say "differs by rounding from the exact value", not "is not the cosine distance". I also checked :936-940 against Lance: normalizing a zero vector gives NaN, and KeepFiniteVectors (vector/transform.rs:84-130) drops that row from the index. So "the scan and the index agree" holds only because the scan drops the row too, as the text says. Good.
  • O2. Two equalities need one owner each. :351-355 now defines two equalities: IEEE comparison (=, in, the E2 zero-divisor test) and the total value order (Order, distinct, grouping). Today both of the engine's evaluators compare with arrow_ord::cmp: the in-memory evaluator (crates/omnigraph/src/engine/expr.rs:258) and the reference interpreter (crates/omnigraph-reference-engine/src/projection.rs:110). In arrow-ord 58.4.1, that kernel compares floats by totalOrder, so -0.0 != 0.0 there (cmp.rs:72-74: "normalize zeros before calling this kernel"). The reference interpreter is the oracle that the break's proof depends on. Name one comparison kernel that normalizes zeros, and use it in the evaluator, the reference interpreter and the pushdown translation. Also state in one place which operator uses which equality. This includes any hash-based in set or hash join, because hashing to_bits splits the two zeros. A GQT case with a stored -0.0, read with and without a let that can fail, would pin the decision log's example.
  • O3. "Expansion" now has two meanings. At :301 it means a compile-time unfolding (an inlined Call, an inline read, an optional). At :162 it means rows produced at run time. B1's wording ("each operator charges the rows it produces") removes the second use.
  • O4. Say how optional grows when it is nested. The row (:371) says that the term "grows linearly in the number of optionals". That is true for sequential optionals. The decision log correctly says that a nested one doubles its inner term. Put that clause in the row.
  • O5. State the pushdown rule once. :364 states it as "the planner's choice wherever it crosses no can_fail term", and :329 states the conjunct-order form. Keep :329 as the one statement and make :364 point to it. A lint that names a total, selective conjunct written after one that can fail would show agents the pushdown they lose. It prints a fix and does not change meaning.
  • O6. Two work settings. The decision log explains why traversal_work_limit is not reused. But the RFC does not say what happens to it: does it stay as a second, pre-scan admission check, or does it go when the produced-rows budget covers every operator? Name the end state so the next change does not add a third setting.

Verified in this delta

  • Lance fuzzy budget (:1038-1045): lance-index 13.0.0 scalar/inverted.rs:117-170 has one max_expansions (default 50, inverted/query.rs:83) for the whole query, walked in position order, with lexicographic candidates (BTreeSet), and it stops when the budget is used up. Correct.
  • Long tokens (:910-913): the default max_token_length is Some(40) (inverted/tokenizer.rs:641), and RemoveLongFilter keeps len() < 40 (lance-tokenizer remove_long.rs:55), so tokens of 40 bytes or more are dropped. Correct.
  • Whole-number float literals (:2354-2357): edition 2 narrows a literal that round-trips (omnigraph-compiler ir/coerce.rs:198-215,355-369), so the Compatibility row is accurate.
  • Stop list (:891-905): Lance 13.0.0 builds the analyzer as long-token filter, lowercase, stemmer, stop words, then ASCII folding (inverted/tokenizer.rs:1079-1095), and it accepts a custom stop list (:1110-1116). So stemming before stop words is the only native order, and an explicit list works. One small point: after folding, "résumés" indexes as resume, not résumé. The two words still stay apart, as the text says. I did not count the 13 stemmed NLTK words.
  • optional: under bag semantics, the new term gives the same rows as the old one for each outer row, including duplicates. The column order is unchanged.
  • The LanceDB search_effort quote and the RRFReranker K = 60 default: I did not verify these against LanceDB's documentation.

Still open from earlier reviews

This delta does not touch these lines, so I checked them by reading the head text:

  • B1 and B2 of the review on e56a86e: there is still no test over the gq fences in docs/user and the skill, and "the fixed map" (9 mentions, including blocked_on at :15) still names one move.
  • B2 of the review on 76feb8a: blocked_on (:17) still says the closure table is green on spike/gq3-typeck.
  • B2 of the review on 7d57cec: the Cut row (:248) still declares cost k over an unordered input that it orders first.
  • The other items listed in the review on e56a86e are not in this delta.

Tradeoffs and liability

  • Removed. Four plan-dependent answers (and/or pushdown, -0.0, a let that changes row counts, literal narrowing above 2^53), each fixed by one rule with its alternative weighed. optional's term growth goes from exponential to linear for sequential optionals. A compile that is refused now costs at most its budget. explain now reaches composed reads, which closes a gap in A3. The Lance facts now match 13.0.0, and "Left out" gives each missing capability a place, with no new core.
  • Added. A second run-time work setting, and a conjunct-order rule that agents must learn. Left to right is the right choice: the alternative scans the whole type for every where that has arithmetic. But it moves the cost of a badly ordered where onto the author, which O5's lint would show.
  • After five more changes of this kind. If each new operator must decide whether it charges the budget, one will forget, as Correlate has here. B1's rule makes the charge a property of the operator table, so it holds by construction.

Validation

  • Exact head f6038ef, previous reviewed head e56a86e. I wrote this review on 45354fb. The head then moved to f6038ef (one commit, the english_v1 stop list), and I checked that commit before posting. I reviewed in a clean detached worktree. The delta is one file, with no code.
  • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt: "Documentation OK (473 Markdown files checked)". git diff --check e56a86e4 f6038ef2: clean. typos is not installed on this host, so I did not run it.
  • Lance claims: read in the pinned lance, lance-index, lance-linalg and lance-tokenizer 13.0.0 sources (Cargo.lock at head). The references are above.
  • B1: code inspection of #1316 at eebe693 (git grep charge_produced, correlate.rs, cross_join/mod.rs). I did not build #1316 or run a product against a budget.
  • No Decision needed comments on this PR.
  • GQT regression rule: this delta changes no behavior on main, so no regression applies. B1, B2 and O2 each name the case the implementation should add.
  • CI at the exact head f6038ef: 20 checks passed, 4 skipped and 5 still running when I posted. DST pinned suite failed in "Install system dependencies" (E: Unable to locate package protobuf-compiler, run 43774389101), which is a runner apt failure, not this change. The same suite passed on main at 4f7c207.
  • Other open PRs: #1316 is the implementation (B1 applies to both). #1405 and #1244 do not touch these sections.

…file

A profile fixes the analyzed terms. Whether a native index stores positions is recorded in its RFC 0043 certificate, not in the schema fingerprint; indexes are built without them. A later phrase predicate runs on any profile through the exact scan and goes native when the index is rebuilt with positions, with no schema change and no new profile.
…alar index kinds by type

Every full-text field has a profile, so edition 2's native-only analysis does not survive the break and FullTextIndexRequired is removed: an unindexed or partial field is answered by the exact scan, the edit relation holds on every field, and bm25 is bm25_v1's F64 everywhere. @index builds BITMAP for enums and Bool, LABEL_LIST for a list's in, NGRAM for a String's contains and BTREE otherwise, each derived acceleration with an exact recheck where it returns candidates.
…A6 a default

The break rewrites every stored text, so the three lexical synonyms protect nothing; a call to one is refused at type check with its match_terms fix, as infix contains is, and the formatter's lexical rewrite and lint row go. A6 states its reason, agents and skill distribution, and admits a second spelling where it earns its place, named in the formatter's list.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the delta f6038ef..0c7f88c (4 commits, 59df9c0 to 0c7f88c). The review on f6038ef and the reviews before it cover the earlier commits.

Why, what, how, impact. This delta removes the last two-regime rules from lexical search and makes index kinds follow from the type. It changes only the RFC text (docs/rfcs/2026-11-07-gq-3.md +130 -67) and one line in each of two assets. There is no code. The changes:

  • An omitted analyzer is now english_v1. Edition 2's native-only analysis does not survive the break, and FullTextIndexRequired is removed: the exact scan answers a field with no index, a partial index or an index for another analysis (:923-938).
  • Term positions are an index build option recorded in the RFC 0043 certificate, not part of a profile (:911-918).
  • @index builds the Lance scalar index kinds that serve the property's type (:940-950).
  • search, match_text and fuzzy are removed at the break and refused at type check with the match_terms fix (:1110-1117, :829).
  • A6 becomes a default, not an absolute (:183-191).
  • queries fmt is renamed queries format.

Outcome: 2 blocking gaps. The direction is sound and most of the delta removes liability. One analysis per field, one lexical predicate and one oracle path are simpler than two regimes. Both blocking items are about what the new defaults cost and what they say, not about the choice.

Blocking

B1. The default profile moves every full-text read in the fleet to the exact scan from B until L, and the RFC neither states nor measures that cost.

  • Before this delta, a field without analyzer kept edition 2's native index. So existing graphs kept native full-text speed through B. Now every field is english_v1 (:923).
  • No native path can serve english_v1 in B. english_v1 has NFC (:893), the Lance index has no NFC step, and the NFC route into the index is a stage L decision (:1074-1080, Unresolved 10 at :2421). Stage B ships "the exact lexical scan on every full-text field" (:2098), and native acceleration is stage L (:2100).
  • So at B, every bm25 and match_terms read on every graph tokenizes the whole field on each query. A1's read work budget charges those rows (:162-165), so a read that the index answers today can now fail with the typed budget error on a large field. The RFC gives no cost figure for the exact scan. M0 measures competence, not latency.
  • The Compatibility row (:2371-2373) says the field is "served by the exact scan until its index is rebuilt with the profile". That is not correct for B: a rebuild does not restore a native path until L qualifies one. The decision log (:2795-2800) gives the row changes but not the speed change.
  • The violated invariant: A1 says that a read's cost is "known before execution". The RFC has to state the cost of its own default too. A break that changes every fleet full-text read from an index probe to a full scan is a cost change of the same order as the row changes it lists.
  • Fix (keep the default, which removes real liability): (1) Correct :2371-2373 to say that native acceleration comes back only at L, after rebuild. (2) Add a B ship condition: the exact scan over the largest full-text field in the fleet completes inside the default read work budget and a stated latency, with the measurement in this RFC. If it does not, move L's zero-edit path and the NFC route into B. (3) Put one line on speed in the decision log entry. This adds no state and no new rule.

B2. The two statements of the scalar index kinds disagree for String.

  • Section 4 (:942-944): "BTREE for equality, ranges and order; BITMAP for an enum or Bool; LABEL_LIST for a list, serving in; and NGRAM for a String, serving contains". This reads as BTREE and NGRAM on an indexed String.
  • The decision log (:2801-2804): "BITMAP for enums and Bool, LABEL_LIST for a list's in, NGRAM for a String's contains, BTREE otherwise". This reads as NGRAM instead of BTREE on a String.
  • Under the second reading, name: String @index loses its BTREE, and every equality, range and order on an indexed String goes from an index probe to a scan. NGRAM does not serve equality. Today @index on a String builds BTREE (table_store.rs:3425, :6003 at main 4f7c207).
  • Fix: state the rule once, by predicate: each kind is built for the predicates it serves on the property's type, so a String gets BTREE and NGRAM. Make the decision log entry point to section 4. Lance 13 permits this: load_scalar_index_info keeps a fragment bitmap per (column, index name) and merges several query parsers for one column (lance 13.0.0 src/index.rs:3919, :3947-3952).

Optional

  • O1. A stale rule in the closure table. The "Calls" row (:1511) still says "the registry holds today's functions" and "a registry row is never removed". After this delta the break removes three rows. Say "today's functions less search, match_text and fuzzy" and "after the break, a registry row is never removed", so that the rule has one statement that matches :1897 and the decision log.
  • O2. Give A6's admission test one owner. "Where it earns its place" (:186) has no test. Governance rule 2 (:2299) already says that each new form is "answered in the change's Alternatives". Make A6 point to it: a second form is admitted only by an Alternatives entry, and it is then listed in the formatter's list. That keeps A6 a default with a check, not a judgement each time.
  • O3. The cost of NGRAM on every indexed String. NGRAM is built for every @index String, including long text fields, whether or not any read uses contains. The "Index segments" fence (docs/dev/lance.md:187) rebuilds a lagging index whole, so each added kind adds one more whole rebuild per fold. After five more kinds of this type, every write pays for each kind on each property. Two options: measure NGRAM build time and size on a long text field and record it, or make NGRAM opt-in with an option on @index, as fulltext already is. I do not ask for a change if the measurement is acceptable.
  • O4. The fence rows do not exist yet. :948 says "Each kind has its fence row and guard in lance.md". NGRAM has a guard (lance_surface_guards.rs::contains_filter_routes_to_ngram_index_and_rechecks_exactly) but no fence row. BITMAP and LABEL_LIST have neither. Write it as a B deliverable and add it to the B row (:2098).
  • O5. "Additive" no longer holds for the analyzer. :2370 says that "every declaration of section 4 is additive". The default analyzer changes the meaning of every accepted @index(fulltext), which :2377-2381 then admits. Say "applied in place" without "additive", or name the analyzer as the one exception.
  • O6. The implementation lags the new rules. #1316 at f1f2b80 still prints "edition 2 analysis" for a field without analyzer (omnigraph-cli/src/output.rs:1002) and still has FullTextIndexRequired (omnigraph-planner/src/error.rs, the issue_747 GQT case). That is expected for a draft. List it in #1316's plan so that the break does not ship with the old regime.

Verified in this delta

  • Positions: Lance 13.0.0 builds an inverted index without positions by default (lance-index scalar/inverted/tokenizer.rs:640, with_position: false). OmniGraph builds with InvertedIndexParams::default() (table_store.rs:4773, :6013). So "Indexes are built without positions" is today's behavior. Lance also treats with_position as a storage setting outside same_tokenization (scalar/inverted/combined.rs:88-90). This supports keeping positions out of the profile.
  • NGRAM recheck: main has a guard that contains routes to an NGRAM index and rechecks to exact rows, including needles shorter than a trigram (lance_surface_guards.rs:6359-6406, tests/integration/literal_filters.rs:423). So "rechecks the candidates where Lance's index returns a superset" matches the engine.
  • The search/match_text/fuzzy removal is consistent across the error table (:829), Migration (:1110-1117), the respelling map (:1546-1547), the formatter list (:1636-1639), Alternatives (:1897, :1908) and stage B (:2098). The fuzzy with no budget maps to max_edits: 2 in each place.
  • The queries format rename is complete in the RFC and both assets. Only the 2026-10-29 decision log entry still says fmt, which is history.

Still open from earlier reviews

This delta does not touch these lines, so I checked them by reading the head text:

  • B1 and B2 of the review on f6038ef: A1 (:162-165) still charges only "each expansion", and the conjunct-order rule (:328-334) still does not limit which conjuncts pin a product component (:375).
  • B1 and B2 of the review on e56a86e, B2 of the review on 76feb8a and B2 of the review on 7d57cec: the lines are not in this delta.

Tradeoffs and liability

  • Removed. Two analysis regimes per field, FullTextIndexRequired and its refusal path, three synonyms of match_terms with their formatter rewrite and lint row, and the phrase-search rebuild that would have needed a new profile. Each was a rule that agents and the skill had to carry.
  • Added. A one-time row change on every full-text field at the break, a speed change from B to L (B1), and up to four index kinds per property with their maintenance cost (O3).
  • After five more changes of this kind. "Default, one regime" scales well. "Every applicable index kind by type" does not scale by itself: each new kind adds build and rebuild cost to each property that has the type. A rule that ties each kind to a predicate in use, or an opt-in, keeps that cost bounded.

Validation

  • Exact head 0c7f88c, previous reviewed head f6038ef. I reviewed in a clean detached worktree. The delta is RFC text and two asset lines, with no code.
  • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt: "Documentation OK (473 Markdown files checked)". git diff --check f6038ef2 0c7f88cf: clean. typos is not installed on this host.
  • Lance claims: read in the pinned lance and lance-index 13.0.0 sources (Cargo.lock at head). Engine claims: read at main 4f7c207 and #1316 at f1f2b80. I did not build either, and I did not measure the exact scan (B1 asks the RFC to do that).
  • No Decision needed comments on this PR.
  • GQT regression rule: this delta changes no behavior on main, so no regression applies.
  • CI at the exact head 0c7f88c: 27 passed, 4 skipped.
  • Other open PRs: #1316 is the implementation (O6). #1405 and #1244 do not touch these sections.

…nonical forms

queries format is a printer for the whole grammar, cheapest to build once the grammar is frozen. Stage B keeps the canonical forms as lint's list, each named with its fix, so agents and the skill still carry one form; A6 points at that list, and stage F builds the formatter.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the delta 0c7f88c..ddc3b55 (2 commits, 85fbcf7 and ddc3b55). The review on 0c7f88c and the reviews before it cover the earlier commits.

Why, what, how, impact. This delta takes the formatter out of the break. It changes only RFC text (docs/rfcs/2026-11-07-gq-3.md +46 -35) and one line in each of two assets. There is no code. The changes:

  • The section "queries format" is now "Canonical forms" (:1630-1656). The list of forms is the same. Lint now names each form, with the canonical form as its printed fix. The formatter does not apply it at B.
  • A new stage F builds queries format "after the grammar is frozen" (:1652-1656, the F row at :2108). The B row no longer lists queries format (:2104).
  • A6 (:183-190), the closure paragraph (:395-398), the let/from order rule (:685-686), bm25 (:1003), the L lint bullet (:1427-1431), the M0 note (:2029) and governance rule 2 (:2306-2308) now point to lint's list, not to the formatter.
  • The two assets now say that the reversed triple is T82 with the swapped triple as its printed fix. This matches the RFC (:486, :812, :1509). Before, they said that the formatter rewrites it.

Outcome: no blocking defects in this delta. It removes liability. The formatter is a printer for the whole grammar. Building it at B, while the grammar changes, would make each grammar change also change the printer. Lint already carries the list, and agents read lint's fix. So one owner (lint's list) now holds the canonical forms, and the printer comes later. #1316 at ba6d015 has no formatter yet, so the deferral removes planned work and does not undo work. The two asset fixes correct statements that were false against the RFC's own T82 rows.

The blocking items of earlier reviews are still open (see below).

Blocking

None in this delta.

Optional

  • O1. Governance rule 2 is not a complete sentence. :2306-2308 reads "a form that falls out of a general construct is canonicalised by named by lint, with the canonical form as its fix". Write "is named by lint, with the canonical form as its fix". Also, the earlier O2 is still open: A6 "where it earns its place" (:186) has no test, and rule 2 is that test. Make A6 point to rule 2.
  • O2. "With the grammar frozen" is not a point that the sequence table states. Rule 3 says "an order the sequence table does not state does not exist". The F row's gate is "B, with the grammar frozen" (:2108). The freeze list (:1495-1499) freezes edition 3 at B. But productions continue to arrive after B: "Each production below is an amendment under rule 1 when its stage lands" (:2133), for RP and Path. So the reason given at :1654-1656 ("while the grammar still changes every change would have to update it") also applies after B. Two options, each small:
    • Gate F on B only (the freeze list), and add to rule 1 that an amendment that adds a production also adds its canonical forms to lint's list and, after F, its print rule, in the same change. This states the maintenance cost once.
    • Or gate F on the last stage that adds productions (name it), and accept that F comes late.
  • O3. Nothing at B makes stored text use the canonical forms. A6 used to say "so stored text and the skill still carry one form". Now it says "so the skill teaches one form" (:189). That is correct, but the B row (:2104) has no check that holds the forms in stored text. Today, query lint runs only from the CLI lint command (crates/omnigraph-cli/src/commands/schema/lint.rs is its only caller). Its L201 finding is a warning (crates/omnigraph-compiler/src/query/lint.rs:303) and does not fail the run. cluster validate and the stored-query registry do not run it. The fix costs no new tool: add to B's ship condition that the rewritten GQT corpus, user guide, skill and fleet text give no canonical-form lint finding, and state the severity of those findings (warning or error) in the L bullet (:1427).
  • O4. Two stale statements.
    • The comparison asset still says "a formatter that writes schema order" (docs/rfcs/assets/2026-11-07-gq-3-grammar-comparison.md:257). After 85fbcf7, the reversed triple is refused (T82) and no formatter rewrites it. This is the same class as the line that 85fbcf7 corrected at :70.
    • :1427 says "The lint prefix L is added". The L prefix exists today (L201, crates/omnigraph-compiler/src/query/lint.rs:12). Say that the canonical-form codes are added to it.

Still open from earlier reviews

This delta does not touch these lines, so I checked them by reading the head text:

  • B1 and B2 of the review on 0c7f88c: the Compatibility row (:2378-2380) still says that a rebuild restores acceleration, and the B row (:2104) still has no measured exact-scan cost. Section 4 and the decision log still give two different rules for the String index kinds. O1 of that review is also still open: the "Calls" row (:1510) still says "a registry row is never removed".
  • B1 and B2 of the review on f6038ef, B1 and B2 of the review on e56a86e, B2 of the review on 76feb8a and B2 of the review on 7d57cec: the lines are not in this delta.

Tradeoffs and liability

  • Removed. A printer for the whole grammar at B, which every grammar change before the freeze would have had to update. The formatter's lexical rewrite of the reversed triple, which the assets still described.
  • Added. A gap from B to F in which only lint, run by hand, keeps stored text in one form (O3). A new stage, F, with a gate that is not yet a stage (O2).
  • After five more changes of this kind. One list with one owner (lint) scales well: each new non-canonical form is one lint row with its fix. The printer then reads the same list. That stays true only if rule 1 makes each amendment update the list (O2).

Validation

  • Exact head ddc3b55, previous reviewed head 0c7f88c. I reviewed in a clean detached worktree. The delta is RFC text and two asset lines, with no code.
  • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt: "Documentation OK (473 Markdown files checked)". git diff --check 0c7f88cf ddc3b55e: clean. typos is not installed on this host.
  • Code claims: read at ddc3b55 (query lint callers and severity). #1316 at ba6d015: no queries format command. I did not build either.
  • No Decision needed comments on this PR.
  • GQT regression rule: this delta changes no behavior, so no regression applies.
  • CI at the exact head ddc3b55: 27 passed, 4 skipped (the tag-release jobs).
  • Other open PRs: #1316 is the implementation (see above). #1405 and #1244 do not touch these sections.

…rough a derived token column in stage L

The certificate records a digest of the native pipeline's terms over a checked-in probe corpus. A native index qualifies only when that digest matches the running binary's and equals the profile analyzer's; a test pins each profile's digests, and verification always names the accepted fingerprint.

Stage L writes the profile's analyzed terms to a derived token column during reconciliation and indexes it with Lance's whitespace tokenizer, so index and scan share one analyzer and NFC reaches the index. Unresolved questions 9 and 10 narrow to the probe corpus and the column's placement and coverage.

Also records the String index rule (plain @index builds BTREE and NGRAM, @key alone a BTREE, full text only with @index(fulltext, ...)) and the rollback rule (an older binary refuses a rebuilt full-text index; restore the backup).

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the delta ddc3b55..44010cc (1 commit, 44010cc). The review on ddc3b55 and the reviews before it cover the earlier commits.

Why, what, how, impact. This delta closes two of the RFC's open questions about lexical search, and records two rules. It changes only RFC text (docs/rfcs/2026-11-07-gq-3.md +74 -15). There is no code. The changes:

  • RFC 0043 certificates record a measured analyzer fingerprint: a digest of the terms that the native pipeline emits over a probe corpus that is checked in. An index qualifies only when its digest equals the running binary's digest and the profile analyzer's digest (:939-957). Stage B ships it (:2143).
  • Stage L shares one analyzer between index and scan. Reconciliation writes the profile's analyzed terms to a derived token column. Lance indexes that column with its whitespace tokenizer and all of its filters off (:1100-1119, L row :2145). Unresolved 9 and 10 become narrower (:2467-2473).
  • String index rule: plain @index on a String builds BTREE and NGRAM. @key alone builds BTREE. Only @index(fulltext, ...) builds a full-text index (:963-966, :2883-2885).
  • Rollback: an older binary refuses a full-text index that the new binary built, so the supported rollback is to restore the backup (:2444-2446, :2886-2887).

Outcome: 1 blocking defect, with a small fix. The two analyzer decisions remove liability and I agree with them. The derived token column makes "the index holds the scan's terms" true by construction. Before, it was a parity condition that each Lance upgrade could break. The measured digest turns a silent dependency drift into a test failure that names the profile. The String rule settles B2 of the review on 0c7f88c: section 4 and the decision log now give the same rule. But the same rule changes what today's schemas mean (B1).

Blocking

B1. The String rule gives edition 2's full-text spelling a new meaning, and nothing refuses it or rewrites it.

  • Today, @index on a String that is not an enum builds a full-text index. @key does the same, because @key adds its columns to the index list (crates/omnigraph-compiler/src/types.rs:242-247, crates/omnigraph-compiler/src/catalog/mod.rs:431-434). I checked this with the v0.15.0 binary. On node P { name: String @key title: String @index body: String }, omnigraph index build --json reports full_text for name and for title, and BTREE only for __id. The RFC agrees that "edition 2 spells a String's full-text index @index" (:1558).
  • After this delta, edition 3 accepts the same .pg text with a different meaning. The field gets BTREE and NGRAM and no full-text index. bm25 on it fails with T27, because bm25 needs "indexed full-text" (:1362).
  • The migration path does not reach this case. Agents rewrite .pg text "from the map below and the fix each refusal prints" (:1566-1569). Plain @index is not refused, so it prints no fix. The respelling map has no schema row. Compatibility still calls the schema change "additive" (:1558, :2416). It also says "an index without analyzer takes english_v1" (:2417-2418), which assumes that today's full-text fields stay full-text.
  • The size: 41 GQT cases declare a plain @index String and call a lexical function on it. An example is crates/omnigraph-gqt/cases/bm25_distinct_scores_rank_descending.gqt:13-15. B's condition is that every rewritten case keeps its rows "except where the fixed map names a move" (:15, :18). No map row names this move. On stored graphs, the first reconciliation after the upgrade would replace each full-text index with BTREE and NGRAM. NGRAM would then be built on long text bodies (O3 of the review on 0c7f88c).
  • #1316 at 4fce388 implements the other reading. Plain @index on a String builds FullText, NGram and Btree (types.rs:323-336 there), and any single-column String index is full-text (catalog/mod.rs:138-146 there). So the RFC and its implementation disagree.
  • The violated invariant is in this delta: "a binary upgrade never changes an accepted schema's fingerprint" (:956-957). The index kinds of an accepted schema need the same guarantee. Today the kind comes from the binary's rule (index_kind()), not from the accepted IR, so a rule change moves every accepted schema.
  • Fix (small, and it keeps the new rule):
    1. Store the index kind in accepted SchemaIR from B on. When the new binary reads an edition-2 IR, read a String Index as fulltext. Then a binary upgrade does not change what an accepted schema builds.
    2. Add a respelling-map row: @index on a String that edition 2 indexed as full text becomes @index(fulltext). A String that only needs equality or contains keeps @index. Add the matching Compatibility row. The 41 GQT cases then change their schema text but not their rows.
    3. Make #1316 follow the RFC, or change the RFC to match #1316. Record the choice in the decision log.

Optional

  • O1. The rollback sentence gives the wrong reason. Today's binary refuses a certificate that it cannot read: Certificate has deny_unknown_fields (crates/omnigraph-core/src/fts_compat.rs:27), and a changed or unknown field gives FullTextIndexRebuildRequired (tests at fts_compat.rs:480-503). That error tells the user to run index rebuild --kind full-text (crates/omnigraph-cli/src/commands/maintenance/mod.rs:270-277). So an older binary can repair a refused full-text index in place. The certificate alone does not force a restore. Other things do. An older binary refuses an IR that names a feature it does not know (crates/omnigraph-compiler/src/catalog/schema_ir.rs:102-105), and the analyzer fingerprint is such a feature name (:934-936). Edition-3 stored queries do not compile on an older binary. At L, a derived column in the graph table may also stop an older binary. A restore loses every write made after the upgrade, so the operator must know the exact boundary. Fix: say that the boundary is the first accepted IR that carries the analyzer feature (the break itself, for any graph with a full-text field). Remove "since it cannot read that certificate". Add "an older binary can still open the table" as a criterion in Unresolved 10.
  • O2. State the analyzer condition behind "by construction". Lance 13's whitespace tokenizer splits only on ASCII whitespace (lance-tokenizer 13.0.0 src/whitespace_tokenizer.rs:38, :50). Space-joined terms come back unchanged only if the analyzer never emits an empty term or a term that contains ASCII whitespace. Write that as a property of every profile, and put an input in the probe corpus that would break it (Unresolved 9). The parity check then tests the condition. It does not only assume it.
  • O3. Track coverage per fragment, not per row. The text "a row whose column is missing or was written under another fingerprint is uncovered" (:1113-1114) implies a fingerprint stored for each row. Lance tracks index coverage per fragment (the fragment bitmap). The certified fetch already needs full coverage (:1127-1128). A content write puts the new row in a new fragment, whose column is empty. So "a fragment whose column was written under the accepted fingerprint" gives the same guarantee with no state per row. I suggest this as the starting answer to Unresolved 10.

Verified in this delta

  • Lance 13.0.0 has the configuration that stage L needs. whitespace is a base tokenizer (lance-index 13.0.0 src/scalar/inverted/tokenizer.rs:1143). Each filter is a switch, and max_token_length: None removes the length filter (tokenizer.rs:1082-1096, :150-152). Lance's own test builds this configuration (builder.rs:4046-4052).
  • Lance's corpus statistics agree with the field corpus. Lance omits documents with zero tokens, null included, from row-index statistics (builder.rs:1521-1524, :1725-1736). The RFC counts only entities "with non-empty analyzed values" (:1046-1047). So a row with no analyzed terms changes neither side's N or mean length.
  • The delta settles B2 of the review on 0c7f88c: section 4 (:960-966) and the new log entry (:2883-2885) both say BTREE and NGRAM for an indexed String.

Still open from earlier reviews

This delta does not touch these lines, so I checked them by reading the head text:

  • B1 of the review on 0c7f88c: the Compatibility row (:2417-2419) still says that a rebuild restores acceleration, and the B row (:2143) still has no measured exact-scan cost. With this delta, native acceleration still waits for L, because the shared analyzer is an L deliverable (:2145). O3 of that review (NGRAM cost) matters more now; see B1 above.
  • O1 to O4 of the review on ddc3b55, and the earlier blocking items listed there: the lines are not in this delta.

Tradeoffs and liability

  • Removed. The parity gap between Lance's tokenizer and the profile analyzer, which each Lance upgrade could reopen. The question of how NFC reaches the index. Silent stemmer or tokenizer drift under an existing index.
  • Added. One derived column for each full-text field, about the size of the analyzed text, with its own coverage state (O3). A probe corpus and two pinned digests for each profile, which each dependency bump must keep. A meaning change for every edition-2 @index String unless B1 is fixed.
  • After five more changes of this kind. "The index stores what the scan computes" scales well: a new profile is a new analyzer and two pins, with no new parity work in Lance. Index kinds that come from the binary's rule do not scale: each new rule moves every accepted schema. Kinds stored in the accepted IR (B1 fix 1) keep each rule change local to new schemas.

Validation

  • Exact head 44010cc, previous reviewed head ddc3b55, merge base ea4a16d. I reviewed in a clean detached worktree. The delta is RFC text only.
  • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt: "Documentation OK (473 Markdown files checked)". scripts/check-agents-md.sh: OK. git diff --check ddc3b55e 44010cc3: clean. typos is not installed on this host.
  • Today's index kinds: v0.15.0 release binary, a scratch graph in an isolated OMNIGRAPH_HOME, init, load --mode merge with 2 rows, index build --json. The results are in B1. Code claims: read at 44010cc and in #1316 at 4fce388. Lance claims: read in the pinned lance-index and lance-tokenizer 13.0.0 sources (Cargo.lock at head). I did not build the engine.
  • No Decision needed comments on this PR.
  • GQT regression rule: this delta changes no behavior on main, so no regression applies.
  • CI at the exact head 44010cc: 27 passed, 4 skipped.
  • Other open PRs: #1316 is the implementation (B1). #1395 builds BTREE indexes concurrently and does not change which kinds are built. #1156 (incremental index reconciliation) is where the derived column's reconciliation would sit. It does not conflict with this delta.

…primary sources

An algorithm with a finite fixpoint (closure, connected components over both directions, weighted shortest distances under a round budget) is a monotone declared read typed to Fix. One that converges only in the limit, runs two passes per source, or depends on node order or a seed (PageRank, betweenness, Louvain, Leiden) is a relation function over a node population and an edge relation: declared options and bounds, a convergence report under A8, no predicate moved across the call, determinism rules for replay, whole-graph passes over a derived adjacency index, and the reference interpreter as its oracle. Cyclic pattern counts stay a match, accelerated by the planner's join choice. Adds unresolved question 18 on the adjacency index.
… name, trailing defaults, per-query refusal, a server budget ceiling, T86 messages

null is a literal wherever its context fixes its type; union matches arms by column name; a defaulted parameter is followed only by defaulted ones; a stored query that fails to compile makes only itself unavailable, with a strict boot setting; a declared plan budget above the server ceiling is refused; a T86 message names the construct, not the stage. The forms B does not yet lower stay in B, and section 7's grammar stands where the implementation differed.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the delta 44010cc..7bfff4d (3 commits: c933562, bff2863, 7bfff4d). The review on 44010cc and the reviews before it cover the earlier commits.

Why, what, how, impact. This delta gives stage R's graph algorithms a contract, and it decides eight implementation questions that #1316 raised. It changes only RFC text (docs/rfcs/2026-11-07-gq-3.md +167 -15). There is no code. The changes:

  • Graph algorithms (stage R) (:2298-2403). An algorithm with a finite fixpoint (closure, components, weighted shortest distances) is a monotone declared read that types to Fix. An algorithm that converges only in the limit, or that needs two passes, an order or a seed (PageRank, betweenness, Louvain, Leiden), is a relation function. The section gives its input, scope, options, output, bounds, determinism, execution and oracle, an example, and 24 sources. Unresolved 18 asks where the adjacency index lives.
  • union matches arms by name (:248, :637), not by position.
  • null is a literal wherever its context fixes its type. = null and != null are type errors with is null as the fix (:1331-1335, Null row :1561).
  • Trailing defaults: a defaulted parameter is followed only by defaulted ones (:1562).
  • Plan budget ceiling: a declared budget above the server's ceiling is refused, not clamped (:303-305).
  • T86 messages name the construct, not the stage (:1442-1446, :1468).
  • Stored queries at the break: a stored query that fails to compile makes only itself unavailable. A strict boot setting keeps the old behavior (:2531-2539).
  • Section 7 stands where #1316 differed: whitespace after a unary minus, no reserved word as a call name, no () on an annotation without arguments (:3037-3039).

Outcome: 2 blocking defects, each with a small fix. I agree with the direction of every decision in this delta. Union by name removes a class of silent wrong rows. The null rule, the trailing-default rule and the ceiling rule each give one clear refusal in place of a silent meaning. The two-route split for algorithms keeps the operator set at thirteen: an algorithm with a fixpoint uses Fix, and the others are registry rows. That is the smaller design. But the algorithm contract brings back the positional match that this delta removes from union (B1). The null decision is not in section 7's grammar, which the log says "stands" (B2).

Blocking

B1. The algorithm contract reads its edge columns by position, which is the failure that this delta removes from union.

  • The contract reads "an edge relation whose first two columns are references of that binding's type, read as source and target in written order, with an optional numeric third column as the weight" (:2319-2321).
  • The decision log gives the reason for union by name: "a positional match turns two swapped columns of one type into silently wrong rows" (:3027-3028). The Alternatives table refuses positional destructuring on from for the same reason (:1920).
  • An edge relation has exactly that shape: two columns of one type. A helper that returns { $b, $a } in place of { $a, $b } reverses every edge. PageRank, betweenness and closeness then return different rows, with no error. A weight column in the wrong place is the same.
  • The RFC already has the rule for this case: "A ranking function names the column it ranks as a property access on the population, $d.embedding, resolved at compile time, never a string" (:639-640). The contract follows it for the node binding ($p), but not for the edges.
  • Fix: name the edge columns at the call, as nearest names its field. For example, pagerank($people, $p, $follows, source: $a, target: $b, weight: $w), with each option naming a column of the edge relation. A missing or extra name is then a type error, as it is for a union arm. The rule for an undirected algorithm stays as it is (:2323-2324). With union by name, the reverse arm only binds the names in the other order: { match { $a: Person, $b: Person, $b follows $a } return { $a, $b } }.

B2. The null decision is not in section 7's grammar, and the log says that section 7 stands.

  • Section 7 has null only as an assignment value (null_value, :1763). operand (:1734) reads literal, and section 7 does not redefine literal. Today's literal has no null (crates/omnigraph-compiler/src/query/query.pest:266), and null is a reserved word (query.pest:226). So in section 7, if($c, null, $x) and coalesce($x, null) do not parse.
  • #1316 at a514157 follows section 7 here. I parsed four texts with its grammar.pest (pest 2.8.7, the version in Cargo.lock). if($p.age > 30, null, $p.name) and coalesce($p.nick, null) match reserved_property. Its parser then refuses them with Q005: "null is a reserved word; a property of that name is written $p.null" (syntax/parse.rs:4301-4310). That fix is wrong for this text.
  • The Summary also contradicts the decision. It keeps "three-valued logic with no null literal in reads" in its list of things that do not change (:82-83). The Null row (:1561) now names three places for null, but the grammar has one.
  • Fix: add null to literal in section 7, or add an operand alternative for it (null_lit = { kw_null }), and give the type rule at :1331-1335 as the owner of where it is accepted. Delete "with no null literal in reads" at :82-83. Then = null can parse and be refused by the type checker, as :1333-1335 says. Today #1316 refuses it with a parse recognizer (null_compare_rec, grammar.pest:380). Either works, but the RFC says "type error", so say which one.

Optional

  • O1. The new example does not parse. tolerance: 1e-9 (:2370) is not a float_lit in section 7, which needs a fraction (:1748). #1316 matches it with exp_float_rec and refuses it ("an exponent needs a fraction", grammar.pest:444-445). 1.0e-9 parses as a float_lit with both grammars. Write 1.0e-9.
  • O2. Three sentences still say that a T86 names a stage. After :1442-1446, a T86 message names no stage. But :1447 says "T86 never names the release's own stage", :1450 says "a T86 that names B", and the B row (:2152) says "with no T86 naming B". Write "a T86 whose documented stage is B". The Annotations row (:1563) says "every other name refused with the stage that types it". The reason at :1445-1446 (the message goes stale when the stage ships) applies there too.
  • O3. "In the same step as the binary switch" is not an operation that the deployment has. A server target's cluster apply runs in the server that is serving. An older server cannot load the new syntax (:2536). So the order is always: the new binary boots with the old registry, then the rewritten registry is applied. In that window, the new default makes the failing queries unavailable, and that is the reason the new default helps. Write that order, and state the window. Then say what the strict setting does: today a breakage blocks boot (crates/omnigraph-server/src/queries.rs:204-208), so a strict operator has no served path. That operator must apply the rewritten registry with the new binary to the stopped root (cluster apply --target ROOT) before the server starts. Also say whether the new default applies only at the break or always. If always, it also replaces the boot check for schema drift that queries.rs:204-208 describes ("surfacing schema drift at the deploy boundary"). In that case, cluster apply should still refuse a registry that fails, so that the deploy boundary keeps the check.
  • O4. Say where the budget ceiling is checked. lint and cluster validate run offline and do not know the server's setting. So a stored query can pass validation and then be refused at boot or apply. Say that the ceiling is checked at apply and at boot, with the deployment's setting. Also say what happens to a stored query when an operator lowers the ceiling below its declared budget. With the new default (O3), it becomes unavailable.
  • O5. Algorithm contract: two small points. "Output. One row per node of the population" (:2333) and "an edge with an endpoint outside the population is dropped" (:2322-2323) together define the subgraph, which is good. But say what happens at a node with no out-edges (a dangling node): its rank goes nowhere unless the rule spreads it. Say how dangling nodes are handled (spread evenly, or to the personalization vector), because systems differ and the oracle (:2356-2359) must use the same rule. Also, #1334 (a reference interpreter of the core IR, draft) is the natural owner of the oracle rows. Name it, or name the crate that will own them.

Still open from earlier reviews

This delta does not touch these lines, so I checked them by reading the head text:

  • B1 of the review on 44010cc (plain @index on a String changes meaning with no refusal and no respelling row): the String rule (:963-966, :3012), the respelling map and Compatibility (:2528 and :2542 still call the schema change "additive") are not in this delta.
  • B1 of the review on 0c7f88c, and O1 to O4 of the review on ddc3b55: the lines are not in this delta.

Tradeoffs and liability

  • Removed. Positional union, which turned two swapped columns of one type into silent wrong rows. A T86 text that would go stale when its stage ships. A clamped budget, which would hide a client's request. A whole graph lost to one stored query that fails at the break.
  • Added. A contract for a family of registry rows, with eight clauses that each new algorithm must meet. A server setting (the ceiling) that changes which stored queries are accepted, so a stored query's acceptance now depends on the deployment (O4). A second availability state for a stored query, which queries list and the health route must report (O3).
  • After five more changes of this kind. Five more algorithms, each a registry row under one contract, scale well. That is better than five operators. But each row repeats the edge-column rule, so B1 is cheaper to fix now than after the first row ships. A positional rule copied into five rows is five places to fix.

Validation

  • Exact head 7bfff4d, previous reviewed head 44010cc. I reviewed in a clean detached worktree. The delta is RFC text only.
  • python3 scripts/check-docs.py with the pinned scripts/requirements-docs.txt: "Documentation OK (473 Markdown files checked)". scripts/check-agents-md.sh: OK. git diff --check 44010cc3 7bfff4db: clean.
  • Grammar claims (B2, O1): a scratch crate with pest and pest_derive 2.8.7 and #1316's crates/omnigraph-compiler/src/query/syntax/grammar.pest at a514157, parsing with query_file, and printing the rules each text matched. Results: the RFC example matches exp_float_rec at 1e-9, and the same text with 1.0e-9 matches float_lit. null in if and coalesce matches kw_null under reserved_property. null as an assignment value matches null_value. The parser's handling of those rules I read in syntax/parse.rs at a514157. I did not build #1316.
  • Today's code claims: read at 7bfff4d (query.pest, omnigraph-server/src/queries.rs). #1316 also still checks union arms by position (check/mod.rs:3637-3650, test union_agrees_by_position_and_widens_nullability). That is expected until it follows this decision.
  • I did not open the 24 cited sources. The algorithm facts I checked are standard (min-label propagation over both directions, the round count, the PageRank limit, Brandes's two passes). I found no error in them.
  • No Decision needed comments on this PR.
  • GQT regression rule: this delta changes no behavior on main, so no regression applies.
  • CI at the exact head 7bfff4d when checked: 23 passed, 3 pending, 4 skipped.
  • Other open PRs: #1316 is the implementation (B2, O1). #1334 is the reference interpreter (O5). No other open PR touches these sections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant