Repository navigation
Conversation
Phase 0 of the GQ 3 RFC (draft): the compiler's half of edition 3. The one front door, parse_query, reads the edition line: no line or `gq 2;` is edition 2, unchanged; `gq 3;` selects the edition-3 parser, whose refusals name the construct and carry byte-range suggestions; any other line is Q006 with `gq 3;` as the fix. An edition-3 declaration is typed into the core IR (ir::core3) for every form of sections 2 to 4 and 7, recursion typed to Fix, and the planner refuses it with P006 (plan_edition3) at each engine door before a snapshot is captured. Nothing in edition 3 executes. Tests: the rule-set pin, the red-team corpus with pinned repaired trees (4 listed mismatches), the parse-print-parse property (2,000 cases, and 100,000 as a hunt), the desugaring-uniqueness test and the closure table, the before-and-after asset type-checked, edition-2 texts lowered to the same IR as before, a zero-I/O engine test, and one GQT case per code. The GQT runner keeps a step the compiler refuses at parse under `expect error:` and renders parse diagnostics with their code.
ragnorc
left a comment
There was a problem hiding this comment.
Review of #1316 at 81d69a1d
Recommendation: revise one CI gap before this leaves draft. I found 1 blocking item and 4 optional items. The edition-2 path is unchanged in every test I ran. The engine doors refuse edition 3 before they read a snapshot.
Reviewed head: 81d69a1d (two commits on main 3f249cc0). I reviewed 2c3c7a5b first. 81d69a1d came in during the review, and I reviewed that delta too. No other review or comment covers either head.
Why, what, how, impact
Why. RFC #1302 (head d0155e50) needs a compiler front end for edition 3 before any planner work can start.
What. A gq 3; line selects a new parser and type checker, which produce the typed core IR (ir/core3.rs). The planner refuses every edition-3 declaration with P006. A source with no edition line, or with gq 2;, takes the old edition-2 path.
How. parse_query_diagnostic reads the edition line first. QueryDecl::edition3 carries the edition-3 tree, and QueryDecl::is_mutation() now classifies a declaration for the server, lint and GQT. Omnigraph::refuse_edition3 runs at the four Session read doors before capture_read_view. The mutation, catalogue and compile paths each keep a backstop.
Impact. Edition 2 sees one extra trivia scan per source, plus edition_near_miss. No edition-2 text starts with gq, so no edition-2 result changes. The corpus walk (more than 2,500 texts) shows this. Edition 3 executes nothing.
Blocking
B1. A docs-only pull request can break edition_corpus without any CI job running it.
81d69a1d correctly fixes the first head's failure. At 2c3c7a5b, scripts/check-change-classes.py refused 7 compiler-test literals that read crates/omnigraph-gqt/cases (exit 1). At 81d69a1d it passes ("reader closure holds ... 5 listed readers"). The tests now live in the crate that owns the corpus, and gq-logic-tests.yml:223 runs them. A cases-only pull request now runs them too.
But edition2_corpus() (crates/omnigraph-gqt/tests/integration/edition_corpus.rs:95-120) also reads every fenced gq block in docs/**/*.md, except docs/rfcs. The classifier arm docs/*.md|...) ;; in ci.yml sets no run flag for those files. Only docs/user/queries/catalogue.md runs CI. So the class is no longer closed. The same rule applies here as for the corpus: "a class is sound only while the set of jobs reading its paths is closed" (check-change-classes.py docstring; docs/dev/ci.md, Changing CI rule 3). The sweep cannot see this reader, because it builds the path at runtime (root().join("docs")). The sweep's docstring makes such a reader the reviewer's to list.
Demonstration at 81d69a1d. I added this example to docs/user/queries/editions.md, the page this PR adds, and it is the obvious next edit to that page:
```gq
gq 3;
query people() { match { $p: Person } return { $p.name } }
```
cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --test integration -- edition_corpus:every_edition2_text_in_the_repository_parses_as_beforeFAILED (assertion left == right failed: docs/user/queries/editions.md#1). The other 3 passed. Without the edit, all 4 pass.- The
Classify changed pathsscript fromci.yml, replayed withchanged=(docs/user/queries/editions.md):run_full_ci=false,run_gqt=false,run_deployment=false.
So the pull request merges green and main goes red at the next full run. Other docs examples have the same effect, by code reading: a block that edition 3 refuses with Q018 changes the pinned bare_names list, and a block whose refusal code has no rewriter row fills unexpected.
Options, smallest first:
- Walk only the
.gqfiles underdocs/. A change to one of them already runs full CI, because it falls to the*arm. The fenced blocks stop being corpus input. - Keep the fenced blocks and close the class. Make a changed
docs/*.mdthat holds agqfence setrun_gqt=true, and add a replay fixture for it tocheck-change-classes.py(for example "docs with a gq fence" →('false', 'true', 'false')). Nameedition_corpus.rsas a docs reader beside it. This keeps the coverage, but it adds a content check to the classifier.
Option 1 removes the liability. Option 2 keeps the evidence, at the cost of one more classifier rule to maintain. GQT cannot express this contract. The owner of the regression is the classifier replay in check-change-classes.py.
Optional
O1. lint and the stored-query validators disagree about one declaration. For a valid edition-3 declaration, lint_query_file reports status: error, with the uncoded Plan message from descriptor.rs:94 ("its operation descriptor waits for the planner"). cluster validate (omnigraph-cluster/src/config.rs:1413), the server registry check (omnigraph-server/src/queries.rs:265) and queries validate call only typecheck_query_decl, which returns Ok(Edition3). So a cluster can publish a stored query that lint refuses and that can only ever answer P006. It also appears in the MCP catalog. I suggest one answer for "can this release serve this declaration": P006, with its code, from lint and from registry validation alike. If you intend to register edition-3 stored queries in the preview, say so in editions.md, and make lint agree.
O2. The corpus tests pin exact counts over a corpus that grows. the_edition2_corpus_type_checks_alike_in_both_editions pins "edition 2 ok / edition 3 T65": 156 and eight more counts. the_edition2_corpus_read_as_edition_3 pins the four-file bare_names list. I copied issue_640_search_score_projection.gqt to a new case at 2c3c7a5b: T65 went from 156 to 164, the bare_names list gained the new file, and both tests failed. The same pins moved unchanged to 81d69a1d (I compared the assertions). After 81d69a1d, the failure lands in the case author's pull request, which is the right place. But every future GQT case with a ranking in order, or a Q018 shape, must also edit edition_corpus.rs. Five more search cases mean five edits there. Pinning which rules occur, plus a lower bound, keeps the RFC evidence and removes that coupling. So does holding the exact counts on a frozen copy of the corpus.
O3. One concept is checked in five places. "An edition-3 declaration does not reach an edition-2 lowerer" is checked at refuse_edition3 (four doors), compile_named_query (query_doors.rs:418), the two mutation paths (mutation.rs:1092, :1590) and catalogue.rs:117. The pre-snapshot door check is needed for the zero-I/O contract. The backstops all repeat if decl.edition3.is_some() { return Err(edition3_refusal(..)) } before an edition-2 check. One engine function that returns the edition-2 checked query, or the P006 refusal, would make a new door correct by construction. The planner already depends on the compiler, so the engine is the right owner.
O4. The door checks against the handle's catalog, not the read target's. refuse_edition3 uses self.catalog() before capture_historical_read_view(version). For an old snapshot this can answer a T code where the target's schema would answer P006. This is harmless while edition 3 only refuses, and docs/dev/execution.md states it. When a later stage executes edition 3, the check must move behind view capture (invariant 3, one coherent accepted view).
Small things: edition_near_miss skips a set line with rest.find(';'), so a ; inside a string or comment ends the line early. edition_line and edition_near_miss also treat whitespace differently between gq and the number (skip_trivia against trim_start). Neither changes an edition-2 result.
Tradeoffs and liability
This adds about 24,700 lines: about 11,000 of compiler and planner code (parser3.rs and typeck3.rs hold 8,450 of them), about 5,300 of Rust tests, and the rest test corpora. The language executes nothing yet. The edition-2 cost is low: one prefix scan per source, and the equality walk over the corpus pins it. The real liability is a second grammar and type checker that must follow every RFC revision until edition 3 runs. #1244 also proposes an additive alternative without a language break. Keeping this a draft until #1302 is accepted is the correct gate. Merging earlier would turn RFC edits into code edits. QueryDecl::is_mutation() replacing the scattered mutations.is_empty() checks removes liability, and it is worth keeping whatever happens to edition 3.
Validation
All in a clean detached worktree. I restored every temporary edit.
cargo test -p omnigraph-compiler --lib --lockedat2c3c7a5b: 560 passed, 3 ignored (the description says 561).cargo test -p omnigraph-planner --lib --locked -- edition3: 1 passed.- At
81d69a1d:cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --test integration -- edition_corpus: 4 passed.--test gq_logic_tests -- gq3: 69 passed. python3 scripts/check-change-classes.py: exit 1 at2c3c7a5b(7 FAIL lines), exit 0 at81d69a1d.- B1 and O2 demonstrations: as described above.
- Exact-head CI at
81d69a1d, when I posted: 27 pass, 4 skipped, 0 fail, includingTest Workspaceand the GQT jobs. At2c3c7a5b,Check Workflow Action Pinsfailed on the class sweep that81d69a1dfixes. - Not run locally: the engine test
warm_read_cost::an_edition3_declaration_is_refused_before_any_storage_access. It needs the engine's test feature graph, so the evidence for it is the passingTest Workspacerun at this head. No performance claims. The change reads no Lance storage, so no Lance domain applies.
…est to its code Engine v1, its frozen pin, the session's read-executor seam it ran through, the compiler's untyped read IR and GQT's `expect same as v1` section go; every case keeps its pinned rows. The red-team corpus and the RFC asset copies leave the tree: each distinct refusal shape the corpus found is a repair row in the parser or checker tests, and the RFC's examples are type-checked inline. The release note, the release.json retarget and the user page are withdrawn until the RFC is accepted.
ragnorc
left a comment
There was a problem hiding this comment.
Delta review of #1316 at f6946f58
Recommendation: keep as draft. B1 from my review at 81d69a1d is still open, by a different path. I also recommend two scope changes. The new commits are correct as far as I tested them. The v1 retirement removes real liability, and every step that compared with v1 still pins its rows.
Reviewed head: f6946f58. My last review covered 81d69a1d (review). Since then:
b4ab1564and562a3bcamergemain. The only conflict wasCargo.lock(the deleted crate), andgit show --remerge-diff 562a3bcashows nothing else.changelog.d/release.jsonat the head is the same as onmain.75cb05fcis the release-note retarget.ec18d6fawithdraws it again with the note and the user page.ec18d6faretires engine v1 and moves the red-team corpus and the RFC asset copies out of the tree (474 files, +1,431/−14,978).f6946f58removes unused imports. It also removes the#[cfg(feature = "test-util")]thatec18d6faleft abovepub mod retentioninomnigraph-catalog/src/lib.rs. Squash merge makes that intermediate state harmless.
No other review or comment covers these commits.
Why, what, how, impact
Why. The draft RFC #1302 (now at 88cb905d) puts "One IR" in its blocked_on list. It also says the red-team corpus and the competence instrument stay outside the repository.
What. This change deletes crates/omnigraph-reference-engine (about 6,700 lines with its frozen pin), the ReadExecutor seam in Session and Detached, the engine's test-util feature, ir/untyped.rs and ir/erase.rs (1,140 lines), and GQT's --- expect same as v1 section. It also deletes 397 red-team corpus files and parser3_redteam.rs. parser3_tests.rs gets 95 parse repair rows and typeck3_tests.rs gets 5 type-level rows instead.
How. The GQT parser now refuses --- expect same as v1 as an unknown expect mode (omnigraph-gqt-core/src/lib.rs:1077), so a leftover section fails loudly. forbidden_apis.rs loses the v1 guards. split_crate_test_util_is_enabled_only_by_dev_dependencies stays, and it is now stricter: no regular dependency may enable any test-util.
Impact. Production builds do not change. GQT loses its independent second engine. The cost is low: I checked all 34 steps that carried expect same as v1 (15 case files) at ec18d6fa^, and each one also has its own expect rows, unordered or ordered section. So each step still pins its answer. What goes is the second opinion when someone re-pins rows. No other open PR adds the section or uses the removed APIs. I scanned every open PR's diff, and only RFC prose in #1302 mentions it.
Blocking
B1 (still open). A docs-only pull request can still break edition_corpus without any CI job running it.
corpus() now keeps only texts with no edition line (edition_corpus.rs:141). This closes the gq 3; example I used last time. I re-ran it at f6946f58, and all 4 tests pass. But corpus() still reads every fenced gq block in docs/**/*.md outside docs/rfcs (edition_corpus.rs:115-140). the_edition2_corpus_read_as_edition_3 then pins the exact bare_names list (:266-273) and requires unexpected to be empty. The docs/*.md arm in ci.yml:103 still sets no run flag. So the class is still not closed.
Demonstration at f6946f58. I added a T45 example to docs/user/queries/diagnostics.md. It is the kind of example a diagnostics page shows, and it is the same text as a step in expression_type_errors.gqt:
```gq
query property_leaf_in_an_inline_match() {
match { $p: Person { adult: age > 30 } }
return { $p.slug }
}
```
cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --test integration -- edition_corpus:the_edition2_corpus_read_as_edition_3FAILED.bare_namesgaineddocs/user/queries/diagnostics.md#…. The other 3 tests passed. Without the edit, all 4 pass.- The
ci.ymlclassifier, replayed withchanged=(docs/user/queries/diagnostics.md):run_full_ci=false run_gqt=false run_deployment=false.
A bare-name example (Q018) does this, and so does any block that edition 3 refuses with a code that has no rewriter row. The two options from my last review still apply. Option 1 (walk only the .gq files under docs/, because a change to them already runs full CI) removes the liability. Option 2 (a docs/*.md change that contains a gq fence sets run_gqt=true, with a replay fixture in check-change-classes.py) keeps the coverage but adds a classifier rule. The owner of the regression is still the classifier replay. GQT cannot express this contract.
Recommended (not blocking)
S1. Move the v1 retirement into its own pull request against main.
The retirement does not depend on edition 3. v1 read ir::untyped, which erase.rs built from today's typed pipeline IR. Before ec18d6fa, no edition-3 file named erase, untyped, ReadExecutor or the reference engine (git grep at ec18d6fa^). Each comparison already had pinned rows. So the change is complete on its own, and on its own it removes liability now: a 6,700-line frozen engine compiled into every GQT build, a cfg(feature = "test-util") field threaded through Session and Detached, and an allowlist in forbidden_apis.rs.
In this draft it waits for RFC acceptance, which this PR correctly holds as its gate. If the RFC changes direction (for example toward #1244's additive path), someone must extract it later. A revert of the preview would also bring v1 back. A separate pull request is also easier to review. It deletes a shipped test-format section and a crate with CODEOWNERS, and that is a different risk from adding a language. The title "parse and type-check GQ edition 3" no longer covers it either.
S2. The PR and RFC #1302 now disagree about One IR. One of them should change.
RFC #1302 at 88cb905d, section "One IR", and its blocked_on item 4 say two things:
- "Today's pipeline IR is deleted when the planner reads the core, and the frozen v1 reference engine with it: v1 reads that IR." This PR deletes v1 while the planner still reads the pipeline IR, and v1 never read that IR (see S1). The PR shows that the coupling in the RFC is not real. I suggest that the RFC say v1 goes independently, because every comparison sits beside pinned rows.
- "No module carries an edition number in its name." This PR adds
ir/core3.rs,query/ast3.rs,parser3.rs,typeck3.rs,print3.rs,habits3.rs,desugar3_tests.rsandomnigraph-planner/src/edition3.rs, which is about 15,500 lines (10,800 of them outside tests) under edition-numbered names. Under the RFC's end state these modules become the single front end, and the edition-2 parser is "the one edition-specific part". Renaming is cheapest now, while the PR is a draft and nothing else imports them. Give the new modules role names, and give the edition-2 parser the edition-specific name. If you want the numbered names during the preview, the RFC should say so and name when they go.
Earlier items
O1 (lint and the registry validators disagree on a valid edition-3 declaration), O2 (exact count pins), O3 (the edition-3 check in five places) and O4 (the door checks the handle's catalog) are unchanged. The PR's own commits since 81d69a1d do not touch those files. The pins in O2 are identical (T65 156 and the four-file bare_names list).
Small things:
cases/v2/planner/hydrate_return_columns_above_the_limit.gqt:12still says "main is compared with engine v1".crates/omnigraph/tests/integration/engine_v2.rs:1says "differs from the frozen engine v1 by design". "The retired engine v1" would keep it true.
Tradeoffs and liability
The delta removes about 13,500 lines net, and most of that is real liability: a frozen second engine, a test-only seam in production types, a second read IR, and 397 corpus files with two pinned lists. The new repair rows are the right replacement. One row per distinct refusal shape, each suggestion applied and reparsed, tests behaviour rather than a corpus snapshot. The four known mismatches were design decisions, and the one that a test can express ($a not $b) is a row at parser3_tests.rs:1000. Dropping the 218 "accept" texts loses positive parse coverage. The property test (2,000 generated declarations) and the RFC examples, which are now type-checked inline, cover most of it. The PR still adds about 10,800 lines of front end that executes nothing, and the liability from my last review is unchanged. S2 is about making that code the one front end the RFC describes, not a second one.
Validation
All in a clean detached worktree at f6946f58. I restored every temporary edit (git status clean).
cargo test -p omnigraph-compiler --lib --locked: 547 passed, 0 ignored. This matches the description.cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --test integration -- edition_corpus: 4 passed. With the B1 docs edit: 3 passed, 1 failed, as described. With last review'sgq 3;block: 4 passed. With a T44count(age)mutation example: 4 passed. I found the Q018 texts by temporarily printing them in the test.python3 scripts/check-change-classes.py: ok ("reader closure holds ... 5 listed readers"). Theci.ymlclassifier, replayed for the docs path: no job.scripts/check-docs.py(pinnedrequirements-docs.txt, Python 3.12): "Documentation OK (481 Markdown files checked)".scripts/check-agents-md.sh: OK.- A script over
ec18d6fa^checked that all 34expect same as v1steps pin rows. I also scanned every open PR's diff for the removed section and APIs. - Exact-head CI at
f6946f58when I posted: 28 pass, 4 skipped, 0 fail. - No GQT regression applies. This delta deletes code and fixes no behavioural bug. Not run locally: the full GQT corpus and the engine tests (
forbidden_apis,warm_read_cost). The evidence for them is the passing exact-head CI. No performance claims. The change reads no Lance storage, so no Lance domain applies.
The edition-3 parser, tree, printer, foreign-habit table and checker move under query/syntax and query/check, the core IR to ir/core.rs and the planner's refusal to core_plan.rs; edition-2 parsing reads through syntax/edition_window and the front door stays query/parser.rs. Types and functions drop their edition suffixes (SourceFile, Expr, check_file, plan_core, CheckedQuery::Core, QueryDecl::syntax). Tests sit in tests.rs beside each module, and the diagnostics cases move to cases/diagnostics. No behaviour changes.
# Conflicts: # crates/omnigraph-compiler/src/query/codes.rs # crates/omnigraph-compiler/src/query/parser.rs
…it arithmetic IRExpr, its block predicate and their tests move from ir/mod.rs to ir/expr.rs. The expression gains the nodes the core IR needs and has no other spelling for: BinaryOp::Arith over one numeric type, a unary Neg, and a typed scalar registry Call (if, coalesce, concat, match_terms), with local type checks, GQ rendering at the right precedence and a can_fail walk. The bound-plan mirror reads and writes them as new variants, so every existing plan serializes as before. The engine evaluates them per batch. An integer overflow, a zero divisor and a non-finite float refuse the row with execution diagnostics E1, E2 and E3; a null operand gives a null row. Arithmetic is never folded or pushed into a scan, so its failure keeps its place. Nothing produces these nodes yet.
The core's terms, aggregates, scores, call bindings and sink assignments store IRExpr instead of their own scalar type. The checker keeps its typed working form private (query/check/scalar.rs) and converts it as it builds each term: a reference column becomes a Variable, a value column an AliasRef, a property or meta field a PropAccess, membership a Contains comparison, and the registry rows their own nodes. ExprType gains Edge so an edge reference column has a type. No behaviour changes; edition 3 still stops at P006.
ragnorc
left a comment
There was a problem hiding this comment.
Delta review of #1316 at a14fe7ba
Recommendation: keep as draft. The rename answers S2 from my last review, and moving the core IR onto IRExpr is the right step toward "One IR". But the new arithmetic evaluator breaks its own documented contract in two cases, and nothing yet enforces the guard rule that the RFC and the code comments promise (B2). B1 is still open, unchanged.
Reviewed head: a14fe7ba. My last review covered f6946f58 (review). Since then:
f7ec10b1gives the edition-3 modules role names (query/syntax,query/check,ir/core.rs,planner/src/core_plan.rs) and moves the edition-2 parser tosyntax/edition_window. No tracked.rsfile has an edition number in its name now.dc95d4aachecks the RFC's recursive example as the RFC writes it.64e727bcmergesmain. I checked the conflict resolution withgit show --remerge-diff. Main'sQ006(nesting depth) keeps its number, and the PR's unreleased "unknown edition" code becomesQ022.check_query_nestingstill runs before the edition dispatch, so it covers edition 3 as well.3ce87e25movesIRExprtoir/expr.rsand addsBinaryOp::Arith,Negand a typed registryCall(if,coalesce,concat,match_terms). It also adds mirror variants,E1–E3and an engine evaluator (engine/arith.rs). The commit says "Nothing produces these nodes yet."a14fe7bastoresIRExprin every core term. The checker keeps a private working form (query/check/scalar.rs) and converts it withSExpr::to_ir. It also addsExprType::Edge.
No other review or comment covers these commits.
Why, what, how, impact
Why. RFC #1302 lists "One IR" in blocked_on: both editions lower into one core, with one scalar expression language.
What. The core IR no longer has its own scalar type. Filters, projections, group keys, order keys, aggregate arguments, score queries, call bindings and assignments now hold the same IRExpr that the pipeline IR uses. IRExpr gains the arithmetic and registry nodes the core needs. The engine can evaluate them per batch.
How. Membership becomes Contains with the list on the left, which is how edition 2 already spells it. A meta field becomes a PropAccess of its @ name, as in edition 2. fold.rs never folds arithmetic, is_constant never treats it as constant, and typed_expr_to_df_expr never pushes it into a scan.
Impact. No behaviour change today: edition 3 still stops at P006, and old bound plans serialize as before. But the engine now has about 300 lines of evaluator that no plan reaches, with semantics the RFC does not yet fully specify.
Blocking
B1 (still open, unchanged). A docs-only pull request can still break edition_corpus without any CI job running it.
f7ec10b1 renames the tests and the import paths, but corpus() still reads every fenced gq block in docs/**/*.md outside docs/rfcs. the_corpus_read_under_the_edition_line still pins bare_names and requires unexpected to be empty. ci.yml and scripts/check-change-classes.py do not change in this delta, so the docs/*.md arm still sets no run flag. I did not run the demonstration again because none of the code it depends on changed. The two options from my earlier reviews still apply.
B2 (new). A failing arithmetic term is evaluated on rows where it should not run.
The module doc of engine/arith.rs says: "Only valid slots are computed, so a null operand gives a null result and is never refused." RFC #1302 says a function that can fail "keeps its failure where it was written; the planner may not move it past a guard (A3)". The code comment on is_constant says the same. The evaluator does not do this in three cases:
null / 0is refused withE2.binary(engine/arith.rs:83) checkshas_zero(right)before the division, over the divisor alone. A row with a null dividend and a zero divisor is refused, not null. This breaks the module's own contract.ifevaluates both branches on every row.evaluate_expr(engine/expr.rs:301) evaluates allCallarguments over the whole batch, thencallzips them. Soif($p.x = 0, 0, 10 / $p.x)is refused withE2on the row it was written to protect. This is the standard guard, and it is the reasonifexists besidecan_failarithmetic. The same applies tocoalesce(…)with a failing later argument.and/orevaluate both sides.evaluate_boolean(engine/expr.rs:165) computes both operands over the whole batch before the Kleene combination. So$p.x != 0 and 10 / $p.x > 1is refused. The RFC does not say whetherandis a guard. SQL does not promise short-circuit evaluation either, so this one needs a decision more than a fix.
Also, nothing outside tests reads IRExpr::can_fail (ir/expr.rs:323), so the planner does not "keep such a term behind its guards" yet, although the doc comment says it does. If the optimizer merges two filter clauses into one conjunction, case 3 applies to clauses the user wrote apart.
Demonstration at a14fe7ba. GQT cannot express this, because no edition-3 query reaches the planner (P006). So I used the cheapest existing owner, the engine::arith::tests module. I added three temporary tests (x = [0, 5], column p.x):
binary(Div, [null], [0]): expected a null row; gotErr("E2").evaluate_exprofif($p.x = 0, 0, 10 / $p.x): expected[0, 2]; gotErr("E2").evaluate_filterof$p.x != 0 and 10 / $p.x > 1: expected[false, true]; gotErr("E2").
All three failed for that reason. The four existing arith tests passed. Then I made two small temporary changes to show that fixes of this kind work:
- Check for a zero divisor only where the dividend is present (
zeros AND is_not_null(left)). - For
if, evaluate the condition, filter the batch by the mask and by its negation, evaluate each branch only on its own rows, andinterleavethe results.
With those changes, tests 1 and 2 passed, and the four existing tests still passed. Test 3 still failed, as expected, because I did not change and. I then restored both files.
The cause is one missing rule: "a can-fail term runs only on the rows that reach it". The evaluator is the layer that owns it. There are two coherent options:
- Option 1 (less liability now). Remove the engine evaluator (
engine/arith.rs, the newevaluate_exprarms,E1–E3) from this PR, and keep the IR nodes, the mirror and the checker conversion. Add the evaluator with the first planner rule that produces these nodes, together with GQT cases that reach it. This also avoids publishingEcodes before the RFC fixes their meaning, becausecodes.rsfreezes a code once it ships. - Option 2. Keep the evaluator, and do three things: (a) state in the RFC which constructs are guards (
ifandcoalesceat least, and say yes or no forand/or); (b) evaluate selectively, as in the sketch; (c) make the planner readcan_failbefore it reorders or merges a filter, with a test for each.
Recommended (not blocking)
S3. ExprType::Edge puts a type list in a string. types.rs:322 says type_name "may hold the alternatives of an edge selector, joined by |". The checker builds it with .join("|") (check/mod.rs:1573), in member order. This was private to the core before. Now it is in the shared ExprType that the planner and the engine match on, and ExprType equality compares the joined strings. Invariant 9 asks for typed structure here. Today only refusals read it, so this is the cheapest time to change it, for example to a sorted Vec<String> or the resolved EdgeSelection.
S4. Two copies of can_fail. SExpr still computes and stores can_fail (check/scalar.rs:20), but nothing reads it after to_ir. IRExpr::can_fail is the copy that is used. Remove the field so the two rules cannot drift.
S5. match_terms' mode goes through a string. The checker turns mode: all|any into a bool, then into a Literal::String("all"|"any") option, and to_ir matches it back, with anything else falling back to Any (check/scalar.rs:263). So the same mode has three spellings: LexicalQuery.mode_all: bool, TermsMode, and the mirror's all: bool. Today this is correct because the checker writes only those two strings. It is cheaper to store TermsMode in SKind::Call (and to resolve the function name to ScalarFunc in the checker). Then to_ir is total, without the unreachable! on an unknown name, and a new registry row touches one place fewer.
Earlier items
- S2 is resolved for module names. If RFC #1302 still says that v1 retirement waits for the planner to read the core, it should change (S2, point 1).
- S1 (move the v1 retirement into its own PR) is unchanged.
- O1–O4 are unchanged. This delta does not touch those owners except for renames.
- These two small comments are still stale:
cases/v2/planner/hydrate_return_columns_above_the_limit.gqt:12("main is compared with engine v1") andcrates/omnigraph/tests/integration/engine_v2.rs:1("the frozen engine v1"). - Open PR #1101 edits
query/parser.rs, which this PR moves intosyntax/edition_window, so expect a rebase conflict there. ItsT60is still unallocated here.
Tradeoffs and liability
The rename and a14fe7ba remove liability. There is now one stored scalar language instead of two, membership and meta fields have the edition-2 spelling, and the rename makes the edition-specific part look edition-specific. The private SExpr is a fair cost: it carries reference nullability that ExprType does not have, and the code never stores it.
3ce87e25 adds the opposite kind of liability. It is engine code with no producer, so no end-to-end test can pin it, and its semantics are not yet decided. It also adds E codes that freeze on release. What does this look like after five more registry rows? Each row now touches the checker (by string name), call_to_ir, ScalarFunc, ScalarFuncMirror, typed.rs, the optimizer walks and the engine call. That is acceptable for typed rows. But eager evaluation would apply to each of them, and can_fail would still be advisory. Fix the evaluation rule once, at the evaluator, before the first producer lands, or wait until then to add the evaluator.
Validation
All runs were in a clean detached worktree at a14fe7ba. I restored every temporary edit (git status clean).
cargo test -p omnigraph-compiler --lib --locked: 552 passed, 0 failed.cargo test -p omnigraph-engine --lib --locked engine::arith: 4 passed at the head. With the three B2 tests: 3 failed withE2. With the two sketch fixes: 6 passed and 1 failed (theandcase, which I did not change).- Merge check:
git show --remerge-diff 64e727bc(codes and parser conflicts, as described above). - B1: code inspection of
edition_corpus.rs,ci.ymlandcheck-change-classes.py(no change sincef6946f58). I did not run it again. - Exact-head CI at
a14fe7ba: 26 pass, 4 skipped, 2 fail. The two failures (Azurite Azure Integration,RustFS S3 Integration (default), 403 instead of 200 ins3::server_opens_azure_graph_directly_and_serves_snapshot_and_read) also fail onmainatc61f242e(run). They come fromc8dcb315(deny reads without a policy), and #1332 addresses them. They do not come from this PR. - Not run locally: GQT and the engine integration tests. A Cargo.lock bump on
main(arrow 58.4) made a second cold build too expensive for this run. The evidence for them is the passing exact-head CI. No performance claims. The delta reads no Lance storage, so no Lance domain applies.
GQ 3 replaces the language whole (docs/rfcs/2026-11-07-gq-3.md, sections 6 and 11): one grammar with plain rule names, one type checker, one IR. The edition line, the old parser, AST, checker and lowering, and the pipeline IR are gone. The type checker binds every expression (physical system columns, the casts `coerce` chooses, folded literals) and infers result schemas from the core. The planner reads only the core through the read shape, which lowers the pattern, its projection, grouping, order and cut, and the `from nearest/bm25/rrf` rankings to today's ranked scans; the engine lowers write groups to today's staged writes. Any other core node is P006. Every header consumer checks through the core. The repository's query text was rewritten by the one-shot rewriter; the seventeen fusion cases the RFC lists were edited by hand, and the cross-version journeys keep predecessor syntax for predecessor binaries. Five diagnostics improve: SQL `union` names the relation function, T74 names the return item, T8 prints the block form, T65 carries an applicable edit, and type errors carry the position of their clause.
ragnorc
left a comment
There was a problem hiding this comment.
Delta review of #1316 at ae9c2a42
Recommendation: keep as draft. The break is the right shape: one grammar, one checker, and a planner that reads only the core. But seven defects reach users, and five of them return wrong rows or wrong effects with no error. Each one has a GQT reproduction at this head. Also, the new commit ae9c2a42 makes CI pass by weakening a contract test, not by fixing the served runner.
Reviewed head: ae9c2a42. My last review covered a14fe7ba (review). Since then:
6d40bc42replaces the language in one break (519 files, +9,677/−21,742). It deletes the edition-2 parser, AST, checker, lowering and the pipeline IR. It addsplanner/src/read_shape.rs, which lowers the core to today's read plan, andexec/write_plan.rs, which lowers write groups to today's staged writes. The one-shot rewriter rewrote the repository's query text.ae9c2a42changesboolean_literal_refused_where_a_property_is_named_false.gqtfromexpect error: Q008toexpect error: is not a statement.
No other review or comment covers these commits.
Why, what, how, impact
Why. RFC #1302 (sections 6 and 11) makes GQ 3 the only language and asks for "One IR": the planner and the engine read the checked core, not a second pipeline form.
What. Edition 2 text no longer compiles. Reads lower through ReadShape::of (core → scans, expansions, filters, blocks, return, order, cut, ranked scans). Writes lower through WritePlan::of (groups → staged inserts, updates and deletes). Any core node outside those tables is P006.
How. The checker binds every expression (physical columns, casts, folded literals). The planner's acceptance now takes its requirements from the same ReadShape. The GQT corpus, docs, skill and benchmarks were rewritten. 17 fusion cases were edited by hand.
Impact. Every stored query must be rewritten before upgrade. The rewritten corpus passes: 536 of 536 GQT cases pass in process here. The problems are in the forms that are new in GQ 3: write groups, scoped edge bindings, if, rrf populations, parameter defaults. The rewritten corpus does not exercise these forms, so CI does not see them.
Blocking
Each item has a temporary GQT case that I ran at this head with cargo test -p omnigraph-gqt --locked --test gq_logic_tests <name>. The PR contains no fix for any of them, so there is no "after fix" run. See Limitations.
B3. A write group's statements do not share one selection. WritePlan::of copies the group's filter into each update and delete (exec/write_plan.rs:96, :105). execute_update then re-runs that filter over committed plus pending rows (exec/mutation.rs:1632-1684). The RFC (section 3) and ir/core.rs say a group's clauses are evaluated once and every sink sees that one snapshot.
query publish() {
$x: Doc
$x.status = "draft"
update $x set { status: "published" }
update $x set { n: 1 }
}
- Two drafts and one published doc. Expected: both drafts get
status = "published"andn = 1. - Actual:
nis not set on either draft. The second update matches 0 rows and the mutation reports success. - A per-row
inserttakes its count from the last statement (matched,mutation.rs:1684), soupdate; update; insertinserts 0 times. - An insert between two updates can add rows that the second update then selects.
- Cause: the group selection is not a value the executor holds. Fix it in the executor, which owns group execution: select the group's rows once (ids, on the group's snapshot), then apply every statement to those ids. Then the filter is not repeated and is not re-evaluated against pending rows.
- The rewriter puts each edition-2 statement in its own group, so rewritten text is not affected. No case has two
update/deletestatements in one group.
B4. $e.@type resolves by name across the whole declaration. Binder::collect (check/bind.rs:74-100) builds one name → edge-types map over every scope: blocks, groups, inline reads. A later binding overwrites an earlier one. expr() (bind.rs:222-228) then folds $e.@type to a literal when the map says one type.
match { $p: Person exists { $p $e:knows $x } $p $e:(knows | likes) $f }
return { $f.name as f, $e.@type as kind }
- Expected:
{b, Knows},{c, Likes}. Actual:{b, Knows},{c, Knows}, which is silently wrong. - In a write group, the same leak can fold a
$e.@type = "Likes"filter to false, so a delete deletes nothing. - Cause: the binder keeps scope-free state that the checker already has with scope. Fold
@typein the checker, where the binding's scope is known, and delete the map.
B5. Filters in an rrf arm's population are dropped. fused (read_shape.rs:672-690) keeps each arm's ranking key and discards its population. The Fuse then lowers to a scan of the whole type.
let $en = { match { $c: Chunk $c.lang = "en" } return { $c } }
from rrf(nearest($en, $c.embedding, $v, k: 3), bm25($en, $c.text, $q), k: 60)
- Expected:
en-1only. Actual:de-1,de-2,en-1. - The RFC says hand-written text "ranks the population, which is the canonical form". Every GQT
rrfcase uses an unfiltered$all, so none catches this. fusedalso dropsweights:, the bm25 arm keeps onlytextfrom itsLexicalQuery(read_shape.rs:640, so aterms(…, mode: all)runs as match-any), andnearestdropsrecall:. The freeze list says these work. The T75 message tells users to writeweights.- Fix: lower a filtered population as the ranked scan's prefilter, as the single-ranking path does. Refuse with
P006any option the scan cannot carry. Never drop an option.
B6. An anonymous source on a free traversal does not terminate. For match { $_ knows $p } with $p not yet bound, traversal() renames the anonymous end to "_" (read_shape.rs:799-804), but its scan keeps __anon_N. The fallback at read_shape.rs:1171-1180 then pushes NodeScan { variable: "_" } on every pass, and "_" never counts as bound, so the loop never ends.
- On a 3-node graph the case hits the runner's 10 s wall-time budget. The old lowering refused this shape with an error.
issue_659_compiler_scopes.gqtcovers only the form where$pis bound first. - Fix: give the anonymous end one name in both places, and require each pass to bind a variable (refuse otherwise).
B2 (from my last review, now reachable). A can-fail term runs on rows its guard excludes. Arithmetic and if now reach the engine from user text, so the evaluator issue is no longer theoretical:
return { $p.slug, if($p.x = 0, 0, 10 / $p.x) as r }givesE2: '/' has a zero divisor. Expected:0and2.return { $p.slug, $p.y / $p.x as r }withynull andx = 0givesE2. Expected: null. The module doc ofengine/arith.rspromises null here.
Option 2 from my last review now applies: selective evaluation for if/coalesce, and the zero check limited to rows with a present dividend. The RFC still has to say whether and and filter-clause order are guards.
B7. Parameter defaults are type-checked but never applied. into_query_file builds the header Param without its default (syntax/parse.rs:292-296). The checker reads defaults only for calls to declared reads. query q($n: String = "a") { … $p.name = $n … } run without n gives parameter 'n' not provided. The RFC's own example $k: I64 = 5 can therefore not run, and it also hits P006 (limit over a parameter). Parameter metadata now lives in three types (syntax::Param, file::Param, the checked params). Keep one, so a field cannot be lost in a copy.
B8. ae9c2a42 weakens a break contract test instead of fixing the runner. The served GQT job failed at 6d40bc42 because only the in-process runner splices a parse code into the error text (error_text, omnigraph-gqt-core/src/lib.rs:3634). The served runner reads only the error line (served.rs:258-261) and ignores diagnostic.code. The server sends the code correctly. RFC section 6 says the fix of Q008 is "part of the break's contract, tested per code". The case now matches prose and no longer tests the code. Choose one owner for the error text:
- (preferred) Print the code in
QueryDiagnostic'sDisplayfor parse diagnostics, as type and plan diagnostics already do. Then deleteerror_text. The CLI, HTTPerrorline and both runners all showQ008. - or decode
diagnostic.codein the served runner and render it aserror_textdoes.
Then restore expect error: Q008.
B1 (still open). edition_corpus.rs is gone, but the new gqt/tests/integration/query_corpus.rs reads every gq block in docs/**/*.md. In ci.yml, the docs/*.md arm still sets no run flag. A docs-only PR can still add a block that does not compile under GQ 3, and no CI job runs on it.
Needs a decision
D1. Assigning a wider numeric value to a narrower property is now T3. numeric_fits (check/mod.rs:3375) refuses I64 → I32 and F64 → F32. The old checker accepted them, because numeric types were assignment-compatible, so stored mutations with $a: I64 for an I32 column stop loading. The rule ignores signedness: I32 → U32 and U64 → I64 pass. Reproduced: insert Person { …, age: $a } with $a: I64 and age: I32? gives T3: property 'age' has type I32? but the value is I64. The RFC says "an assignment never narrows" for arithmetic results, not for parameters. If this is intended, add it to the compatibility list, make signedness consistent, and give the error a fix (declare $a: I32).
Recommended (not blocking)
- S6. Validation accepts text that never runs.
queries validate, server boot (queries.rs:265) andcluster validate(config.rs:1413) only callcheck_query_decl. The checker accepts many shapes the planner refuses (limit $k,optional, computed order keys, block values compared with columns, writes with traversals), so a stored query deploys and then failsP006on every call.ReadShape::of(read, catalog)needs only the catalog. Run it, and the write-sideWritePlan::ofcheck, in registry validation andlint. That also moves the writeP006out of the engine. - S7. Plan acceptance lost its independent input.
validate/requirements.rsnow takes bindings, counts and rankings from the sameReadShapethe plan is built from. So acceptance cannot catch a lowering that drops a filter or an option, which is why B5 passes. RFC 0047 asks that requirements come from the checked query. Derive them from the core. - S8.
@description/@instructionare no longer validated.@description(42), a bare@description, and duplicates are accepted (syntax/parse.rs:275-283). The header then showsNoneor the first value to MCP and the catalog. The old parser refused these. - S9. A per-row insert of a keyed node runs once per matched row (
mutation.rs:1640-1650) and reportsnodes=Nfor one upserted row. - S10.
reads_search(write_plan.rs) only looks insideBinaryandNot. A search underCall,CastorIsNullgets the untyped "unsupported expression in mutation predicate" instead ofP006. - S11. The T42 needles are bare.
aggregate_identity_order,aggregate_metadata_name_still_ordersandconstant_edge_type_still_orders_groupschanged from rows toexpect error: T42with no message. The refusal is intended (compatibility list, "3 stored cases"). Name the message and rename thestill_orderscases, so the corpus states what is refused. - S3, S4 and S5 from my last review are unchanged:
ExprType::Edgestill joins with|(check/mod.rs:2013),SExprstill storescan_fail, and thematch_termsmode still goes through"all"/"any".
Tradeoffs and liability
Removing about 21,700 lines and a second checker and lowering removes much liability. One grammar and one checker are easier to keep right than two editions. But "One IR" is not yet one IR. PatternStep/ReadShape is the old IROp/QueryIR under a new name, and place, FreshNames and scan_root are copied from the deleted lower.rs. The read path is now core → Facts → pipeline steps, which is two intermediate forms to keep in step.
Meaning is still carried in strings (__anon_, __temp_b, ^name, "_", the hard-coded "score"), and B4 and B6 are both name-string bugs. After five more changes of this kind, each new core node will touch read_shape's item() and key_columns separately; they have already drifted for Neg and Call. The by-construction fixes are: a typed binding identity (scope, var) instead of name strings, and requirements derived from the core (S7). Together they remove this class of bug instead of patching each case.
Write groups are the largest new semantic surface. B3 shows that lowering a group to today's per-statement writes is not the same relation. That needs one selection owner in the executor before more statements (returning, @if_commit) arrive.
Validation
All runs were in a clean detached worktree. I restored every temporary case (git status clean).
- At
6d40bc42:cargo test -p omnigraph-compiler --lib --locked: 312 passed, 0 failed.cargo test -p omnigraph-gqt --locked: 536 of 536 cases passed in process, and the harness tests passed.cargo test -p omnigraph-planner --locked: 190 passed, 0 failed. ae9c2a42changes only one.gqtexpectation (the B8 case), so these results apply to the head.- The B2–B7 and D1 reproductions are temporary GQT cases, each failing at this head as described.
- B3: the second update left
nunset on both drafts. - B4:
kind: "Knows"for the Likes edge. - B5: 3 rows instead of 1.
- B6: the case hit its 10 s wall-time budget.
- B2:
E2in both cases. - B7:
parameter 'n' not provided. - D1:
T3as quoted. - B8: exact-head CI at
6d40bc42:GQT (served)failed only onboolean_literal_refused_where_a_property_is_named_false("served and in-process verdicts differ … error does not contain "Q008""). TheGQT (dst)job was cancelled by fail-fast; its cases that ran passed. - Exact-head CI at
ae9c2a42: still running when I posted. - Code inspection only: S6–S10, the
rrfweights/recall/termsoptions, and the B1 CI routing.
Limitations:
- The PR has no fix for B2–B7, so I could not run "before fix / after fix". The cases fail at this head for the stated reason, and each needs a fix and the case kept as its regression.
- One temporary case (a filter clause
10 / $p.x > 1behind$p.x != 0) failed in the GQT plan check ("malformed typed expression tree") before execution, so I could not test filter-clause guard order. - I did not run the served GQT locally, the engine integration tests or the workspace tests; the evidence for them is exact-head CI.
- I make no performance claims. The delta does not touch Lance storage paths.
…of the core IR Add `omnigraph-reference`, a test-only interpreter that evaluates a checked read's core IR as the RFC's operator table states it: nested loops over a branch's export, no index, planner, optimizer or cost model. It depends on the compiler and serde_json alone, which `tests/dependencies.rs` holds. Every operator the planner lowers has an evaluation and a unit test, and so do `Union`, `Fix` (naive iteration to the fixpoint) and `Call`, which the planner refuses with P006. GQT's `--- expect same as reference` evaluates a query step's read with it and compares the rows with the engine's, ordered or unordered as the step's rows expect says. The comparison goes through an `ExecutionHost` hook, so only the GQT runner depends on the interpreter; the served runner refuses the section by name, the DST runner skips it, and other executors refuse the case. The section is restored on the 15 cases that carried the old engine comparison and added to 17 more, so every lowered operator, aggregate and traversal form runs against the reference at least once.
… type A parameter default was range-checked only as a scalar integer or F32, so $xs: [I32] = [3000000000] compiled, and an assigned literal was not range checked at all. check_literal_type, which every default and assignment passes, now refuses an integer outside its type's range, a float that is not a finite F32 for an F32, and each element of a list or a vector by its element type, with T3 naming the value and the type.
…o test passes on a floor fix_convergence passed a refused step whose fixes printed no edit and never checked what the fixes wrote. Every corpus step whose refusal prints a machine-applicable fix is now listed with the text its fixes are meant to write, the repair must equal it, a listed step that prints no edit fails, and an unlisted one that does fails too. Its corpus floors become walks: every cases/diagnostics case holds a step the compiler refuses. The typing-work test discarded its check's result and timed it; it now asserts the result and the expressions typed. The plan-budget test read the term size out of the message; the checker records the largest term it charged. The numeric operand test counts every outcome exactly, and the bounded-refusals test expects exactly 256.
…es by what they own The engine test files engine_v2.rs, engine_v2_memory.rs and engine_v2_plan_replay.rs become execution.rs, execution_memory.rs and plan_replay.rs, and engine_v2_scrubbed_replay.rs becomes scrubbed_replay.rs. The GQT corpus drops its v2/ directory: plan assertions move to cases/planner/, the other cases to the corpus root (engine_v2_now_is_query_scoped becomes now_is_query_scoped), and cases_slow/v2/ to cases_slow/. Every reference follows.
Every read runs from its plan, and the session setting engine had one value, v2, which changed nothing. The setting, its Engine enum and the OMNIGRAPH_ENGINE variable are removed, with the engine field of a request's settings, the CLI's mapping of it, the GQT runner's OMNIGRAPH_GQ_ENGINE baseline and the engine field of a worker input. Every set engine = v2; line leaves the GQT corpus, the input_engine_route inventory case goes, and the tests that spelled the setting or listed it in show output drop it. The settings guide, the GQT README and the developer guides follow, and a breaking note says what callers delete.
ragnorc
left a comment
There was a problem hiding this comment.
Delta review of #1316 at 8e3a59a7
Recommendation: revise before merge. The three commits fix real defects, and the before/after runs below prove each fix. But the Q006 change adds a new blocking defect: an or chain written with is null, contains or starts_with now passes the depth bound, and a chain of 1000 such terms (about 30 KB of source) overflows the 16 MiB query stack and aborts the process. A second defect is in the same class as the guard fix: a block whose only failing part is its aggregate argument is still placed as if it cannot fail, so the core IR runs a where before the elements, and the reference interpreter refuses where the engine returns rows. The open findings from my earlier reviews (N1–N4, B8, D1, D2) are unchanged; this delta does not touch their code.
Reviewed head: 8e3a59a7. My last review covered aad3ca07 (posted at b429323f, review 5935982718). This review covers the three commits b429323f..8e3a59a7:
fa81835efix(compiler): a guarded block reads its guard as a column instead of typing a copy of it;0445816efix(compiler): Q006 measures each expression's depth, not the sum of its siblings;8e3a59a7fix(compiler): printed fixes take their positions from parse tokens, and positions from a line index.
While I reviewed, the head moved to 11c4b379 (four commits, 10:16Z–10:31Z: nearest over an optional or a union, one literal range rule, a served null parameter default, fix-convergence tests). I did not review those commits. They do not change depth.rs, syntax/parse.rs or diagnostic.rs, and the three check/mod.rs lines cited in D4 are the same at 11c4b379 (each moves down 8 lines). No Decision needed comment exists on this PR, and no inline comment covers this delta.
Why, what, how, impact
Why. Three compiler defects. (1) A block that can fail, written under a guard, held a copy of the guard and was typed again. A guard that held blocks was copied again inside them, so typing work grew exponentially, a name the guard bound was bound again (T68), and an aggregate argument was not guarded. (2) The Q006 estimate added up the and/or chains of sibling expressions, so a write of about 250 groups was refused although no tree was deep. (3) A missing comma's fix rescanned text for a trailing comment and cut a multiline string that holds //, then inserted the comma inside the string.
What and how. (1) Each guard is now the earlier operand as typed (lazy_guard). A guarded block's guard is projected once as a column of the outer row, and condition_env filters the block's Env leaf on that column (check/mod.rs:5914-6012). The checker counts typed expressions, and a test holds that count linear in the guarded blocks. (2) measure ends an expression at a list separator, or where "a second operand follows" an operand, and keeps only the deepest expression per level (depth.rs:173-180, :241-250, :308-317). (3) token_end takes a construct's end from its last parse token (syntax/parse.rs:225), and a thread-local LineIndex positions refusals from line starts computed once per source.
Impact. Queries that the guard rules promise to evaluate safely now run, and typing a source with many guarded blocks is linear. Large write sources pass Q006. Printed repairs no longer corrupt string literals. D3 below reverses part of the Q006 protection.
Blocking
D3. Q006 now undercounts an or chain whose terms use an infix word, and a long chain aborts the process. The new rule ends the expression at "a word right after an operand that is not an operator" (depth.rs:241-250). Only and, or, not, in, is, when, then and else are exempt. But the grammar has more infix words (grammar.pest:379-389, :124): null after is ($x is null), contains, starts_with, and the habit spellings like, between, starts with/ends with, and case … end. Each of these words resets the chain count in the middle of one expression. So $p.nick is null or $p.nick is null or … measures one level, whatever its length. The tree it builds is one level per or.
Measured through the public check_source_all, on the 16 MiB on_query_stack thread, with RUST_MIN_STACK=16777216 (debug build). Schema node Person { handle: String @key, nick: String?, age: I32 }; the query is match { $p: Person } where <n terms joined by or> return { $p.handle }:
| Term | n | b429323f (before) |
8e3a59a7 (head) |
|---|---|---|---|
$p.handle = "a{i}" |
300, 1000, 5000 | Q006 | Q006 |
$p.nick is null |
300 | Q006 | accepted |
$p.nick is null |
1000, 5000 | Q006 | stack overflow, process aborts |
$p.handle contains "a{i}" |
300 | Q006 | T7 (habit refusal; Q006 passed) |
$p.handle contains "a{i}" |
1000, 5000 | Q006 | stack overflow, process aborts |
$p.handle starts_with "a{i}" |
1000, 5000 | Q006 | stack overflow, process aborts |
is null is valid GQ, so a 300-term chain is a valid query with a 300-level tree that the bound must refuse. A stack overflow is not a panic: it aborts the whole process. The server and the CLI give each query thread this same 16 MiB stack (depth.rs:39-45), so one ad-hoc read could stop a server. I did not run this against a server or a release build. Release frames are smaller, so the threshold is higher, but the undercount has no limit: 5000 terms also pass the bound.
- Invariant: the estimate never undercounts the tree a source can build (
depth.rs:23, and the Q006 row indiagnostics.md:183); failures are bounded and observable (invariants 8 and 11). - Class: every infix word in
comparison, and every word added to it later. The rule is a deny-list of words that continue an expression, so each new infix word reopens the defect without a test failing. - Fix by design: invert the rule. Unknown words continue the expression. Only the words that start a clause or an element end one (
match,where,return,order,limit,let,insert,update,delete,set,as, …), together with the list separators. The estimate then errs high, as its contract requires. A word you forget then costs a false Q006, never a crash. Sibling write groups still reset atmatch/update/insert/delete, so the 250-group write that0445816efixes still passes. - Regression: GQT can express it as an error. Please add a case whose steps are a 300-term
is nullchain and a 300-termcontainschain, each with--- expect error: Q006. At8e3a59a7the first is accepted and the second is refused T7, so both fail for the predicted reason. Keep thedepth.rssibling test as it is.
D4. A block whose only failing part is its aggregate argument is placed as if it cannot fail. fa81835e makes the aggregate argument count toward whether a block can fail, but only for the guard decision (check/mod.rs:5157: term.can_fail() || arg…can_fail). Two other sites decide placement from a different answer:
- a
whereconjunct is fenced after the elements whenpred.can_fail || blocks.iter().any(|b| b.inner.can_fail())(:2815).b.inneris the block's term without theGroup, so the argument is not counted; - a
letfences later conjuncts only whenv.can_fail(:3289). The block's value is a column reference, so no block counts at all.
Observed at 8e3a59a7 with a temporary GQT case (persons ada age 1 and bob age 5, each working at a company; dan age 0 works nowhere, so the element $p worksAt $c drops dan):
| Step | Result |
|---|---|
where count { $d: Company where 100 / $p.age > 0 } > 0 (inner where fails) |
ada, bob; same as reference |
where sum($x) { $d: Company, let $x = 100 / $p.age } > 0 (same division, as an inner let) |
ada, bob; same as reference |
where sum(100 / $p.age) { $d: Company } > 0 (same division, as the argument) |
the engine returns ada, bob; the reference refuses E2: a divisor is zero |
match { $p: Person, let $s = 100 / $p.age } where $p.age != 0 |
E2, as traversal.md:237-245 states |
match { $p: Person, let $s = count { $d: Company where 100 / $p.age > 0 } } where $p.age != 0 |
ada (engine and reference agree); the where ran below the let |
Row 3: the core IR runs the conjunct before $p worksAt $c, against "where … is evaluated after all of them" (traversal.md:33). The engine happens to return the documented rows, so engine and reference disagree. AGENTS.md calls that a bug in one of the two, and here it is the IR. The same step fails the same way at b429323f, so this delta did not cause row 3, but fa81835e fixes the same cause through the guard path and leaves this path open. Row 5: a let whose value is a failing block behaves differently from the scalar let in row 4, against the documented let rule.
- Invariant: an expression that can fail runs only on the rows the written order selects (A3), and the core IR states that order (the reference interpreter is its oracle).
- Class: every site that asks "can this expression fail?" The answer has four owners today:
SExpr::can_fail,Term::can_fail, the localcan_failat:5157, and the block clause at:2815. They disagree. - Fix by design: one owner. Let the block's value expression carry
can_fail = term.can_fail() || arg.can_fail(that isTerm::can_failof theGroupthe block lowers to, which already counts aggregate arguments). Thenpred.can_failat:2815andv.can_failat:3289cover blocks with no extra clause, the clause at:2815and the local at:5157go away, and the guard decision reads the same flag. Please check the other readers ofSExpr::can_failwhen you make this change. If row 5 is intended, change thelettext intraversal.mdinstead. Either way, the doc and the IR must agree. - Regression: extend
guarded_block_runs_only_where_its_guard_selects.gqt(or a sibling case) with rows 3 and 5. At8e3a59a7, row 3 fails with the reference refusal and row 5 returns ada.
Optional
- O13.
condition_envdrops the guard silently for a term shape it does not know. It walksExpand,Filter,Project,Group,Correlate(outer),OrderandCut, and returns any other variant unchanged (check/mod.rs:6010). I checked that today theEnvleaf is always on that spine:finishstarts fromEnvand every step inapplywraps it in one of those variants (OptionalthroughCorrelate's outer side). So this is not a defect now. But a future step that wraps the scope in aUnion,ScoreorCallwould make a guarded block run on every outer row without an error. Please make the fallback loud (a typed internal error, or adebug_assert!plus a test that walks each step kind), so the invariant holds by construction and not by today's step list. - O14.
token_endvisits every inner pair, not only the last..into_inner().map(token_end).max()(syntax/parse.rs:235-238) recurses through the whole subtree on each call, andspan_of/source_spancall it for each construct. The oldtrim_endcost only the trailing whitespace. Total work is now the sum over the tree of subtree sizes, bounded by source size × depth. I did not measure it. Pest pairs are in source order, so the last non-empty inner pair is enough.
Earlier blocking findings
| # | Status at 8e3a59a7 |
Evidence |
|---|---|---|
| N1–N3 | Open, unchanged | The delta changes no planner, optimizer, shaping, engine/scan.rs or lexical_scan.rs code |
| N4 | Open | upgrade.md and lance.md rollback text unchanged |
| B8 | Open, unchanged | omnigraph-gqt-core unchanged |
| D1 | Open | Cargo.toml and the lance.md NGRAM row unchanged |
| D2 | Open | dataset_index.rs, error.rs and the troubleshooting text unchanged |
Tradeoffs and liability
fa81835e removes liability. It replaces re-typing of copied source with one typed guard column, which makes typing linear, removes the T68 false refusal, and closes an escape from T39. The test that holds typing work linear is the right kind of guard. It adds condition_env, a second structural walk over the term with a silent fallback (O13), and it leaves the "can fail" answer spread over four places (D4). After five more changes of this kind, each new construct that can fail must remember to update each site. One flag on the value expression would make that hold by construction.
0445816e fixes a real false refusal, but it trades a safe overcount for an unsafe undercount (D3). For a safety bound, the cost of a missed word must be a false refusal, not a crash. That is why a list of words that end an expression is the right shape, and a list of words that continue one is not.
8e3a59a7 replaces a text rescan with parse facts, which is one source of truth for where a construct ends. string_lit is atomic, so its text never reaches the trivia scan, and lists hold only element and missing_comma_rec children, so last_end is always set before a gap. The LineIndex is keyed by the borrowed source's address and length, and it is removed on unwind. I found no defect. O14 is a cost note only.
Validation
Clean detached worktrees at b429323f (before) and 8e3a59a7 (head). GQT built in the shared target directory; compiler tests in a separate one. RUST_MIN_STACK=16777216 as in CI.
| Proof | Command | b429323f |
8e3a59a7 |
|---|---|---|---|
fa81835e: the PR's case, copied unchanged |
cargo test -p omnigraph-gqt --test gq_logic_tests --locked -- guarded_block_runs_only |
FAIL at step 2 (if guards the argument): E2: '/' has a zero divisor. Each other step alone: 1 (or) ok; 3 (coalesce) T68; 4 (and, inner let) E2; 5 (a guard holding a block) T68 |
1 passed (0.96 s) |
0445816e: siblings_are_measured_by_the_deepest_not_their_sum, copied into depth.rs |
cargo test -p omnigraph-compiler --locked --lib |
FAILED: depth(write(1)) = (2, 6), depth(write(1000)) = (2, 2005) |
passed |
8e3a59a7: the three new PARSE_REPAIRS rows, copied |
same | FAILED: the repair reads let $s = "x, $p knows $q, the string cut and the comma inside it |
passed |
| Compiler lib at head | same | — | 384 passed, 0 failed |
| D3 | temporary integration test calling check_source_all inside on_query_stack |
table in D3 | table in D3 |
| D4 | temporary GQT case, table in D4 | row 3 fails the same way | rows as in D4 |
Each "before" failure is a predicted behavioral reason, not a compile, setup or empty-run failure. I rebuilt the head GQT binary after the base build and ran the head cases again with the same results.
CI: at 8e3a59a7 every check passed (among them GQ Logic Tests, all GQT jobs, Test Workspace, Lint (clippy), DST pinned suite, Storage Upgrade Compatibility) except RustFS S3 Integration (default), which was cancelled when the head moved. At 11c4b379, at 10:59Z, GQ Logic Tests, GQT (dst-clippy), Lint (clippy), DST pinned suite, Dependency Guard (cargo deny) and Check AGENTS.md Links had failed. Those runs are outside this delta and I did not read their logs.
Limitations:
- D3 was run in a debug build through the compiler API, not against a server or a release build.
- I did not run
case … end,likeorbetweenchains. The same word rule applies to them by inspection. - I did not run the workspace, DST or Clippy suites. I did not review
0efb4ca0..11c4b379. - All temporary test edits and probe files were removed, both worktrees are clean, and nothing was pushed to the PR.
…int the resolved analyzer T2, T14, T21, T32, T33, T35, T36, T45, T46, T54, T58 and T59 had a catalog entry and no emitter: no compiler, planner or engine source raises them. They leave the catalog, and their numbers stay unallocated. T17, T19 and T20 describe what raises them now: a nearest call without its window, a full-text call's field or terms operand, and a lexical query that is not text or has no searchable terms. schema plan printed an omitted analyzer as "edition 2 analysis"; it prints the profile and scorer the index resolves to (FullTextAnalysis::resolved), so an omitted analyzer reads analyzer = "english_v1".
Every ranked scan is a nearest scan, so RankKind had one variant. It goes, with RankedAccess's kind: the score is the distance column (DISTANCE_COLUMN), sorted ascending, and the code that matched on the kind uses it directly. Explain keeps a ranked scan's documented kind: "nearest"; a saved plan no longer carries the kind, so BOUND_PLAN_VERSION is 15. The full-text validation probe's comment states what enters the validation now.
ragnorc
left a comment
There was a problem hiding this comment.
Delta review of #1316 at 11c4b379
Recommendation: revise before merge. The three fixes in this delta are real, and the before/after runs below prove each one. One new blocking defect: the new nearest path returns the right rows, but it drops the ranking object that the search page promises for every read that draws from a ranking. The CI failures that this delta caused (Clippy and the MCP page length) are already fixed at 88bd29c3. The findings from my earlier reviews (N1–N4, B8, D1–D4) are unchanged. This delta does not touch their code.
Reviewed head: 11c4b379. My last review (review 5937237232) covered b429323f..8e3a59a7. I posted it at 11c4b379, but it said that I did not review the four newer commits. This review covers those four commits, 8e3a59a7..11c4b379:
0efb4ca0fix(planner):nearestranks a population that holds anoptionalor aunionby its rows' exact distance;4abe9fe4fix(api): a declarednullparameter default survives the served catalog;f9ea81cefix(compiler): one range rule for every literal that meets a declared type;11c4b379test(compiler,gqt): fix convergence checks the intended repair, and test floors become exact counts.
While I reviewed, the head moved to 88bd29c3 (seven commits, 10:36Z–11:20Z). I checked only what touches this review: f57db50d and 88bd29c3 fix the two CI failures below, and 221c4bf2 removes RankKind but leaves the logic of planned_ranking that D6 cites unchanged. I did not review a9bf35cf (test file renames), 5ed97b54 and be9d7172 (the engine setting removal) or 8ef2695a (diagnostic codes and the analyzer print). Line numbers below are at 88bd29c3. No Decision needed comment exists on this PR, and no inline comment or later review covers this delta.
Why, what, how, impact
Why. Three defects. (1) A nearest over a population that holds an optional or a union failed with a planner internal error. The population is read as a read of its own, so the ranked node is a column of its rows, but the shaper still scanned it first (the ranked binding $d has 2 scans). (2) The served catalog read a declared = null default as "no default". (3) The range check covered only a scalar integer or F32 default, so $xs: [I32] = [3000000000] and an assigned literal such as set { age: 3000000000 } passed type checking and failed later, at execution, with an untyped error.
What and how. (1) ranking_steps (shape/read.rs:1762) checks whether the population binds the ranked node itself. If it does not, it adds an exact-distance Score step and ranks by that column. The optimizer then always adds the same Sort and TargetCut (optimizer.rs:582-632). (2) ParamDescriptor.default deserializes through present_value, so a present null is Some(Null) and an absent key is None. (3) out_of_range in check_literal_type (literal.rs:95) is now the one range rule for defaults and assignments, element by element for lists and vectors.
Impact. Queries that ranked such populations now run, and the rows agree with the reference interpreter. MCP and queries list show a declared null default. Out-of-range literals are refused at type check with T3 before anything runs.
Blocking
D6. A nearest on the new exact-distance path returns no ranking object. The search page states: "A read that draws from a ranking carries a ranking object in its JSON envelope" (docs/user/search/index.md:265). planned_ranking (omnigraph-planner/src/ranking.rs:62) finds the ranking from a ranked Scan, a Fuse, or an exact bm25 Score. The new path has none of these. It has a distance Score, so planned_ranking returns None.
Observed with the CLI built at 11c4b379 (omnigraph query --json, direct local graph, Doc { slug @key, embedding: Vector(4) @index }, two docs, one tag):
| Query | Rows | ranking |
|---|---|---|
nearest over { match { $d: Doc } return { $d } }, k: 2 |
a, c | {"by":"nearest","order":"ascending","exhaustive":true,"score_columns":["distance"]} |
the same over { match { $d: Doc, optional { $t: Tag, $d tagged $t } } return { $d, $t } } |
a, c (correct) | absent |
A client that reads ranking to find the score column or to check exhaustive gets nothing for a query that the user wrote as nearest.
- Invariant: the read reports the ranking it draws from, whatever physical path computes it. Query semantics are typed structures (invariant 9). The report must not depend on which operator happens to compute the score.
- Class: every ranking that is computed without a ranked scan. Today that is this path. A future exact path for
bm25in some shape, or anearestthat the planner chooses to compute exactly, would hit the same problem. The reporting is inferred from physical operators, so each new physical path must remember to updateplanned_ranking. - Fix by design: the shaper already knows that this read draws from
nearest(ranking.keyisNearest). Record the ranking function on the step (for example onPatternStep::Rank, or as the planned ranking of the read), and letplanned_rankingread it. Exhaustiveness then comes from the scans only where scans exist. An exact path is exhaustive by construction. This removes the inference instead of adding a fourth case to it. - Regression: GQT has no
rankingexpectation, so the cheapest owner iscrates/omnigraph/tests/search.rs, which already asserts therankingobject (result.ranking()). Add theoptionalpopulation there and assertby: nearest,exhaustive: true, and the score column.
Optional
- O15. A list literal for a scalar gets a range message instead of a type message.
out_of_rangeruns beforetypes_compatible(literal.rs:95and:120), and for aLiteral::Listit checks the elements even when the declared type is not a list. So$x: I32 = [3000000000]is refused withT3: `x` has type I32, but 3000000000 is outside its range. The real error is that a list is not anI32. The code is still T3, so this is only a diagnostic issue. Check the type first, then the range. - O16. Two range owners remain.
integer_range(literal.rs:399) is the new rule, butfits(check/mod.rs:6658) still decides integer ranges for negated literals (:6619,:6630). Letfitscallinteger_range, so one table holds the ranges. - O17.
fix_convergence'sINTENDEDlist is a second home for case data. The intended repair of each diagnostic step now lives in a Rust table keyed by"<file> step N"(fix_convergence.rs:224), not in the.gqtcase. A change fails loudly, which is good: an unlisted step that prints a fix fails, and a listed step that prints none fails. But each new diagnostic case needs edits in two files, and inserting a step renumbers the keys. An--- expect fixsection in the case would keep one owner. - O18. The typing measure is production state that only tests read.
11c4b379made Clippy fail onfields `exprs` and `largest_term` are never read(Clippy, GQT dst-clippy and DST pinned were red at11c4b379, then green at88bd29c3).f57db50dfixes this with#[cfg_attr(not(test), allow(dead_code))]onMeasured(check/mod.rs:311). That works, butlargest_charged(:1161) is still a productionCellthat everychargecall updates, only so that a test can read it. The T70 refusal already holds that size in its message. If a test needs the size, a typed field on the refusal gives it one owner, andMeasuredcould then becfg(test)only.88bd29c3also bringsdocs/user/operations/mcp.mdback to 350 lines, which clearsCheck AGENTS.md Links. - O19. Put the range regression in GQT. The regression for
f9ea81ceis a compiler unit test. GQT can express this contract (expect error: T3), andparameter_defaults_bind_omitted_parameters.gqtalready holds the scalar range steps. Please add the list-default and assignment steps there (see Validation for the exact steps that I ran).
What I checked that holds
0efb4ca0over ties, null vectors, a parameter window andrrf. A temporary GQT case agrees with the reference interpreter on all five steps at11c4b379:k: 1with two docs at the same distance keeps the smaller identity; a doc with a nullembeddingis left out (nearestskips null vectors);k: $kbinds throughfold_pipeline; aunionpopulation with a tie; and anrrfarm over anoptionalpopulation where one target has two rows (each target is ranked once). At8e3a59a7, every step fails with the planner internal error.- The
Rankstep had onlyNearestkeys before this delta, becausesearch_nodereturnedNonefor any other key. So making theTargetCutunconditional (optimizer.rs) and treating anyRankas the window (requirements.rs:177) changes no earlier plan. - The exact path scores with the same kernel as Lance's flat kNN (
docs/dev/lance.md, "Vector distance"), so theF32values equal an exhaustivenearest. This path calls no Lance API. Its cost is one exact distance for each population row with no index. That is acceptable, because before this delta these queries were refused. 4abe9fe4: the otherOption<Value>fields inomnigraph-api-typesareparams, wherenulland absent mean the same.F64overflow such as1.0e309is already refused at T3, so a non-finite default cannot reachliteral_jsonand appear as"default": null.
Earlier blocking findings
| # | Status at 11c4b379 |
Evidence |
|---|---|---|
| N1–N3 | Open, unchanged | The delta changes ranking shaping only for populations that do not bind the ranked node; it does not touch the code these findings cite |
| N4 | Open | upgrade.md and lance.md rollback text unchanged |
| B8 | Open, unchanged | omnigraph-gqt-core unchanged |
| D1, D2 | Open | Cargo.toml, dataset_index.rs and error.rs unchanged |
| D3 | Open | depth.rs unchanged |
| D4 | Open | The can_fail sites in check/mod.rs are unchanged in logic |
Tradeoffs and liability
0efb4ca0 removes liability. It fixes the cause, which is "the shaper assumed that the ranked node is a binding", with one general rule rather than a special case for optional and union. Any population shape that yields the node as a column takes the exact path. The cost is D6: the ranking report is still inferred from physical operators, so the new path is invisible to it. 4abe9fe4 is small and correct, and present_value is the right shape. f9ea81ce moves a late, untyped execution error to a typed T3 at check time, and replaces one special case with one rule. O15 and O16 are what is left to make it fully one owner. 11c4b379 makes tests stricter and removes message parsing. The cost is test-only state in the production checker (O18) and a second home for case data (O17).
Validation
Clean detached worktrees at 8e3a59a7 (before) and 11c4b379 (head). The GQT runner was built with cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --no-run at each commit, and each case was run with omnigraph-gqt <case> and RUST_MIN_STACK=16777216.
| Proof | Command | 8e3a59a7 |
11c4b379 |
|---|---|---|---|
0efb4ca0: the PR's case, unchanged |
omnigraph-gqt cases/nearest_ranks_a_population_holding_an_optional_or_a_union.gqt |
FAIL step 1: planner internal error: the ranked binding $d has 2 scans under its search node; the union step alone: … has 3 scans … |
ok (1.55 s) |
0efb4ca0: ties, null vector, k: $k, union tie, rrf arm (temporary case, each step expect same as reference) |
omnigraph-gqt /tmp/probes/nearest_over_rows_probe.gqt |
FAIL step 1, same planner error | ok, all five agree with the reference |
| D6 | head CLI: init, load --mode merge, query --json |
— (the query is refused) | ranking absent for the optional population; present for the plain one |
4abe9fe4: the PR's unit test, copied |
cargo test -p omnigraph-api-types --locked --lib a_null_default_survives |
FAILED: {"…","default":null} reads back None, expected Some(Null) |
1 passed |
f9ea81ce: the PR's unit test, copied |
cargo test -p omnigraph-compiler --locked --lib a_literal_meeting_a_declared_type |
FAILED at its first row on wording only (base refuses with the default … is outside the range of I32), so I used the GQT steps below for the behavior |
1 passed |
f9ea81ce: GQT steps added to a copy of parameter_defaults_bind_omitted_parameters.gqt, one step per run |
omnigraph-gqt <copy> |
[I32] = [1, 3000000000] default: param 'xs': value Integer(3000000000) exceeds I32 range (no T3); update … set { age: 3000000000 }: value Integer(3000000000) exceeds I32 range at execution; the same for insert; Vector(2) = [0.5, 1.0e39]: value Float(1e39) exceeds F32 range |
each refused with T3; the whole extended case passes |
| O15 | same, step $x: I32 = [3000000000] |
— | T3: `x` has type I32, but 3000000000 is outside its range |
| O18 | exact-commit CI | Clippy, DST and GQT passed at 8e3a59a7 |
Clippy dead fields and mcp.md at 351 lines at 11c4b379; at 88bd29c3 every check passes except Dependency Guard (cargo deny) |
Each "before" failure has the predicted behavioral reason, except the unit test row marked as wording, which I replaced with GQT steps. Steps to add for f9ea81ce:
--- query
query out_of_range_list_default($xs: [I32] = [1, 3000000000]) { match { $p: Person } where $p.age in $xs return { $p.handle } }
--- expect error: T3
--- mutate
query out_of_range_assignment() { match { $p: Person } where $p.handle = "p1" update $p set { age: 3000000000 } }
--- expect error: T3
Limitations:
- I did not run the workspace, DST or Clippy suites locally. I relied on exact-commit CI for O18.
Dependency Guard (cargo deny)fails at88bd29c3(chrono,chrono-tz,humantimeunmaintained). It is a new advisory, not this PR: #1438 fails the same check atde2d9307, andmainpassed it at 07:46Z.- I ran D6 at
11c4b379. At88bd29c3I checked it by inspection only:shape/read.rsis unchanged andplanned_rankingstill finds a ranking only from a ranked scan, aFuseor an exactbm25score. - The D6 run used a direct local graph through the CLI, not a server. The server uses the same
planned_ranking. - I did not measure the cost of the exact path on a large population.
- All temporary cases were outside the repository, the copied unit tests were removed, both worktrees are clean, and nothing was pushed to the PR.
The GQ 3 RFC (section 2) makes `and` and `or` guards: `a and b` evaluates `b` only where `a` is not false, `a or b` only where `a` is not true. The interpreter evaluated both sides, so a right side that can fail refused a row its left side had already decided, where the engine answered.
`Score` refuses a non-finite input (the GQ 3 RFC, section 2), and a distance from finite components can overflow `F32`. The interpreter reported the infinity as the row's score, which renders as null, and kept an overflowing target in a `nearest` window. `distance` now refuses any non-finite score with E3 and `nearest` a non-finite one among the targets it keeps, as the engine does.
The interpreter bound only the parameters a caller supplied, at whatever value it held. It now binds as the engine's binding does: an omitted parameter takes its declared default, an omitted nullable one without a default is null, and an omitted required one, a null for a non-nullable one, a number outside its declared type, a DateTime with a non-zero digit past the millisecond and a value bound to now()'s reserved name are refused. A window or constant read from a parameter is refused outside its range (a limit below zero, a nearest window below one, rrf's constant outside 1..=RRF_MAX_K), and so is a null lexical query, which the RFC makes a non-null String (section 4). A DateTime is its instant: spellings of one instant, another offset or zero padding, compare and render as one value.
The interpreter refused every lexical call over an english_v1 field, the default profile, because it carried no stemmer, and applied no NFC. The GQ 3 RFC (section 4) fixes both: NFC before tokenization, and english_v1 stems with the English Snowball stemmer, then removes the stop list of each word and its stem, then folds. The interpreter now runs the whole pipeline in that order, with frostem's English algorithm and the compiler's stop list; unicode-normalization supplies NFC.
Reading the entity behind a reference scanned its type's rows for the id, so a read over n entities of one type took n² comparisons; a 16,384-edge case spent seconds there. Each type now keeps its entities' positions by id; scans still read the rows in export order.
Export read only a branch head. `Omnigraph::export_jsonl_at` exports at any read target, a branch or a captured snapshot, through the same coherent cut; GQT's reference comparison reads a snapshot step's rows through it.
A step compared with the core reference interpreter only when its case wrote `--- expect same as reference`, about a third of the rows steps, and no error step. Every read step on the direct engine is now compared: - rows against rows, ordered or as bags as the step's expect compares; - an `E`-coded engine refusal against the same code from the reference; - an engine refusal of input the query cannot read (a parameter value, a read target) against the reference refusing its input; - an engine resource bound against the reference's rows, since the reference has no bound. A step refused at compile time has no core term and is not compared. The DST, served and benchmark executors do not compare: the comparison is a property of the direct target, not of the case. A step opts out only with `--- reference skipped: <reason>` after its last expect section; the reason is required, the section is a new `reference_skip` rule, the runner prints the reason under the case's result line, and a skip on a step the comparison never reaches fails the step. The comparison reads the step's read target, a snapshot included. `--- expect same as reference` is removed, and the case format is version 2.
Drop every `--- expect same as reference` section, which the default comparison replaces; the steps commit ce8af54 took out of comparison, because the served suite ran their cases, compare again on the direct engine. Steps that scan a catalogue type, which the export does not carry, and the two scale cases whose data the reference's nested loops cannot read within the case's budget opt out with that reason. The fix convergence test names the typed_search_options step by its new section index.
…ory bounded by the window The exact lexical scan kept every matching document's identity and its per-term alternatives until the corpus statistics were known, then a score per identity, so its memory grew with the corpus: TREC-COVID query 48 over 171,000 documents exceeded the 150 MiB pool. The field corpus is now read once for its statistics alone, keeping no document, and the Score operator scores each row of its population from the row's own text, which its scan projects; a sort with a limit then keeps the window. A corpus several times the pool scores within it.
…ch is refused for an index The upgrade guides described a stored-query upgrade procedure the owner has not settled; they now say it is pending and keep only what a refused stored query prints, the same error[CODE] line for a parse and a type failure. An explicit l2 distance is stored as an omitted one, so only another distance keeps an earlier binary out. A lone re-mention's filter is written with coalesce when its test can be null, and an optional adds plan nodes in proportion to its own pattern. No guide or skill line says this binary refuses a search with FullTextIndexRebuildRequired: only an earlier binary does, on an index this one built.
The parse-print-parse generator now builds whole source files (one to three declarations, or declarations and an `explain`) and covers the header (parameters of every type with list and negative defaults, annotations with and without parentheses, `as commit` and `as unattached branch`), string escapes, raw control characters and comment-like text inside strings, negative numbers in lists, infix `contains`/`starts_with`, bare `@meta`, keyword-named calls and catalogue edges in alternations. Each case also requires printing to be a fixed point and reparses a respelling with comments between tokens and raw newlines, tabs and carriage returns in strings. A coverage test parses the generated texts of a fixed seed and requires every rule of grammar.pest except a commented exclusion list (silent rules, rules inside atomic tokens, recognizers, the statement grammar). A display round trip types generated predicates and return values, displays the stored IRExpr, types the displayed text in the same query and requires the same declaration. Membership, `contains(…)` and `starts_with(…)` are left out: Display prints them infix, which the type checker refuses. The gate runs 450 and 300 cases; the nightly twins run 100,000 and 20,000 and are enrolled in tests-slow-nightly.yml.
ragnorc
left a comment
There was a problem hiding this comment.
Delta review of #1316 at 9888b0e5
Recommendation: revise before merge. This delta adds no new blocking defect. The bm25 memory fix is correct: the rows and scores are the same as before, and its new test fails before the fix and passes after it. The GQT corpus now compares every direct-engine read with the reference interpreter, and all 613 cases pass at the head. The cost of the memory fix is that an exact bm25 now analyzes its population's text a second time. This doubles the work that it charges to read_work_limit, and the execution report does not show the second pass (O20). The earlier blocking findings (D6, N1–N4, B8, D1–D4) stay open. This delta does not change their code.
Reviewed head: 9888b0e5. My last review (review 5938070931) covered 8e3a59a7..11c4b379. It said that I did not review a9bf35cf, 5ed97b54, be9d7172 and 8ef2695a. This review covers those four commits and every commit after 88bd29c3, up to 9888b0e5:
a9bf35cftest file and GQT case directory renames;5ed97b54andbe9d7172remove the one-valueenginesetting;8ef2695aremoves twelve diagnostic codes that no stage raises and prints the resolved analyzer;2f3c724b,f7fbbf30,070b06d3,6d1fd000,ef87065c: reference interpreter fixes (and/orguards, non-finite distance, parameter binding,english_v1and NFC, lookup by id);c0dbc286export at a read target;3732684aand9943820fcompare every direct-engine read step with the reference;cc3cb99cthe exactbm25scores each row from its own text;37d04499the exact scan's cost;4ed86485upgrade and GQ 3 guide edits;9888b0e5the round-trip generator covers every grammar production.
No Decision needed comment exists on this PR. No inline comment or later review covers this delta.
Why, what, how, impact
Why. The exact bm25 kept every matching document's identity and term hits until it knew the corpus statistics. Its memory grew with the corpus, and a 171,000-document query exceeded the 150 MiB pool. Also, GQT compared only about a third of the read steps with the reference interpreter, and no error step.
What and how. cc3cb99c reads the field corpus once for its statistics only (Bm25Scorer, lexical_scan.rs:588). The Score operator then scores each population row from the text that its scan now projects. A sort with a limit keeps the window. 3732684a makes the reference comparison the default for every read step on the direct engine. Rows are compared with rows, and an E-coded refusal with the same code. A step opts out only with --- reference skipped: <reason>. The reference interpreter fixes make the interpreter follow the RFC where the engine already did.
Impact. A bm25 over a corpus larger than the query pool now answers within the pool. The GQT corpus is a much stronger oracle: 12 steps opt out, each with a reason (11 catalogue types that the export does not carry, 1 scale case).
Blocking
None new in this delta.
Optional
- O20. The exact
bm25analyzes its population's text twice, and the report shows only one pass. The corpus pass analyzes every value of the field. ThenBm25Scorer::scoreanalyzes each population row again and charges its tokens and comparisons toread_work_limit. For a whole-type population, the charged work doubles. A temporary GQT case (5 docs,bm25over{ match { $d: Doc } return { $d } }) needsread_work_limit15 at88bd29c3and 30 at9888b0e5. At 29 the head refuses withresource limit exceeded for read_work_limit: actual 30, limit 29. Under the default limit, the largest corpus that abm25can answer is therefore about half of what it was. Also,ScoreExec::executecreates the second pass's memory without the operator's metrics (score.rs:359; comparescan.set_metricsat:355). Solexical_scan_rowsandlexical_scan_tokenscount only the corpus pass, while the work charged is two passes. I checked the metrics by code inspection only.- Tradeoff: memory bounded by the window, for about twice the analysis time and work. I think that this is the right trade until stage L, because N3 already makes the cost linear in the field. But the cost must be visible.
- Fix: attach the operator's metrics to the second pass (or report it under its own metric names), and say in "The exact scan's cost" (
analyzers.md:89) thatbm25analyzes the field once and its population rows again. This is the same class as N3: each lexical call adds a full pass that the query's shape does not bound.
- O21. The
Scorerow of the operator table still describes the oldbm25.docs/dev/execution.md:666says that abm25row's "identity [is] looked up in the scoresLexicalMembers::resolvecomputed".cc3cb99cupdated theFilterrow next to it, but not this row. It should say that each row is scored from its own text byBm25Scorer::score, and that the analysis is charged asexact lexical scan. - O22. Any other engine error counts as agreement with a reference input refusal.
EngineRefusal::ofmaps every error that is not a compiler diagnostic or a resource bound toInput(omnigraph-gqt-core/src/lib.rs:4537). That includes internal errors (Manifest),DataFusion,StorageandIo.judge_referencethen passes the step when the reference also refuses its input (:4654). So an internal engine error on a step whose parameters the reference refuses is reported as "same as reference". Today this needs a case whoseexpect error:text matches the internal error, so the risk is small. A safer rule:Inputonly for the typed parameter-binding and read-target errors, and an internal error always fails the comparison. - O23.
Displayprints three forms that the type checker refuses. The new round-trip test leaves out list membership,contains(…)andstarts_with(…), becauseDisplayprints them infix and the checker refuses that spelling (round_trip.rs:82). The same note says that explain output, GQT plan expectations and the docs print this spelling. A user who copies a filter fromexplaingets a refusal. One spelling for the printer and the checker would remove the exclusion. This is not new in this delta, but the delta makes it visible. - O24. Two GQT case notes cite codes that no longer exist.
issue_640_search_score_projection.gqt:4citesT33, andrank_key_asc_direction_refused.gqt:4citesT54, but that case now expectsT65.8ef2695aremoves both codes from the catalog, but these notes still describe them as refusals.
What I checked that holds
cc3cb99cgives the same rows and scores as before over every population shape that I tried. A temporary GQT case with five steps passes at9888b0e5(with the reference comparison) and gives the same rows at88bd29c3:bm25over a population that holds anoptional(d1keeps both tag rows with one score), over aunion, over a traversal with fan-out, insiderrfwithnearest, and a plain window withlimit 2. The scored text reachesScorein each shape, so the lowering's "field … is not in its input" refusal does not occur.- The new
Bm25Scoreruses the same analyzer profile,Corpus::scoreand IDF as the old code. A row's length and hits come from the same analysis of the same value at the same snapshot, so the score is the same. 3732684a: the judge requires rows to match rows (ordered or as bags, as the step's expect does), anEcode to match the same code, and a stale skip to fail. A static refusal has no core term and is correctly not compared. The DST, served and benchmark executors do not compare, which is right, because the comparison belongs to the direct target.5ed97b54: theenginesetting had one value. The removal is complete in the code paths (settings.rs,api-types, CLI, GQT runner, bench). The breaking note says that a staleset engine = v2;is refused as an unknown setting.8ef2695a: no compiler, planner or engine source raises the twelve removed codes. Only GQT notes (O24), release notes and RFCs mention them.a9bf35cfonly renames (34 lines in, 34 out).c0dbc286passes aReadTargetthrough the existing export path, with no new read logic.9888b0e5: the round-trip tests pass locally (8 passed, the 2 nightly twins ignored).4ed86485removes the stored-query upgrade procedure and says that it is pending. That is honest, but a release with this PR has no upgrade procedure for stored queries until it is written.
Earlier blocking findings
| # | Status at 9888b0e5 |
Evidence |
|---|---|---|
| D6 | Open | ranking.rs and shape/read.rs unchanged since 88bd29c3 |
| N1–N3 | Open, unchanged | The delta does not touch their code. cc3cb99c adds a second pass in the N3 class (O20) |
| N4 | Open | 4ed86485 edits upgrade.md, but the in-place rollback paragraph (upgrade.md:151-157, "or rebuild the indexes with the earlier binary") is unchanged |
| B8 | Open | served.rs only drops the removed same as reference refusal; the served runner still reads only the error line |
| D1–D4 | Open | check/mod.rs, depth.rs, dataset_index.rs and Cargo.toml unchanged in this delta |
Tradeoffs and liability
cc3cb99c removes a memory bound that grew with the corpus, at the cause: the scan no longer holds per-document state. It adds about one more analysis pass per bm25 (O20). That cost goes away when stage L shares one analyzer between the index and the scan. 3732684a removes liability. The reference comparison now has one rule for every step instead of an opt-in section, and an opt-out must say why. The main new cost is that each new form must be supported by the reference interpreter or carry a skip reason. The reference fixes add a second implementation of parameter binding and DateTime rendering (canonical_datetime). That is the purpose of a reference, and the default comparison catches any drift. 5ed97b54 and 8ef2695a remove state that did nothing.
Validation
Clean detached worktrees at 88bd29c3 (base) and 37d04499, moved to 9888b0e5 (head). The engine code is the same at 37d04499 and 9888b0e5. The later two commits change only docs, the nightly workflow and a compiler test file. GQT runners were built with cargo build -p omnigraph-gqt --locked --bins at each commit, and run with RUST_MIN_STACK=16777216.
| Proof | Command | 88bd29c3 |
head |
|---|---|---|---|
cc3cb99c: the PR's test, copied into the base |
cargo dev-nextest -E 'test(bm25_scores_a_corpus_several_times_the_pool_within_it)' |
FAIL: ResourceLimitExceeded { resource: "query_memory_bytes", limit: 4194304, actual: 4196930 } |
PASS (15.1 s) |
cc3cb99c: rows over five population shapes (temporary case) |
omnigraph-gqt /tmp/probes/bm25_population_shapes_probe.gqt |
ok, same rows | ok, and each step agrees with the reference |
| O20: work charged | the same query with set read_work_limit = N; |
answers at 15; refused at 14 (actual 15) |
answers at 30; refused at 29 (actual 30) |
| Corpus with the default reference comparison | omnigraph-gqt crates/omnigraph-gqt/cases |
— | 613 of 613 cases passed; 12 reference skips, each with a reason |
| Workspace and failpoint test build | cargo dev-build-tests |
— | compiles (at 37d04499) |
9888b0e5 round trip |
cargo test -p omnigraph-compiler --locked --lib round_trip |
— | 8 passed, 2 ignored |
The bm25 memory regression is a Rust test, not GQT. That is correct: GQT can set query_memory_bytes, but the case needs 30,000 generated documents and the memory probes that check release, which only the Rust harness has.
CI:
- At
37d04499,Test Workspacefailed in its compile step on an infrastructure error (sccache: error: Server startup failed … dns error: failed to lookup address information), not on the code. The same command compiles locally. The run was then cancelled by the push of9888b0e5. - At
9888b0e5, at the time of this review: Clippy, GQT (ordinary, served, dst, dst-clippy), the DST pinned suite, the aws server test, Format and the other quick checks pass. Test Workspace, RustFS S3 (default) and Storage Upgrade Compatibility are still running.Dependency Guard (cargo deny)fails, but it is not this PR: #1440 (a CLI-only change) fails the same check at5c74be76, and the lastmainrun passed it at 07:45Z, before the new advisories.
Limitations:
- I did not run the workspace, DST, S3 or served suites locally. I checked the O20 metrics claim by code inspection only.
- I did not measure the time of the second
bm25pass on a large corpus. - All temporary cases were outside the repository. I removed the copied test from the base worktree, both worktrees are clean, and nothing was pushed to the PR.
… its own region A region named a kind of scope (an optional over a binding), so two sibling optionals over one binding shared a region and a plan that moved a conjunct from one to the other was accepted, swapping which side is null. Each optional and correlated read is now numbered in written order within the region that holds it, on the requirement side by its step and on the plan side by a walk that numbers a node after its input side, the order the planner builds them in. The broken-plan test moves a conjunct between two sibling optionals (refused now, accepted before) and between two sibling correlated reads.
…hat read's pool and work A write group's selection read ran under its own query pool and read_work_limit, but the pool was dropped with the run: the rows the write kept, the identity and value maps an update or delete built from them and the value map an insert built per row (each a copy of the constant assignments) were charged nowhere, so a selection whose read fit could hold any amount beyond it. The selection now returns its rows held on the read's own context (engine::SelectionRead), whose pool charges the kept batches and every map built from them a batch at a time, and whose read-work budget charges one unit per row each statement reads. LoadCharge becomes PoolCharge, the one named charge on a statement's pool, with work(). The test runs, under a 4 MiB pool, a read that fits and a 6,000-row selected insert whose maps do not (refused as query_memory_bytes by 'write selection inserts', every charge released), and under read_work_limit 15000 a read that fits and an update over the same rows that does not. Both writes succeeded under those bounds before.
ragnorc
left a comment
There was a problem hiding this comment.
Delta review of #1316 at 00a88ce5
Recommendation: revise before merge. This delta adds no new blocking defect. Both fixes are correct, and each new test fails before its fix and passes after it. The earlier blocking findings (D6, N1–N4, B8, D1–D4) stay open, because this delta does not change their code. I ask for one GQT regression for the read_work_limit half of 87e65af4 (below). The PR is a draft again since 14:07Z, but its review requests are still open, so I reviewed the new commits.
Reviewed head: 00a88ce5. My last review (review 5939170541) covered everything up to 9888b0e5. This review covers the three commits after it:
96e26075plan acceptance gives eachoptionaland correlated read its own region;87e65af4what a write keeps of its selection read is charged to that read's pool and work;00a88ce5registersexport_jsonl_atas a read-only surface inforbidden_apis.rs.
No Decision needed comment exists on this PR. No inline comment or later review covers this delta.
Why, what, how, impact
Why. Plan acceptance named a region by its kind only: "the optional over $d". Two sibling optionals over one binding were one region. So a plan that moved a conjunct from one to the other was accepted, and the wrong side became null. Separately, a write group's selection read ran under its own pool and read_work_limit, but the pool was dropped when the run ended. The rows that the write kept, and the maps it built from them, were charged nowhere.
What and how. 96e26075 adds an ordinal to Region::Optional and Region::Correlate (requirements.rs:165). Both sides count them with one helper, RegionOrdinals (requirements.rs:177): the requirements in written step order, and the plan by a walk that numbers a node after its input side (invariants.rs:166). 87e65af4 returns the selection's rows as engine::SelectionRead (context.rs:426). It keeps the read's QueryContext, holds the rows on its pool, and gives each statement a PoolCharge for its maps and one unit of read_work_limit per row it reads.
Impact. A broken plan of this shape is now refused at acceptance, before it runs. A write whose selection fits the pool but whose derived maps do not is now refused with query_memory_bytes. A write whose statements read more rows than the work budget allows is now refused with read_work_limit. Under the default limits I found no write that the change newly refuses (see O26).
Blocking
None new in this delta.
GQT regression requested
The read_work_limit half of 87e65af4 is behavior that GQT can express with three nodes. The PR tests it only in the Rust test with 6,000 rows. Please add the minimal case below. I suggest extending read_work_limit_bounds_a_chain_of_hops.gqt, which already owns "every read runs under read_work_limit" (it needs note: String? on Person). A temporary copy fails at 9888b0e5 and passes at 00a88ce5:
--- mutate
set read_work_limit = 4;
--- expect ok
--- query
query picked() {
match { $a: Person, $a knows $b } where $a.handle = "a"
return { $b.handle }
}
--- expect unordered
{"b.handle":"b"}
{"b.handle":"c"}
--- expect shape
b.handle: String
--- mutate
query stamp() {
match { $a: Person, $a knows $b } where $a.handle = "a"
update $b set { note: $b.handle }
}
--- expect error: read_work_limit
--- mutate
reset read_work_limit;
--- expect ok
(seed: a, b, c; a knows b, a knows c.) The read costs 4 units. The update costs the read's 4 and 2 for its pass over the two rows, so at the head it is refused with resource limit exceeded for read_work_limit: actual 6, limit 4. At 9888b0e5 the same update succeeds (affected: nodes=2).
The memory half needs thousands of generated rows and the release probes, so the Rust test is the correct owner for it. The 96e26075 defect is acceptance of a plan that the planner never builds, so only a test that edits a plan can show it. The unit test in validate/tests.rs is the correct owner.
Optional
- O25. A region's identity is its position, and two walks compute it. The requirements count
optionals by step index, and the plan counts them by a post-order walk over input sides. The two agree only becauseresolve_pipeline(optimizer.rs:459) builds each element over the rows of the elements before it, in written order. If a later planner reorders independentoptionals or correlated reads, acceptance refuses the valid plan as an internal error. (The index-facts RFC names join order as future planner input,2026-10-08-index-facts-as-planner-input.md:660.) That failure is safe, because nothing wrong runs, but the rule is in two places.- Option: the shaper gives each
optionaland correlated read a scope id. The logical and physical nodes carry it, and acceptance compares ids instead of counting. This adds one field to two node kinds and the mirror. It removes the agreement between the two walks, and a reordered plan is still accepted when it is correct. - Tradeoff: the ordinal is cheaper today and fails closed. The id is worth its cost when the first reordering rule is written, not before.
- Option: the shaper gives each
- O26. For
insertandupdate, the write's own row bound already limits what the maps can usefully hold.inserted_entities(mutation.rs:1171) andselected_entities(mutation.rs:1094) build a map for every selected row. Only then does the staging refuse more than 8,192 rows per table (admit_keyed_rows,staging.rs:359): an insert as it stages each row, and an update after the build (mutation.rs:2594). With this delta the build is bounded by the pool. So a large selectedinsertorupdategets one of two errors, depending on its row count and row width:query_memory_bytesfromwrite selection inserts/entities, orkeyed entities for node:T. A user who raisesquery_memory_bytesonly gets the second error.deletehas no row cap, so it needs the pool charge.- Fix: let the build ask the staging's admission as the maps grow (the number of distinct entities per batch), so the write's own bound refuses first and gives the actionable error. The pool charge then stays the bound for the kept rows and the
deleteids. I checked this by code inspection only. I did not run a selection larger than 8,192 rows.
- Fix: let the build ask the staging's admission as the maps grow (the number of distinct entities per batch), so the write's own bound refuses first and gives the actionable error. The pool charge then stays the bound for the kept rows and the
- O27. The
insertcharge counts each value's inline size twice.charge.entries::<(String, Literal)>(mutation.rs:1268) already charges each map entry,Literalincluded.map_bytes(:1207) then addsliteral_bytes, which addssize_of::<Literal>()again (:1085).selected_entitiesis correct, because thereliteral_bytesmeasures theVec<Literal>storage. The effect is small: an insert is refused a little early, and its reportedactualis too high. A helper that counts only heap bytes for map values would fix it.
What I checked that holds
- The two numberings agree for every plan the planner builds. Sibling
optionals and correlated reads wrapcurrentin written order (optimizer.rs:459–861), and an uncorrelated read after other elements is the right input of aCrossJoin. The walk visits a node's first input first, so it reaches them in the same order. Correlated union arms and anearestwindow's population are keyed by their own region path on both sides.searched_side_rightswaps the sides of aCrossJoinonly when both sides are plain table scans, so it cannot reorder these regions.AntiJoinand expression blocks are not numbered on either side. - No valid plan is refused. A temporary GQT case with six queries passes at the head, and each step agrees with the reference interpreter: three sibling
optionals, a nestedoptionalbeside a sibling,optionals between two aggregatefromreads, an uncorrelatedfromfollowed by a correlated one, two orderedfromreads over one binding, and twofromreads with alet … count {}and awhere exists {}block. The CI GQT jobs (ordinary, served, dst) also pass the whole corpus at00a88ce5. SelectionReadholds the context only for the write.selectedis local toexecute_named_mutation(mutation.rs:2227), which stages and returns before its caller commits, so the pool is not held across the commit. The kept batches are charged once byhold, and each map is charged one batch at a time, before the next batch adds to it. The PR's test shows that every charge is released after a refusal.- The update and delete accounting is correct. Each id is counted twice (the order
Vecand the map key), each value once, and the entries of both collections. The keyed insert's index counts its key tuple. - The docs match the code.
writes.md,settings.md(read_work_limit,query_memory_bytes) and the changelog note describe the new charges.00a88ce5adds the read-only surface thatc0dbc286introduced, soforbidden_apislists every public read.
Earlier findings
| # | Status at 00a88ce5 |
Evidence |
|---|---|---|
| D6, N1–N4, B8, D1–D4 | Open, unchanged | This delta changes only the planner's validate/ module, engine/context.rs, engine/mod.rs, exec/mutation.rs, two loader type names, tests and three docs. None of these findings' code is among them. |
| O20–O24 | Open, unchanged | score.rs, the Score row (execution.md:666), omnigraph-gqt-core, Display and the two GQT notes are unchanged. |
Tradeoffs and liability
96e26075 removes a false acceptance at its cause: the region is now one scope, not a kind of scope. It adds one counter type, shared by both sides, and one assumption, that the plan keeps written order (O25). 87e65af4 removes a real gap: the derived state of a write had no bound except the staging limits, which an insert or update reaches only after it builds every map. The change reuses the read's pool and work budget instead of adding a new allowance, and LoadCharge becomes the one PoolCharge. This is fewer primitives, not more. Its cost is a set of hand-written size estimates in mutation.rs (literal_bytes, map_bytes) that must change when Literal or the maps change. O27 shows how easily they drift. Five more statements that keep derived state would each need such an estimate. O26 would keep that to the one place where no other bound exists.
Validation
Clean detached worktrees at 9888b0e5 (base) and 00a88ce5 (head), one shared Cargo target. Built with cargo dev-build-tests and cargo build -p omnigraph-gqt --locked --bins. GQT ran with RUST_MIN_STACK=16777216.
| Proof | Command | 9888b0e5 |
00a88ce5 |
|---|---|---|---|
96e26075: the PR's test, copied into the base |
cargo dev-test … a_conjunct_tested_in_another_scope_is_refused |
FAIL: the broken plan was not refused by a check (tests.rs:146) |
PASS |
87e65af4: the PR's test, copied into the base |
cargo dev-nextest -E 'test(write_selection_maps_are_charged_to_the_read_pool_and_work)' |
FAIL at execution_memory.rs:2422: the insert succeeds (affected_nodes: 6000) |
PASS (28.6 s) |
| GQT regression (temporary case above) | omnigraph-gqt /tmp/…/write_selection_work_probe.gqt |
FAIL: the update succeeds at read_work_limit = 4 |
ok: refused, actual 6, limit 4 |
| Valid plans (temporary case, six queries) | omnigraph-gqt /tmp/…/planner_sibling_regions_probe.gqt |
ok | ok, every step agrees with the reference |
CI at 00a88ce5: 27 checks pass and 4 are skipped, including Test Workspace, Storage Upgrade Compatibility, the DST pinned suite, RustFS S3 (default and failpoints), Dependency Guard and the GQT jobs. At 9888b0e5, Test Workspace and Storage Upgrade Compatibility had failed. I did not read why, because they pass at the head.
Limitations:
- I did not run the workspace, DST, S3 or served suites locally; the CI results above are for the exact head.
- O26 and O27 are from code inspection. I did not measure the size of
Literalor run a selection larger than 8,192 rows. - All temporary cases were outside the repository. I removed the copied tests from the base worktree, and nothing was pushed to the PR.
This replaces GQ with the language of the GQ 3 RFC (#1302) in one break, as the RFC's section 6 "The break" and section 11 "One IR" now state: one grammar, one type checker, one IR, no edition line and no old parser in the product. The RFC is a draft, so this pull request stays a draft until it is accepted; the
changelog.dnote and the user page on the break are added before merge (theskip-changeloglabel stands in meanwhile).What changed
One front end.
query::parser::parse_querychecks the nesting bound and parses the one grammar (query/syntax/grammar.pest, rules named as section 7 names them). The edition line, its dispatch andQ022are gone; a leftovergq 3;line is refused with its deletion as the fix (Q012). The old AST (query/ast.rs), the old checker (typecheck.rs), the old lowering (ir/lower.rs), edition-2 parsing (syntax/edition_window/) and the pipeline IR (QueryIR,IROp,MutationIR) are deleted.query/file.rsholds the file-level types every caller reads (QueryDeclnow carries its parse tree),query/value.rsthe operators and literals,syntax/statement.rsthe statement builders.One type checker (
query/check/), started from the former edition-3 checker. Traversal resolution (check/traversal.rs) and literal typing and type compatibility (check/literal.rs) moved in as the one implementation. A last pass (check/bind.rs) puts every expression in its bound form: system columns under their physical names, comparisons and membership throughir::coercewith each numeric literal at its own type (so the casts of invariant 9 arecoerce's, as before), literal-only nodes folded, a bare search predicate compared withtrue.check/result.rsinfers a read's result schema from the core. Restored from the deleted checker where nothing else covered them:T47($__nanograph_nowas a parameter name),T49(a__variable name),T55(on $base),T24for a Blob under an aggregate,T2for a missing property in a binding's braces.The planner reads only the core.
omnigraph-planner/src/read_shape.rsshapes aCoreReadinto the pattern steps, return items, order and limit the planner already plans: scans, expansions (a triple between two bound ends as a fresh expansion and an identity filter), filters, correlated blocks as anti-joins with their locals renamed, return projections and groups, and the rankingsfrom nearest,from bm25and therrftype-population form lowered to today's ranked scans andRankFuse. Anything else isP006naming it:Union(optional),Fix,Call, a computed order key no return item computes, a block compared with a column. Validation, plan acceptance,plan_sourceand the engine read the shape; nothing re-derives types from the text.Writes.
omnigraph/src/exec/write_plan.rslowers aCoreWrite's groups to today's staged writes: a group whose clauses are a scan of the target filtered by predicates over that variable; anupdateordeletetakes them as its selection, aninsertwith no clauses runs once, one after a matching statement once per matched row. Any other group, a search predicate in a selection and an assignment that reads a row areP006.Header consumers (server, CLI, cluster, MCP, lint, the operation descriptor, GQT's inferred schema) check through
check::check_query_decland read the core. A result descriptor's field names are now the executed column names (p.name, notname); see "What moved".The repository's query text was rewritten by the one-shot rewriter (not committed): every GQT case, the benchmarks (and the measured-step texts and pinned identities in
benchmarks/benchmarks.yamlandgqt-catalog-identity.json, which move because the texts moved),docs/user,docs/dev,skills/omnigraph, the.gqfixtures, and query strings in Rust tests and sources. The cross-version journeys keep predecessor syntax for predecessor binaries (journey-predecessor.gq) and current syntax for the current binary. Prose in the guides and the skill was brought in line by hand; blocks that are part of a source are fencedgq fragment.The five diagnostic findings, each with a test
unionbetween two reads:Q014naming it, with thefrom union({ … }, { … })form as the fix (syntax::tests::a_sql_union_between_reads_prints_the_relation_function).T74names the return item behind an unshaped column (count($p), not the columnp) and its fix aliases that item (t74_names_the_return_item_behind_an_unshaped_column).T8for an aggregate in a filter prints the block form over the clauses it counts (count($f) { $p knows $f } > 3), or, over the read's own rows, thereturnitem (t8_prints_the_block_an_aggregate_filter_means).T65carries a machine-applicable suggestion that draws the ranking withfromover the read's own pattern, the form the rewriter writes (t65_suggests_an_applicable_edit, applied and reparsed fornearest,bm25andrrf).a_type_error_carries_the_position_of_its_clause).Proof that meaning did not move
Before rewriting, a temporary recorder (not committed) captured every GQT step's rows, result schema, explain document and bound plan from the main-equivalent build: 4,119 records over 515 cases. After the change all 535 cases pass, and the comparison gives:
What moved, and why
Error codes (every one either the RFC's fixed map, a Compatibility row, or a deliberate section 5 code):
issue_640_search_score_projection6, 7T33T65returnwithout its rank key (left unrewritten; the call form isT65)issue_640_search_score_projection8T32T65T65rank_key_asc_direction_refused2, 3T54T65ascon a rank key stays refused through the code mapsearch_order_on_an_unranked_binding_and_a_destination1, 2T59T75T75(section 4)boolean_literal_refused_where_a_property_is_named_false1T46Q008delete … whereform (the rewriter refuses it: in the new language the literal would select every row);T46cannot arise and is withdrawnbare_node_projection_object_fields7;issue_659_edge_selection_contract14;issue_640_search_score_projection11;expression_type_errors7, 8T8,T41,T36,T45Q018expression_type_errors3;issue_764_relationship_count_filter1;issue_955_aggregate_comparison12T7T8T8with the block form as its fix (diagnostic finding)expression_type_errors6, 10T44T8T8expression_type_errors9T44P006issue_566_order_key_binds_to_return_item6T42P006issue_764_relationship_count_filter9T40P006T40lifted); the planner does not lower itsubquery_column_aggregate_predicate14T40T7T40lifted: aDateblock value against an integer is the comparison errorT7issue_955_aggregate_comparison13T58T9catalogue/mixed_joins_and_refusals5T23T27match_textover a catalogue binding:T27names the full-text index it lackslist_membership_over_indexed_columns_across_writes_and_restart1T7T6$r in $iwith$iunbound and no edge namedin: the unbound variable is the refusalmeta_fields_resolve_by_role28–31, 36T11T6issue_620_duplicate_projection_alias_refused1–9;issue_953_return_distinct8, 9;issue_954_count_distinct10, 11T25,T56,T42,T57issue_801_in_predicate_over_a_list5T7T7inError texts changed where the code did not (no compatibility with the old messages, as asked); the expectations were updated mechanically to the new text.
Rows (each one the Compatibility section names):
issue_659_compiler_scopes11–13 andissue_659_edge_selection_contract4 (a triple with neither end bound or an inferred wildcard end,T5, the RFC's four cases);expression_type_errors14 (a Boolean item beside aggregates,T9). Their expected rows were added from the run.T42:meta_fields_resolve_by_role9 andissue_955_aggregate_comparison22, 23 (an order key over groups that is a meta-field neither returned nor a key, the RFC's three cases).issue_640_search_score_projection2, 3 (p._distance,p._scorebecome_distance,_score).expect unordered:edge_binding_filters_and_projects_edge_properties2, 6;unkeyed_edge_dup_delete_after_merge6 (bound plans identical).Bound plans (25 steps differ; no version advanced):
issue_659_rrf_typed_edge_identity1–3,issue_787_rrf_order_keys_after_fused_score3,issue_788_search_led_aggregate_order_keys4, 5,rrf_cross_join_batch_order_preserves_rank1,typed_search_options7,v2/rrf_narrow_limit_memory1,v2/rrf_tied_rows_under_limit1–3, and the refusals above;issue_640_search_score_projection2, 3;count($n) as n,$x as x, …):return_exprs[].aliasisNonewhere the alias equals the column's own name. The RFC's formatter drops such an alias "and never changes the typed IR", so the core cannot tell them apart;boolean_parameters_joined_by_and_or_are_filters1, 2:$a and $bis two filters, not one; the formatter splits a top-levelandinto clauses without changing the IR, so the core holds the split form;v2/planner/contains_beside_search_and_ranking3: the scans' binding numbers swap (therrftype population binds$pfirst). The plan's nodes, order and rows are unchanged.Explain documents differ in 78 steps: the rows above, and in 51 steps only
logical_plan_initial, the pre-optimizer tree. The core places each filter after the last binder of a variable it names (section 3), where the old lowering kept the written position; the optimized logical plan, the physical plan and the bound plan are identical in those steps.Replay envelopes carry the query source, so a rewritten case's envelope differs by its text; where the source did not change, the envelope is identical with its bound plan.
Hand edits. The seventeen fusion cases the Compatibility section lists:
issue_659_rrf_typed_edge_identity(3 reads),issue_787_rrf_order_keys_after_fused_score(1),issue_788_search_led_aggregate_order_keys(2),rrf_cross_join_batch_order_preserves_rank(1),typed_search_options(4),v2/rrf_narrow_limit_memory(1),v2/rrf_tied_rows_under_limit(3) with duplicate arms, given a distinct second arm that selects the same rows or rows no join reaches (each case's notes say which); andv2/planner/search_order_on_an_unranked_binding_and_a_destination(2) with arms over two bindings, now theT75refusal. Outside the corpus the RFC counted, Rust tests held 9 more duplicate-arm fusions and 1 two-binding one (engine_v2.rs,engine_v2_memory.rs×2,engine_v2_plan_replay.rs×2,search.rs×3,validate/tests.rs×2,rrf_prefilter_gate.rs), edited the same way. Also by hand:diagnostics/p006_unlowered_read_does_not_runnow usesoptional(the read it used is lowered now); the olddelete … wheretext inboolean_literal_refused_where_a_property_is_named_falseis left for the rewriter's refusal; two Rust test queries namedsearchandtermswere renamed (a declared read may not shadow a registry name,T74).How it was tested
omnigraph-compiler(312 tests): the rule-set pin over the whole grammar; one repair row per distinct refusal shape, each suggestion applied and reparsed; the RFC's examples type-checked as written; the parse, print and parse property; the desugaring-uniqueness test against the bound form; the five findings above.omnigraph-gqt: the complete corpus (535 cases) and the newtests/integration/query_corpus.rs, which parses every documentedgqblock and.gqfile and type-checks each.gqfile that has a.pgbeside it (it replacesedition_corpus.rs).--features omnigraph-engine/failpoints,omnigraph-cluster/failpoints), the DST suite fromcrates/omnigraph-dstwith its storage-cost golden unchanged, andomnigraph-bench. Locally the only failures are tests that need a non-root user or/proc/self/statusCpus_allowed_list(permission and symlink refusals, machine identity), which the sandbox lacks.cargo fmt --check,cargo dev-clippy, GQT clippy,typos,check-docs.py,check-agents-md.sh,check-ci-cells.py, the workflow-pin and dependency-source checks.RFC findings
T74) refuses existing texts (queries namedsearch,terms), which the Compatibility section's "everything else is additive" does not mention.bm25's score type. Section 2 typesbm25F64(itsbm25_v1scorer); today's scans return Lance'sF32, and the planner lowers to them, so the checker types the scoreF32untilbm25_v1runs. Otherwise every bm25 shape would move.nearestquery on a field without@embed. The checker had required@embed; today the default provider embeds it. Section 4's default recipe has no.pgspelling yet (unresolved question 6), so the checker accepts it as today.and; the core therefore cannot reproduce the old plan's alias field or single filter (10 bound-plan steps above). The canonical filter placement of section 3 also changes the pre-optimizer logical plan in 51 explain documents.T58's text (count($p) >= $f.min_members), but section 3's derived forms do not say what a non-key column read after theGroupmeans. Refused withT9here, not given a meaning.insertafter a matching statement in one group runs once per matched row (section 2's "once per row of its group's relation"); seven stored declarations in two cases rely on it, and the RFC's group rule says so only for the rewritten form.$p.age > }reports column 67, not 46); the old parser named the operand. Section 5's positions need a recognizer-aware error position.query.pest; the grammar file isgrammar.pest.rrf'skfrom a parameter (k: $k) is accepted, as the oldrrf(a, b, $k)was; section 4 listskas an integer literal.orderin its direction and whosenearestwindow is thelimit; a read asking for another window, direction or order, alimitwithout the score, or an aggregate over the score isP006(ranking_shapes_the_planner_does_not_lower.gqt). Section 11 does not list these as later stages.