Skip to content

feat(compiler,planner,engine)!: replace the query language with GQ 3 in one break - #1316

Draft
ragnorc wants to merge 221 commits into
ModernRelay:mainfrom
ragnorc:gq3/phase-0-front-end
Draft

ragnorc wants to merge 221 commits into
ModernRelay:mainfrom
ragnorc:gq3/phase-0-front-end

Conversation

@ragnorc

@ragnorc ragnorc commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

This replaces GQ with the language of the GQ 3 RFC (#1302) in one break, as the RFC's section 6 "The break" and section 11 "One IR" now state: one grammar, one type checker, one IR, no edition line and no old parser in the product. The RFC is a draft, so this pull request stays a draft until it is accepted; the changelog.d note and the user page on the break are added before merge (the skip-changelog label stands in meanwhile).

What changed

One front end. query::parser::parse_query checks the nesting bound and parses the one grammar (query/syntax/grammar.pest, rules named as section 7 names them). The edition line, its dispatch and Q022 are gone; a leftover gq 3; line is refused with its deletion as the fix (Q012). The old AST (query/ast.rs), the old checker (typecheck.rs), the old lowering (ir/lower.rs), edition-2 parsing (syntax/edition_window/) and the pipeline IR (QueryIR, IROp, MutationIR) are deleted. query/file.rs holds the file-level types every caller reads (QueryDecl now carries its parse tree), query/value.rs the operators and literals, syntax/statement.rs the statement builders.

One type checker (query/check/), started from the former edition-3 checker. Traversal resolution (check/traversal.rs) and literal typing and type compatibility (check/literal.rs) moved in as the one implementation. A last pass (check/bind.rs) puts every expression in its bound form: system columns under their physical names, comparisons and membership through ir::coerce with each numeric literal at its own type (so the casts of invariant 9 are coerce's, as before), literal-only nodes folded, a bare search predicate compared with true. check/result.rs infers a read's result schema from the core. Restored from the deleted checker where nothing else covered them: T47 ($__nanograph_now as a parameter name), T49 (a __ variable name), T55 (on $base), T24 for a Blob under an aggregate, T2 for a missing property in a binding's braces.

The planner reads only the core. omnigraph-planner/src/read_shape.rs shapes a CoreRead into the pattern steps, return items, order and limit the planner already plans: scans, expansions (a triple between two bound ends as a fresh expansion and an identity filter), filters, correlated blocks as anti-joins with their locals renamed, return projections and groups, and the rankings from nearest, from bm25 and the rrf type-population form lowered to today's ranked scans and RankFuse. Anything else is P006 naming it: Union (optional), Fix, Call, a computed order key no return item computes, a block compared with a column. Validation, plan acceptance, plan_source and the engine read the shape; nothing re-derives types from the text.

Writes. omnigraph/src/exec/write_plan.rs lowers a CoreWrite's groups to today's staged writes: a group whose clauses are a scan of the target filtered by predicates over that variable; an update or delete takes them as its selection, an insert with no clauses runs once, one after a matching statement once per matched row. Any other group, a search predicate in a selection and an assignment that reads a row are P006.

Header consumers (server, CLI, cluster, MCP, lint, the operation descriptor, GQT's inferred schema) check through check::check_query_decl and read the core. A result descriptor's field names are now the executed column names (p.name, not name); see "What moved".

The repository's query text was rewritten by the one-shot rewriter (not committed): every GQT case, the benchmarks (and the measured-step texts and pinned identities in benchmarks/benchmarks.yaml and gqt-catalog-identity.json, which move because the texts moved), docs/user, docs/dev, skills/omnigraph, the .gq fixtures, and query strings in Rust tests and sources. The cross-version journeys keep predecessor syntax for predecessor binaries (journey-predecessor.gq) and current syntax for the current binary. Prose in the guides and the skill was brought in line by hand; blocks that are part of a source are fenced gq fragment.

The five diagnostic findings, each with a test

  • SQL union between two reads: Q014 naming it, with the from union({ … }, { … }) form as the fix (syntax::tests::a_sql_union_between_reads_prints_the_relation_function).
  • T74 names the return item behind an unshaped column (count($p), not the column p) and its fix aliases that item (t74_names_the_return_item_behind_an_unshaped_column).
  • T8 for an aggregate in a filter prints the block form over the clauses it counts (count($f) { $p knows $f } > 3), or, over the read's own rows, the return item (t8_prints_the_block_an_aggregate_filter_means).
  • T65 carries a machine-applicable suggestion that draws the ranking with from over the read's own pattern, the form the rewriter writes (t65_suggests_an_applicable_edit, applied and reparsed for nearest, bm25 and rrf).
  • Type errors carry a position: the parse tree records spans for a read's clauses, return items, order keys and limit and a write group's clauses and statements, and a refusal with no position of its own takes the one it was found in (a_type_error_carries_the_position_of_its_clause).

Proof that meaning did not move

Before rewriting, a temporary recorder (not committed) captured every GQT step's rows, result schema, explain document and bound plan from the main-equivalent build: 4,119 records over 515 cases. After the change all 535 cases pass, and the comparison gives:

  • bound plans byte-identical (versions normalised) for 2,384 steps; 436 refusals keep their code;
  • explain documents identical for 2,339 steps; the 78 that differ are listed below.

What moved, and why

Error codes (every one either the RFC's fixed map, a Compatibility row, or a deliberate section 5 code):

Case, step Was Now Why
issue_640_search_score_projection 6, 7 T33 T65 fixed map: a ranking in return without its rank key (left unrewritten; the call form is T65)
issue_640_search_score_projection 8 T32 T65 fixed map: a ranking under an aggregate, left unrewritten, is T65
rank_key_asc_direction_refused 2, 3 T54 T65 fixed map: asc on a rank key stays refused through the code map
search_order_on_an_unranked_binding_and_a_destination 1, 2 T59 T75 hand-edited fusion: arms over two bindings are T75 (section 4)
boolean_literal_refused_where_a_property_is_named_false 1 T46 Q008 the refused text stays in the old delete … where form (the rewriter refuses it: in the new language the literal would select every row); T46 cannot arise and is withdrawn
bare_node_projection_object_fields 7; issue_659_edge_selection_contract 14; issue_640_search_score_projection 11; expression_type_errors 7, 8 T8, T41, T36, T45 Q018 section 5: a bare identifier is not a value; refused at the word
expression_type_errors 3; issue_764_relationship_count_filter 1; issue_955_aggregate_comparison 12 T7 T8 an aggregate in a filter is T8 with the block form as its fix (diagnostic finding)
expression_type_errors 6, 10 T44 T8 a write's selection is a filter clause; an aggregate there is T8
expression_type_errors 9 T44 P006 a search predicate in a write's selection is a legal filter the staged write does not lower
issue_566_order_key_binds_to_return_item 6 T42 P006 Compatibility: an order key that is not a return item is accepted; a computed key no return item computes is not lowered
issue_764_relationship_count_filter 9 T40 P006 Compatibility: a block compared with a column is accepted (T40 lifted); the planner does not lower it
subquery_column_aggregate_predicate 14 T40 T7 T40 lifted: a Date block value against an integer is the comparison error T7
issue_955_aggregate_comparison 13 T58 T9 RFC unspecified (finding 6): a column beside an aggregate that no item groups by is refused, not given a meaning
catalogue/mixed_joins_and_refusals 5 T23 T27 match_text over a catalogue binding: T27 names the full-text index it lacks
list_membership_over_indexed_columns_across_writes_and_restart 1 T7 T6 $r in $i with $i unbound and no edge named in: the unbound variable is the refusal
meta_fields_resolve_by_role 28–31, 36 T11 T6 a write's selection is an ordinary filter clause; a missing property or meta-field there is the read code
issue_620_duplicate_projection_alias_refused 1–9; issue_953_return_distinct 8, 9; issue_954_count_distinct 10, 11 runtime message T25, T56, T42, T57 refused at type check with its code instead of at execution
issue_801_in_predicate_over_a_list 5 T7 T7 unchanged code; the quoted-edge fix is printed for an unbound name beside an edge named in

Error texts changed where the code did not (no compatibility with the old messages, as asked); the expectations were updated mechanically to the new text.

Rows (each one the Compatibility section names):

  • refused texts now accepted: issue_659_compiler_scopes 11–13 and issue_659_edge_selection_contract 4 (a triple with neither end bound or an inferred wildcard end, T5, the RFC's four cases); expression_type_errors 14 (a Boolean item beside aggregates, T9). Their expected rows were added from the run.
  • accepted texts now refused, T42: meta_fields_resolve_by_role 9 and issue_955_aggregate_comparison 22, 23 (an order key over groups that is a meta-field neither returned nor a key, the RFC's three cases).
  • result column names, the fixed map: issue_640_search_score_projection 2, 3 (p._distance, p._score become _distance, _score).
  • row order only, every one under expect unordered: edge_binding_filters_and_projects_edge_properties 2, 6; unkeyed_edge_dup_delete_after_merge 6 (bound plans identical).

Bound plans (25 steps differ; no version advanced):

  • the 17 hand-edited fusions (their second arm changed): issue_659_rrf_typed_edge_identity 1–3, issue_787_rrf_order_keys_after_fused_score 3, issue_788_search_led_aggregate_order_keys 4, 5, rrf_cross_join_batch_order_preserves_rank 1, typed_search_options 7, v2/rrf_narrow_limit_memory 1, v2/rrf_tied_rows_under_limit 1–3, and the refusals above;
  • the fixed-map rename in issue_640_search_score_projection 2, 3;
  • a redundant alias is not recorded (8 steps: count($n) as n, $x as x, …): return_exprs[].alias is None where the alias equals the column's own name. The RFC's formatter drops such an alias "and never changes the typed IR", so the core cannot tell them apart;
  • boolean_parameters_joined_by_and_or_are_filters 1, 2: $a and $b is two filters, not one; the formatter splits a top-level and into clauses without changing the IR, so the core holds the split form;
  • v2/planner/contains_beside_search_and_ranking 3: the scans' binding numbers swap (the rrf type population binds $p first). The plan's nodes, order and rows are unchanged.

Explain documents differ in 78 steps: the rows above, and in 51 steps only logical_plan_initial, the pre-optimizer tree. The core places each filter after the last binder of a variable it names (section 3), where the old lowering kept the written position; the optimized logical plan, the physical plan and the bound plan are identical in those steps.

Replay envelopes carry the query source, so a rewritten case's envelope differs by its text; where the source did not change, the envelope is identical with its bound plan.

Hand edits. The seventeen fusion cases the Compatibility section lists: issue_659_rrf_typed_edge_identity (3 reads), issue_787_rrf_order_keys_after_fused_score (1), issue_788_search_led_aggregate_order_keys (2), rrf_cross_join_batch_order_preserves_rank (1), typed_search_options (4), v2/rrf_narrow_limit_memory (1), v2/rrf_tied_rows_under_limit (3) with duplicate arms, given a distinct second arm that selects the same rows or rows no join reaches (each case's notes say which); and v2/planner/search_order_on_an_unranked_binding_and_a_destination (2) with arms over two bindings, now the T75 refusal. Outside the corpus the RFC counted, Rust tests held 9 more duplicate-arm fusions and 1 two-binding one (engine_v2.rs, engine_v2_memory.rs ×2, engine_v2_plan_replay.rs ×2, search.rs ×3, validate/tests.rs ×2, rrf_prefilter_gate.rs), edited the same way. Also by hand: diagnostics/p006_unlowered_read_does_not_run now uses optional (the read it used is lowered now); the old delete … where text in boolean_literal_refused_where_a_property_is_named_false is left for the rewriter's refusal; two Rust test queries named search and terms were renamed (a declared read may not shadow a registry name, T74).

How it was tested

  • omnigraph-compiler (312 tests): the rule-set pin over the whole grammar; one repair row per distinct refusal shape, each suggestion applied and reparsed; the RFC's examples type-checked as written; the parse, print and parse property; the desugaring-uniqueness test against the bound form; the five findings above.
  • omnigraph-gqt: the complete corpus (535 cases) and the new tests/integration/query_corpus.rs, which parses every documented gq block and .gq file and type-checks each .gq file that has a .pg beside it (it replaces edition_corpus.rs).
  • The canonical workspace graph (--features omnigraph-engine/failpoints,omnigraph-cluster/failpoints), the DST suite from crates/omnigraph-dst with its storage-cost golden unchanged, and omnigraph-bench. Locally the only failures are tests that need a non-root user or /proc/self/status Cpus_allowed_list (permission and symlink refusals, machine identity), which the sandbox lacks.
  • The cross-version journeys with real predecessors (0.7.2, 0.8.1, 0.9.0, 0.10.0, 0.11.0): 14 pass; the V5 fence, Lance 11 and stamp-13 journeys need binaries built from pinned commits and ran in CI only.
  • cargo fmt --check, cargo dev-clippy, GQT clippy, typos, check-docs.py, check-agents-md.sh, check-ci-cells.py, the workflow-pin and dependency-source checks.

RFC findings

  1. Rust tests outside the counted corpus. The Compatibility section counts 17 hand-edited fusions over the stored declarations; Rust test sources held 10 more (9 duplicate-arm, 1 two-binding), edited the same way.
  2. A declared read may not shadow a registry name (section 5, T74) refuses existing texts (queries named search, terms), which the Compatibility section's "everything else is additive" does not mention.
  3. bm25's score type. Section 2 types bm25 F64 (its bm25_v1 scorer); today's scans return Lance's F32, and the planner lowers to them, so the checker types the score F32 until bm25_v1 runs. Otherwise every bm25 shape would move.
  4. A String nearest query on a field without @embed. The checker had required @embed; today the default provider embeds it. Section 4's default recipe has no .pg spelling yet (unresolved question 6), so the checker accepts it as today.
  5. Formatter invariance makes some plan bytes move. The formatter "never changes the typed IR" yet drops a redundant alias and splits a top-level and; the core therefore cannot reproduce the old plan's alias field or single filter (10 bound-plan steps above). The canonical filter placement of section 3 also changes the pre-optimizer logical plan in 51 explain documents.
  6. A column beside an aggregate. Compatibility accepts T58's text (count($p) >= $f.min_members), but section 3's derived forms do not say what a non-key column read after the Group means. Refused with T9 here, not given a meaning.
  7. Writes over a group. An insert after a matching statement in one group runs once per matched row (section 2's "once per row of its group's relation"); seven stored declarations in two cases rely on it, and the RFC's group rule says so only for the rewritten form.
  8. The Wire row says the explain and bound-plan versions advance once; with byte-identical plans as the bar, neither advanced here. They advance when a plan gains a node the old IR had no form for.
  9. Positions of parse errors. For some refusals the grammar's recognizers move pest's deepest failure to the end of the source ($p.age > } reports column 67, not 46); the old parser named the operand. Section 5's positions need a recognizer-aware error position.
  10. Section 11 rule 1 names query.pest; the grammar file is grammar.pest.
  11. rrf's k from a parameter (k: $k) is accepted, as the old rrf(a, b, $k) was; section 4 lists k as an integer literal.
  12. What the planner lowers of a ranking. Today's ranked scans run a ranking whose score leads the order in its direction and whose nearest window is the limit; a read asking for another window, direction or order, a limit without the score, or an aggregate over the score is P006 (ranking_shapes_the_planner_does_not_lower.gqt). Section 11 does not list these as later stages.

Phase 0 of the GQ 3 RFC (draft): the compiler's half of edition 3. The
one front door, parse_query, reads the edition line: no line or `gq 2;`
is edition 2, unchanged; `gq 3;` selects the edition-3 parser, whose
refusals name the construct and carry byte-range suggestions; any other
line is Q006 with `gq 3;` as the fix.

An edition-3 declaration is typed into the core IR (ir::core3) for every
form of sections 2 to 4 and 7, recursion typed to Fix, and the planner
refuses it with P006 (plan_edition3) at each engine door before a
snapshot is captured. Nothing in edition 3 executes.

Tests: the rule-set pin, the red-team corpus with pinned repaired trees
(4 listed mismatches), the parse-print-parse property (2,000 cases, and
100,000 as a hunt), the desugaring-uniqueness test and the closure
table, the before-and-after asset type-checked, edition-2 texts lowered
to the same IR as before, a zero-I/O engine test, and one GQT case per
code. The GQT runner keeps a step the compiler refuses at parse under
`expect error:` and renders parse diagnostics with their code.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of #1316 at 81d69a1d

Recommendation: revise one CI gap before this leaves draft. I found 1 blocking item and 4 optional items. The edition-2 path is unchanged in every test I ran. The engine doors refuse edition 3 before they read a snapshot.

Reviewed head: 81d69a1d (two commits on main 3f249cc0). I reviewed 2c3c7a5b first. 81d69a1d came in during the review, and I reviewed that delta too. No other review or comment covers either head.

Why, what, how, impact

Why. RFC #1302 (head d0155e50) needs a compiler front end for edition 3 before any planner work can start.

What. A gq 3; line selects a new parser and type checker, which produce the typed core IR (ir/core3.rs). The planner refuses every edition-3 declaration with P006. A source with no edition line, or with gq 2;, takes the old edition-2 path.

How. parse_query_diagnostic reads the edition line first. QueryDecl::edition3 carries the edition-3 tree, and QueryDecl::is_mutation() now classifies a declaration for the server, lint and GQT. Omnigraph::refuse_edition3 runs at the four Session read doors before capture_read_view. The mutation, catalogue and compile paths each keep a backstop.

Impact. Edition 2 sees one extra trivia scan per source, plus edition_near_miss. No edition-2 text starts with gq, so no edition-2 result changes. The corpus walk (more than 2,500 texts) shows this. Edition 3 executes nothing.

Blocking

B1. A docs-only pull request can break edition_corpus without any CI job running it.

81d69a1d correctly fixes the first head's failure. At 2c3c7a5b, scripts/check-change-classes.py refused 7 compiler-test literals that read crates/omnigraph-gqt/cases (exit 1). At 81d69a1d it passes ("reader closure holds ... 5 listed readers"). The tests now live in the crate that owns the corpus, and gq-logic-tests.yml:223 runs them. A cases-only pull request now runs them too.

But edition2_corpus() (crates/omnigraph-gqt/tests/integration/edition_corpus.rs:95-120) also reads every fenced gq block in docs/**/*.md, except docs/rfcs. The classifier arm docs/*.md|...) ;; in ci.yml sets no run flag for those files. Only docs/user/queries/catalogue.md runs CI. So the class is no longer closed. The same rule applies here as for the corpus: "a class is sound only while the set of jobs reading its paths is closed" (check-change-classes.py docstring; docs/dev/ci.md, Changing CI rule 3). The sweep cannot see this reader, because it builds the path at runtime (root().join("docs")). The sweep's docstring makes such a reader the reviewer's to list.

Demonstration at 81d69a1d. I added this example to docs/user/queries/editions.md, the page this PR adds, and it is the obvious next edit to that page:

```gq
gq 3;
query people() { match { $p: Person } return { $p.name } }
```
  • cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --test integration -- edition_corpus: every_edition2_text_in_the_repository_parses_as_before FAILED (assertion left == right failed: docs/user/queries/editions.md#1). The other 3 passed. Without the edit, all 4 pass.
  • The Classify changed paths script from ci.yml, replayed with changed=(docs/user/queries/editions.md): run_full_ci=false, run_gqt=false, run_deployment=false.

So the pull request merges green and main goes red at the next full run. Other docs examples have the same effect, by code reading: a block that edition 3 refuses with Q018 changes the pinned bare_names list, and a block whose refusal code has no rewriter row fills unexpected.

Options, smallest first:

  1. Walk only the .gq files under docs/. A change to one of them already runs full CI, because it falls to the * arm. The fenced blocks stop being corpus input.
  2. Keep the fenced blocks and close the class. Make a changed docs/*.md that holds a gq fence set run_gqt=true, and add a replay fixture for it to check-change-classes.py (for example "docs with a gq fence" → ('false', 'true', 'false')). Name edition_corpus.rs as a docs reader beside it. This keeps the coverage, but it adds a content check to the classifier.

Option 1 removes the liability. Option 2 keeps the evidence, at the cost of one more classifier rule to maintain. GQT cannot express this contract. The owner of the regression is the classifier replay in check-change-classes.py.

Optional

O1. lint and the stored-query validators disagree about one declaration. For a valid edition-3 declaration, lint_query_file reports status: error, with the uncoded Plan message from descriptor.rs:94 ("its operation descriptor waits for the planner"). cluster validate (omnigraph-cluster/src/config.rs:1413), the server registry check (omnigraph-server/src/queries.rs:265) and queries validate call only typecheck_query_decl, which returns Ok(Edition3). So a cluster can publish a stored query that lint refuses and that can only ever answer P006. It also appears in the MCP catalog. I suggest one answer for "can this release serve this declaration": P006, with its code, from lint and from registry validation alike. If you intend to register edition-3 stored queries in the preview, say so in editions.md, and make lint agree.

O2. The corpus tests pin exact counts over a corpus that grows. the_edition2_corpus_type_checks_alike_in_both_editions pins "edition 2 ok / edition 3 T65": 156 and eight more counts. the_edition2_corpus_read_as_edition_3 pins the four-file bare_names list. I copied issue_640_search_score_projection.gqt to a new case at 2c3c7a5b: T65 went from 156 to 164, the bare_names list gained the new file, and both tests failed. The same pins moved unchanged to 81d69a1d (I compared the assertions). After 81d69a1d, the failure lands in the case author's pull request, which is the right place. But every future GQT case with a ranking in order, or a Q018 shape, must also edit edition_corpus.rs. Five more search cases mean five edits there. Pinning which rules occur, plus a lower bound, keeps the RFC evidence and removes that coupling. So does holding the exact counts on a frozen copy of the corpus.

O3. One concept is checked in five places. "An edition-3 declaration does not reach an edition-2 lowerer" is checked at refuse_edition3 (four doors), compile_named_query (query_doors.rs:418), the two mutation paths (mutation.rs:1092, :1590) and catalogue.rs:117. The pre-snapshot door check is needed for the zero-I/O contract. The backstops all repeat if decl.edition3.is_some() { return Err(edition3_refusal(..)) } before an edition-2 check. One engine function that returns the edition-2 checked query, or the P006 refusal, would make a new door correct by construction. The planner already depends on the compiler, so the engine is the right owner.

O4. The door checks against the handle's catalog, not the read target's. refuse_edition3 uses self.catalog() before capture_historical_read_view(version). For an old snapshot this can answer a T code where the target's schema would answer P006. This is harmless while edition 3 only refuses, and docs/dev/execution.md states it. When a later stage executes edition 3, the check must move behind view capture (invariant 3, one coherent accepted view).

Small things: edition_near_miss skips a set line with rest.find(';'), so a ; inside a string or comment ends the line early. edition_line and edition_near_miss also treat whitespace differently between gq and the number (skip_trivia against trim_start). Neither changes an edition-2 result.

Tradeoffs and liability

This adds about 24,700 lines: about 11,000 of compiler and planner code (parser3.rs and typeck3.rs hold 8,450 of them), about 5,300 of Rust tests, and the rest test corpora. The language executes nothing yet. The edition-2 cost is low: one prefix scan per source, and the equality walk over the corpus pins it. The real liability is a second grammar and type checker that must follow every RFC revision until edition 3 runs. #1244 also proposes an additive alternative without a language break. Keeping this a draft until #1302 is accepted is the correct gate. Merging earlier would turn RFC edits into code edits. QueryDecl::is_mutation() replacing the scattered mutations.is_empty() checks removes liability, and it is worth keeping whatever happens to edition 3.

Validation

All in a clean detached worktree. I restored every temporary edit.

  • cargo test -p omnigraph-compiler --lib --locked at 2c3c7a5b: 560 passed, 3 ignored (the description says 561).
  • cargo test -p omnigraph-planner --lib --locked -- edition3: 1 passed.
  • At 81d69a1d: cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --test integration -- edition_corpus: 4 passed. --test gq_logic_tests -- gq3: 69 passed.
  • python3 scripts/check-change-classes.py: exit 1 at 2c3c7a5b (7 FAIL lines), exit 0 at 81d69a1d.
  • B1 and O2 demonstrations: as described above.
  • Exact-head CI at 81d69a1d, when I posted: 27 pass, 4 skipped, 0 fail, including Test Workspace and the GQT jobs. At 2c3c7a5b, Check Workflow Action Pins failed on the class sweep that 81d69a1d fixes.
  • Not run locally: the engine test warm_read_cost::an_edition3_declaration_is_refused_before_any_storage_access. It needs the engine's test feature graph, so the evidence for it is the passing Test Workspace run at this head. No performance claims. The change reads no Lance storage, so no Lance domain applies.

…est to its code

Engine v1, its frozen pin, the session's read-executor seam it ran
through, the compiler's untyped read IR and GQT's `expect same as v1`
section go; every case keeps its pinned rows. The red-team corpus and the
RFC asset copies leave the tree: each distinct refusal shape the corpus
found is a repair row in the parser or checker tests, and the RFC's
examples are type-checked inline. The release note, the release.json
retarget and the user page are withdrawn until the RFC is accepted.
@ragnorc ragnorc added the skip-changelog Typed change with nothing users can see; waives the changelog.d note check label Nov 10, 2026

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of #1316 at f6946f58

Recommendation: keep as draft. B1 from my review at 81d69a1d is still open, by a different path. I also recommend two scope changes. The new commits are correct as far as I tested them. The v1 retirement removes real liability, and every step that compared with v1 still pins its rows.

Reviewed head: f6946f58. My last review covered 81d69a1d (review). Since then:

  • b4ab1564 and 562a3bca merge main. The only conflict was Cargo.lock (the deleted crate), and git show --remerge-diff 562a3bca shows nothing else. changelog.d/release.json at the head is the same as on main.
  • 75cb05fc is the release-note retarget. ec18d6fa withdraws it again with the note and the user page.
  • ec18d6fa retires engine v1 and moves the red-team corpus and the RFC asset copies out of the tree (474 files, +1,431/−14,978).
  • f6946f58 removes unused imports. It also removes the #[cfg(feature = "test-util")] that ec18d6fa left above pub mod retention in omnigraph-catalog/src/lib.rs. Squash merge makes that intermediate state harmless.

No other review or comment covers these commits.

Why, what, how, impact

Why. The draft RFC #1302 (now at 88cb905d) puts "One IR" in its blocked_on list. It also says the red-team corpus and the competence instrument stay outside the repository.

What. This change deletes crates/omnigraph-reference-engine (about 6,700 lines with its frozen pin), the ReadExecutor seam in Session and Detached, the engine's test-util feature, ir/untyped.rs and ir/erase.rs (1,140 lines), and GQT's --- expect same as v1 section. It also deletes 397 red-team corpus files and parser3_redteam.rs. parser3_tests.rs gets 95 parse repair rows and typeck3_tests.rs gets 5 type-level rows instead.

How. The GQT parser now refuses --- expect same as v1 as an unknown expect mode (omnigraph-gqt-core/src/lib.rs:1077), so a leftover section fails loudly. forbidden_apis.rs loses the v1 guards. split_crate_test_util_is_enabled_only_by_dev_dependencies stays, and it is now stricter: no regular dependency may enable any test-util.

Impact. Production builds do not change. GQT loses its independent second engine. The cost is low: I checked all 34 steps that carried expect same as v1 (15 case files) at ec18d6fa^, and each one also has its own expect rows, unordered or ordered section. So each step still pins its answer. What goes is the second opinion when someone re-pins rows. No other open PR adds the section or uses the removed APIs. I scanned every open PR's diff, and only RFC prose in #1302 mentions it.

Blocking

B1 (still open). A docs-only pull request can still break edition_corpus without any CI job running it.

corpus() now keeps only texts with no edition line (edition_corpus.rs:141). This closes the gq 3; example I used last time. I re-ran it at f6946f58, and all 4 tests pass. But corpus() still reads every fenced gq block in docs/**/*.md outside docs/rfcs (edition_corpus.rs:115-140). the_edition2_corpus_read_as_edition_3 then pins the exact bare_names list (:266-273) and requires unexpected to be empty. The docs/*.md arm in ci.yml:103 still sets no run flag. So the class is still not closed.

Demonstration at f6946f58. I added a T45 example to docs/user/queries/diagnostics.md. It is the kind of example a diagnostics page shows, and it is the same text as a step in expression_type_errors.gqt:

```gq
query property_leaf_in_an_inline_match() {
    match { $p: Person { adult: age > 30 } }
    return { $p.slug }
}
```
  • cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --test integration -- edition_corpus: the_edition2_corpus_read_as_edition_3 FAILED. bare_names gained docs/user/queries/diagnostics.md#…. The other 3 tests passed. Without the edit, all 4 pass.
  • The ci.yml classifier, replayed with changed=(docs/user/queries/diagnostics.md): run_full_ci=false run_gqt=false run_deployment=false.

A bare-name example (Q018) does this, and so does any block that edition 3 refuses with a code that has no rewriter row. The two options from my last review still apply. Option 1 (walk only the .gq files under docs/, because a change to them already runs full CI) removes the liability. Option 2 (a docs/*.md change that contains a gq fence sets run_gqt=true, with a replay fixture in check-change-classes.py) keeps the coverage but adds a classifier rule. The owner of the regression is still the classifier replay. GQT cannot express this contract.

Recommended (not blocking)

S1. Move the v1 retirement into its own pull request against main.

The retirement does not depend on edition 3. v1 read ir::untyped, which erase.rs built from today's typed pipeline IR. Before ec18d6fa, no edition-3 file named erase, untyped, ReadExecutor or the reference engine (git grep at ec18d6fa^). Each comparison already had pinned rows. So the change is complete on its own, and on its own it removes liability now: a 6,700-line frozen engine compiled into every GQT build, a cfg(feature = "test-util") field threaded through Session and Detached, and an allowlist in forbidden_apis.rs.

In this draft it waits for RFC acceptance, which this PR correctly holds as its gate. If the RFC changes direction (for example toward #1244's additive path), someone must extract it later. A revert of the preview would also bring v1 back. A separate pull request is also easier to review. It deletes a shipped test-format section and a crate with CODEOWNERS, and that is a different risk from adding a language. The title "parse and type-check GQ edition 3" no longer covers it either.

S2. The PR and RFC #1302 now disagree about One IR. One of them should change.

RFC #1302 at 88cb905d, section "One IR", and its blocked_on item 4 say two things:

  1. "Today's pipeline IR is deleted when the planner reads the core, and the frozen v1 reference engine with it: v1 reads that IR." This PR deletes v1 while the planner still reads the pipeline IR, and v1 never read that IR (see S1). The PR shows that the coupling in the RFC is not real. I suggest that the RFC say v1 goes independently, because every comparison sits beside pinned rows.
  2. "No module carries an edition number in its name." This PR adds ir/core3.rs, query/ast3.rs, parser3.rs, typeck3.rs, print3.rs, habits3.rs, desugar3_tests.rs and omnigraph-planner/src/edition3.rs, which is about 15,500 lines (10,800 of them outside tests) under edition-numbered names. Under the RFC's end state these modules become the single front end, and the edition-2 parser is "the one edition-specific part". Renaming is cheapest now, while the PR is a draft and nothing else imports them. Give the new modules role names, and give the edition-2 parser the edition-specific name. If you want the numbered names during the preview, the RFC should say so and name when they go.

Earlier items

O1 (lint and the registry validators disagree on a valid edition-3 declaration), O2 (exact count pins), O3 (the edition-3 check in five places) and O4 (the door checks the handle's catalog) are unchanged. The PR's own commits since 81d69a1d do not touch those files. The pins in O2 are identical (T65 156 and the four-file bare_names list).

Small things:

  • cases/v2/planner/hydrate_return_columns_above_the_limit.gqt:12 still says "main is compared with engine v1".
  • crates/omnigraph/tests/integration/engine_v2.rs:1 says "differs from the frozen engine v1 by design". "The retired engine v1" would keep it true.

Tradeoffs and liability

The delta removes about 13,500 lines net, and most of that is real liability: a frozen second engine, a test-only seam in production types, a second read IR, and 397 corpus files with two pinned lists. The new repair rows are the right replacement. One row per distinct refusal shape, each suggestion applied and reparsed, tests behaviour rather than a corpus snapshot. The four known mismatches were design decisions, and the one that a test can express ($a not $b) is a row at parser3_tests.rs:1000. Dropping the 218 "accept" texts loses positive parse coverage. The property test (2,000 generated declarations) and the RFC examples, which are now type-checked inline, cover most of it. The PR still adds about 10,800 lines of front end that executes nothing, and the liability from my last review is unchanged. S2 is about making that code the one front end the RFC describes, not a second one.

Validation

All in a clean detached worktree at f6946f58. I restored every temporary edit (git status clean).

  • cargo test -p omnigraph-compiler --lib --locked: 547 passed, 0 ignored. This matches the description.
  • cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --test integration -- edition_corpus: 4 passed. With the B1 docs edit: 3 passed, 1 failed, as described. With last review's gq 3; block: 4 passed. With a T44 count(age) mutation example: 4 passed. I found the Q018 texts by temporarily printing them in the test.
  • python3 scripts/check-change-classes.py: ok ("reader closure holds ... 5 listed readers"). The ci.yml classifier, replayed for the docs path: no job.
  • scripts/check-docs.py (pinned requirements-docs.txt, Python 3.12): "Documentation OK (481 Markdown files checked)". scripts/check-agents-md.sh: OK.
  • A script over ec18d6fa^ checked that all 34 expect same as v1 steps pin rows. I also scanned every open PR's diff for the removed section and APIs.
  • Exact-head CI at f6946f58 when I posted: 28 pass, 4 skipped, 0 fail.
  • No GQT regression applies. This delta deletes code and fixes no behavioural bug. Not run locally: the full GQT corpus and the engine tests (forbidden_apis, warm_read_cost). The evidence for them is the passing exact-head CI. No performance claims. The change reads no Lance storage, so no Lance domain applies.

The edition-3 parser, tree, printer, foreign-habit table and checker move
under query/syntax and query/check, the core IR to ir/core.rs and the
planner's refusal to core_plan.rs; edition-2 parsing reads through
syntax/edition_window and the front door stays query/parser.rs. Types and
functions drop their edition suffixes (SourceFile, Expr, check_file,
plan_core, CheckedQuery::Core, QueryDecl::syntax). Tests sit in tests.rs
beside each module, and the diagnostics cases move to cases/diagnostics.
No behaviour changes.
# Conflicts:
#	crates/omnigraph-compiler/src/query/codes.rs
#	crates/omnigraph-compiler/src/query/parser.rs
…it arithmetic

IRExpr, its block predicate and their tests move from ir/mod.rs to
ir/expr.rs. The expression gains the nodes the core IR needs and has no
other spelling for: BinaryOp::Arith over one numeric type, a unary Neg,
and a typed scalar registry Call (if, coalesce, concat, match_terms),
with local type checks, GQ rendering at the right precedence and a
can_fail walk. The bound-plan mirror reads and writes them as new
variants, so every existing plan serializes as before.

The engine evaluates them per batch. An integer overflow, a zero divisor
and a non-finite float refuse the row with execution diagnostics E1, E2
and E3; a null operand gives a null row. Arithmetic is never folded or
pushed into a scan, so its failure keeps its place. Nothing produces
these nodes yet.
The core's terms, aggregates, scores, call bindings and sink assignments
store IRExpr instead of their own scalar type. The checker keeps its
typed working form private (query/check/scalar.rs) and converts it as it
builds each term: a reference column becomes a Variable, a value column
an AliasRef, a property or meta field a PropAccess, membership a
Contains comparison, and the registry rows their own nodes. ExprType
gains Edge so an edge reference column has a type. No behaviour
changes; edition 3 still stops at P006.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of #1316 at a14fe7ba

Recommendation: keep as draft. The rename answers S2 from my last review, and moving the core IR onto IRExpr is the right step toward "One IR". But the new arithmetic evaluator breaks its own documented contract in two cases, and nothing yet enforces the guard rule that the RFC and the code comments promise (B2). B1 is still open, unchanged.

Reviewed head: a14fe7ba. My last review covered f6946f58 (review). Since then:

  • f7ec10b1 gives the edition-3 modules role names (query/syntax, query/check, ir/core.rs, planner/src/core_plan.rs) and moves the edition-2 parser to syntax/edition_window. No tracked .rs file has an edition number in its name now.
  • dc95d4aa checks the RFC's recursive example as the RFC writes it.
  • 64e727bc merges main. I checked the conflict resolution with git show --remerge-diff. Main's Q006 (nesting depth) keeps its number, and the PR's unreleased "unknown edition" code becomes Q022. check_query_nesting still runs before the edition dispatch, so it covers edition 3 as well.
  • 3ce87e25 moves IRExpr to ir/expr.rs and adds BinaryOp::Arith, Neg and a typed registry Call (if, coalesce, concat, match_terms). It also adds mirror variants, E1–E3 and an engine evaluator (engine/arith.rs). The commit says "Nothing produces these nodes yet."
  • a14fe7ba stores IRExpr in every core term. The checker keeps a private working form (query/check/scalar.rs) and converts it with SExpr::to_ir. It also adds ExprType::Edge.

No other review or comment covers these commits.

Why, what, how, impact

Why. RFC #1302 lists "One IR" in blocked_on: both editions lower into one core, with one scalar expression language.

What. The core IR no longer has its own scalar type. Filters, projections, group keys, order keys, aggregate arguments, score queries, call bindings and assignments now hold the same IRExpr that the pipeline IR uses. IRExpr gains the arithmetic and registry nodes the core needs. The engine can evaluate them per batch.

How. Membership becomes Contains with the list on the left, which is how edition 2 already spells it. A meta field becomes a PropAccess of its @ name, as in edition 2. fold.rs never folds arithmetic, is_constant never treats it as constant, and typed_expr_to_df_expr never pushes it into a scan.

Impact. No behaviour change today: edition 3 still stops at P006, and old bound plans serialize as before. But the engine now has about 300 lines of evaluator that no plan reaches, with semantics the RFC does not yet fully specify.

Blocking

B1 (still open, unchanged). A docs-only pull request can still break edition_corpus without any CI job running it.

f7ec10b1 renames the tests and the import paths, but corpus() still reads every fenced gq block in docs/**/*.md outside docs/rfcs. the_corpus_read_under_the_edition_line still pins bare_names and requires unexpected to be empty. ci.yml and scripts/check-change-classes.py do not change in this delta, so the docs/*.md arm still sets no run flag. I did not run the demonstration again because none of the code it depends on changed. The two options from my earlier reviews still apply.

B2 (new). A failing arithmetic term is evaluated on rows where it should not run.

The module doc of engine/arith.rs says: "Only valid slots are computed, so a null operand gives a null result and is never refused." RFC #1302 says a function that can fail "keeps its failure where it was written; the planner may not move it past a guard (A3)". The code comment on is_constant says the same. The evaluator does not do this in three cases:

  1. null / 0 is refused with E2. binary (engine/arith.rs:83) checks has_zero(right) before the division, over the divisor alone. A row with a null dividend and a zero divisor is refused, not null. This breaks the module's own contract.
  2. if evaluates both branches on every row. evaluate_expr (engine/expr.rs:301) evaluates all Call arguments over the whole batch, then call zips them. So if($p.x = 0, 0, 10 / $p.x) is refused with E2 on the row it was written to protect. This is the standard guard, and it is the reason if exists beside can_fail arithmetic. The same applies to coalesce(…) with a failing later argument.
  3. and/or evaluate both sides. evaluate_boolean (engine/expr.rs:165) computes both operands over the whole batch before the Kleene combination. So $p.x != 0 and 10 / $p.x > 1 is refused. The RFC does not say whether and is a guard. SQL does not promise short-circuit evaluation either, so this one needs a decision more than a fix.

Also, nothing outside tests reads IRExpr::can_fail (ir/expr.rs:323), so the planner does not "keep such a term behind its guards" yet, although the doc comment says it does. If the optimizer merges two filter clauses into one conjunction, case 3 applies to clauses the user wrote apart.

Demonstration at a14fe7ba. GQT cannot express this, because no edition-3 query reaches the planner (P006). So I used the cheapest existing owner, the engine::arith::tests module. I added three temporary tests (x = [0, 5], column p.x):

  • binary(Div, [null], [0]): expected a null row; got Err("E2").
  • evaluate_expr of if($p.x = 0, 0, 10 / $p.x): expected [0, 2]; got Err("E2").
  • evaluate_filter of $p.x != 0 and 10 / $p.x > 1: expected [false, true]; got Err("E2").

All three failed for that reason. The four existing arith tests passed. Then I made two small temporary changes to show that fixes of this kind work:

  • Check for a zero divisor only where the dividend is present (zeros AND is_not_null(left)).
  • For if, evaluate the condition, filter the batch by the mask and by its negation, evaluate each branch only on its own rows, and interleave the results.

With those changes, tests 1 and 2 passed, and the four existing tests still passed. Test 3 still failed, as expected, because I did not change and. I then restored both files.

The cause is one missing rule: "a can-fail term runs only on the rows that reach it". The evaluator is the layer that owns it. There are two coherent options:

  • Option 1 (less liability now). Remove the engine evaluator (engine/arith.rs, the new evaluate_expr arms, E1–E3) from this PR, and keep the IR nodes, the mirror and the checker conversion. Add the evaluator with the first planner rule that produces these nodes, together with GQT cases that reach it. This also avoids publishing E codes before the RFC fixes their meaning, because codes.rs freezes a code once it ships.
  • Option 2. Keep the evaluator, and do three things: (a) state in the RFC which constructs are guards (if and coalesce at least, and say yes or no for and/or); (b) evaluate selectively, as in the sketch; (c) make the planner read can_fail before it reorders or merges a filter, with a test for each.

Recommended (not blocking)

S3. ExprType::Edge puts a type list in a string. types.rs:322 says type_name "may hold the alternatives of an edge selector, joined by |". The checker builds it with .join("|") (check/mod.rs:1573), in member order. This was private to the core before. Now it is in the shared ExprType that the planner and the engine match on, and ExprType equality compares the joined strings. Invariant 9 asks for typed structure here. Today only refusals read it, so this is the cheapest time to change it, for example to a sorted Vec<String> or the resolved EdgeSelection.

S4. Two copies of can_fail. SExpr still computes and stores can_fail (check/scalar.rs:20), but nothing reads it after to_ir. IRExpr::can_fail is the copy that is used. Remove the field so the two rules cannot drift.

S5. match_terms' mode goes through a string. The checker turns mode: all|any into a bool, then into a Literal::String("all"|"any") option, and to_ir matches it back, with anything else falling back to Any (check/scalar.rs:263). So the same mode has three spellings: LexicalQuery.mode_all: bool, TermsMode, and the mirror's all: bool. Today this is correct because the checker writes only those two strings. It is cheaper to store TermsMode in SKind::Call (and to resolve the function name to ScalarFunc in the checker). Then to_ir is total, without the unreachable! on an unknown name, and a new registry row touches one place fewer.

Earlier items

  • S2 is resolved for module names. If RFC #1302 still says that v1 retirement waits for the planner to read the core, it should change (S2, point 1).
  • S1 (move the v1 retirement into its own PR) is unchanged.
  • O1–O4 are unchanged. This delta does not touch those owners except for renames.
  • These two small comments are still stale: cases/v2/planner/hydrate_return_columns_above_the_limit.gqt:12 ("main is compared with engine v1") and crates/omnigraph/tests/integration/engine_v2.rs:1 ("the frozen engine v1").
  • Open PR #1101 edits query/parser.rs, which this PR moves into syntax/edition_window, so expect a rebase conflict there. Its T60 is still unallocated here.

Tradeoffs and liability

The rename and a14fe7ba remove liability. There is now one stored scalar language instead of two, membership and meta fields have the edition-2 spelling, and the rename makes the edition-specific part look edition-specific. The private SExpr is a fair cost: it carries reference nullability that ExprType does not have, and the code never stores it.

3ce87e25 adds the opposite kind of liability. It is engine code with no producer, so no end-to-end test can pin it, and its semantics are not yet decided. It also adds E codes that freeze on release. What does this look like after five more registry rows? Each row now touches the checker (by string name), call_to_ir, ScalarFunc, ScalarFuncMirror, typed.rs, the optimizer walks and the engine call. That is acceptable for typed rows. But eager evaluation would apply to each of them, and can_fail would still be advisory. Fix the evaluation rule once, at the evaluator, before the first producer lands, or wait until then to add the evaluator.

Validation

All runs were in a clean detached worktree at a14fe7ba. I restored every temporary edit (git status clean).

  • cargo test -p omnigraph-compiler --lib --locked: 552 passed, 0 failed.
  • cargo test -p omnigraph-engine --lib --locked engine::arith: 4 passed at the head. With the three B2 tests: 3 failed with E2. With the two sketch fixes: 6 passed and 1 failed (the and case, which I did not change).
  • Merge check: git show --remerge-diff 64e727bc (codes and parser conflicts, as described above).
  • B1: code inspection of edition_corpus.rs, ci.yml and check-change-classes.py (no change since f6946f58). I did not run it again.
  • Exact-head CI at a14fe7ba: 26 pass, 4 skipped, 2 fail. The two failures (Azurite Azure Integration, RustFS S3 Integration (default), 403 instead of 200 in s3::server_opens_azure_graph_directly_and_serves_snapshot_and_read) also fail on main at c61f242e (run). They come from c8dcb315 (deny reads without a policy), and #1332 addresses them. They do not come from this PR.
  • Not run locally: GQT and the engine integration tests. A Cargo.lock bump on main (arrow 58.4) made a second cold build too expensive for this run. The evidence for them is the passing exact-head CI. No performance claims. The delta reads no Lance storage, so no Lance domain applies.

GQ 3 replaces the language whole (docs/rfcs/2026-11-07-gq-3.md, sections
6 and 11): one grammar with plain rule names, one type checker, one IR.
The edition line, the old parser, AST, checker and lowering, and the
pipeline IR are gone.

The type checker binds every expression (physical system columns, the
casts `coerce` chooses, folded literals) and infers result schemas from
the core. The planner reads only the core through the read shape, which
lowers the pattern, its projection, grouping, order and cut, and the
`from nearest/bm25/rrf` rankings to today's ranked scans; the engine
lowers write groups to today's staged writes. Any other core node is
P006. Every header consumer checks through the core.

The repository's query text was rewritten by the one-shot rewriter; the
seventeen fusion cases the RFC lists were edited by hand, and the
cross-version journeys keep predecessor syntax for predecessor binaries.
Five diagnostics improve: SQL `union` names the relation function, T74
names the return item, T8 prints the block form, T65 carries an
applicable edit, and type errors carry the position of their clause.
@ragnorc ragnorc changed the title feat(compiler): parse and type-check GQ edition 3 behind the gq 3 line feat(compiler,planner,engine)!: replace the query language with GQ 3 in one break Nov 10, 2026

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of #1316 at ae9c2a42

Recommendation: keep as draft. The break is the right shape: one grammar, one checker, and a planner that reads only the core. But seven defects reach users, and five of them return wrong rows or wrong effects with no error. Each one has a GQT reproduction at this head. Also, the new commit ae9c2a42 makes CI pass by weakening a contract test, not by fixing the served runner.

Reviewed head: ae9c2a42. My last review covered a14fe7ba (review). Since then:

  • 6d40bc42 replaces the language in one break (519 files, +9,677/−21,742). It deletes the edition-2 parser, AST, checker, lowering and the pipeline IR. It adds planner/src/read_shape.rs, which lowers the core to today's read plan, and exec/write_plan.rs, which lowers write groups to today's staged writes. The one-shot rewriter rewrote the repository's query text.
  • ae9c2a42 changes boolean_literal_refused_where_a_property_is_named_false.gqt from expect error: Q008 to expect error: is not a statement.

No other review or comment covers these commits.

Why, what, how, impact

Why. RFC #1302 (sections 6 and 11) makes GQ 3 the only language and asks for "One IR": the planner and the engine read the checked core, not a second pipeline form.

What. Edition 2 text no longer compiles. Reads lower through ReadShape::of (core → scans, expansions, filters, blocks, return, order, cut, ranked scans). Writes lower through WritePlan::of (groups → staged inserts, updates and deletes). Any core node outside those tables is P006.

How. The checker binds every expression (physical columns, casts, folded literals). The planner's acceptance now takes its requirements from the same ReadShape. The GQT corpus, docs, skill and benchmarks were rewritten. 17 fusion cases were edited by hand.

Impact. Every stored query must be rewritten before upgrade. The rewritten corpus passes: 536 of 536 GQT cases pass in process here. The problems are in the forms that are new in GQ 3: write groups, scoped edge bindings, if, rrf populations, parameter defaults. The rewritten corpus does not exercise these forms, so CI does not see them.

Blocking

Each item has a temporary GQT case that I ran at this head with cargo test -p omnigraph-gqt --locked --test gq_logic_tests <name>. The PR contains no fix for any of them, so there is no "after fix" run. See Limitations.

B3. A write group's statements do not share one selection. WritePlan::of copies the group's filter into each update and delete (exec/write_plan.rs:96, :105). execute_update then re-runs that filter over committed plus pending rows (exec/mutation.rs:1632-1684). The RFC (section 3) and ir/core.rs say a group's clauses are evaluated once and every sink sees that one snapshot.

query publish() {
    $x: Doc
    $x.status = "draft"
    update $x set { status: "published" }
    update $x set { n: 1 }
}
  • Two drafts and one published doc. Expected: both drafts get status = "published" and n = 1.
  • Actual: n is not set on either draft. The second update matches 0 rows and the mutation reports success.
  • A per-row insert takes its count from the last statement (matched, mutation.rs:1684), so update; update; insert inserts 0 times.
  • An insert between two updates can add rows that the second update then selects.
  • Cause: the group selection is not a value the executor holds. Fix it in the executor, which owns group execution: select the group's rows once (ids, on the group's snapshot), then apply every statement to those ids. Then the filter is not repeated and is not re-evaluated against pending rows.
  • The rewriter puts each edition-2 statement in its own group, so rewritten text is not affected. No case has two update/delete statements in one group.

B4. $e.@type resolves by name across the whole declaration. Binder::collect (check/bind.rs:74-100) builds one name → edge-types map over every scope: blocks, groups, inline reads. A later binding overwrites an earlier one. expr() (bind.rs:222-228) then folds $e.@type to a literal when the map says one type.

match { $p: Person  exists { $p $e:knows $x }  $p $e:(knows | likes) $f }
return { $f.name as f, $e.@type as kind }
  • Expected: {b, Knows}, {c, Likes}. Actual: {b, Knows}, {c, Knows}, which is silently wrong.
  • In a write group, the same leak can fold a $e.@type = "Likes" filter to false, so a delete deletes nothing.
  • Cause: the binder keeps scope-free state that the checker already has with scope. Fold @type in the checker, where the binding's scope is known, and delete the map.

B5. Filters in an rrf arm's population are dropped. fused (read_shape.rs:672-690) keeps each arm's ranking key and discards its population. The Fuse then lowers to a scan of the whole type.

let $en = { match { $c: Chunk  $c.lang = "en" } return { $c } }
from rrf(nearest($en, $c.embedding, $v, k: 3), bm25($en, $c.text, $q), k: 60)
  • Expected: en-1 only. Actual: de-1, de-2, en-1.
  • The RFC says hand-written text "ranks the population, which is the canonical form". Every GQT rrf case uses an unfiltered $all, so none catches this.
  • fused also drops weights:, the bm25 arm keeps only text from its LexicalQuery (read_shape.rs:640, so a terms(…, mode: all) runs as match-any), and nearest drops recall:. The freeze list says these work. The T75 message tells users to write weights.
  • Fix: lower a filtered population as the ranked scan's prefilter, as the single-ranking path does. Refuse with P006 any option the scan cannot carry. Never drop an option.

B6. An anonymous source on a free traversal does not terminate. For match { $_ knows $p } with $p not yet bound, traversal() renames the anonymous end to "_" (read_shape.rs:799-804), but its scan keeps __anon_N. The fallback at read_shape.rs:1171-1180 then pushes NodeScan { variable: "_" } on every pass, and "_" never counts as bound, so the loop never ends.

  • On a 3-node graph the case hits the runner's 10 s wall-time budget. The old lowering refused this shape with an error. issue_659_compiler_scopes.gqt covers only the form where $p is bound first.
  • Fix: give the anonymous end one name in both places, and require each pass to bind a variable (refuse otherwise).

B2 (from my last review, now reachable). A can-fail term runs on rows its guard excludes. Arithmetic and if now reach the engine from user text, so the evaluator issue is no longer theoretical:

  • return { $p.slug, if($p.x = 0, 0, 10 / $p.x) as r } gives E2: '/' has a zero divisor. Expected: 0 and 2.
  • return { $p.slug, $p.y / $p.x as r } with y null and x = 0 gives E2. Expected: null. The module doc of engine/arith.rs promises null here.

Option 2 from my last review now applies: selective evaluation for if/coalesce, and the zero check limited to rows with a present dividend. The RFC still has to say whether and and filter-clause order are guards.

B7. Parameter defaults are type-checked but never applied. into_query_file builds the header Param without its default (syntax/parse.rs:292-296). The checker reads defaults only for calls to declared reads. query q($n: String = "a") { … $p.name = $n … } run without n gives parameter 'n' not provided. The RFC's own example $k: I64 = 5 can therefore not run, and it also hits P006 (limit over a parameter). Parameter metadata now lives in three types (syntax::Param, file::Param, the checked params). Keep one, so a field cannot be lost in a copy.

B8. ae9c2a42 weakens a break contract test instead of fixing the runner. The served GQT job failed at 6d40bc42 because only the in-process runner splices a parse code into the error text (error_text, omnigraph-gqt-core/src/lib.rs:3634). The served runner reads only the error line (served.rs:258-261) and ignores diagnostic.code. The server sends the code correctly. RFC section 6 says the fix of Q008 is "part of the break's contract, tested per code". The case now matches prose and no longer tests the code. Choose one owner for the error text:

  • (preferred) Print the code in QueryDiagnostic's Display for parse diagnostics, as type and plan diagnostics already do. Then delete error_text. The CLI, HTTP error line and both runners all show Q008.
  • or decode diagnostic.code in the served runner and render it as error_text does.

Then restore expect error: Q008.

B1 (still open). edition_corpus.rs is gone, but the new gqt/tests/integration/query_corpus.rs reads every gq block in docs/**/*.md. In ci.yml, the docs/*.md arm still sets no run flag. A docs-only PR can still add a block that does not compile under GQ 3, and no CI job runs on it.

Needs a decision

D1. Assigning a wider numeric value to a narrower property is now T3. numeric_fits (check/mod.rs:3375) refuses I64 → I32 and F64 → F32. The old checker accepted them, because numeric types were assignment-compatible, so stored mutations with $a: I64 for an I32 column stop loading. The rule ignores signedness: I32 → U32 and U64 → I64 pass. Reproduced: insert Person { …, age: $a } with $a: I64 and age: I32? gives T3: property 'age' has type I32? but the value is I64. The RFC says "an assignment never narrows" for arithmetic results, not for parameters. If this is intended, add it to the compatibility list, make signedness consistent, and give the error a fix (declare $a: I32).

Recommended (not blocking)

  • S6. Validation accepts text that never runs. queries validate, server boot (queries.rs:265) and cluster validate (config.rs:1413) only call check_query_decl. The checker accepts many shapes the planner refuses (limit $k, optional, computed order keys, block values compared with columns, writes with traversals), so a stored query deploys and then fails P006 on every call. ReadShape::of(read, catalog) needs only the catalog. Run it, and the write-side WritePlan::of check, in registry validation and lint. That also moves the write P006 out of the engine.
  • S7. Plan acceptance lost its independent input. validate/requirements.rs now takes bindings, counts and rankings from the same ReadShape the plan is built from. So acceptance cannot catch a lowering that drops a filter or an option, which is why B5 passes. RFC 0047 asks that requirements come from the checked query. Derive them from the core.
  • S8. @description/@instruction are no longer validated. @description(42), a bare @description, and duplicates are accepted (syntax/parse.rs:275-283). The header then shows None or the first value to MCP and the catalog. The old parser refused these.
  • S9. A per-row insert of a keyed node runs once per matched row (mutation.rs:1640-1650) and reports nodes=N for one upserted row.
  • S10. reads_search (write_plan.rs) only looks inside Binary and Not. A search under Call, Cast or IsNull gets the untyped "unsupported expression in mutation predicate" instead of P006.
  • S11. The T42 needles are bare. aggregate_identity_order, aggregate_metadata_name_still_orders and constant_edge_type_still_orders_groups changed from rows to expect error: T42 with no message. The refusal is intended (compatibility list, "3 stored cases"). Name the message and rename the still_orders cases, so the corpus states what is refused.
  • S3, S4 and S5 from my last review are unchanged: ExprType::Edge still joins with | (check/mod.rs:2013), SExpr still stores can_fail, and the match_terms mode still goes through "all"/"any".

Tradeoffs and liability

Removing about 21,700 lines and a second checker and lowering removes much liability. One grammar and one checker are easier to keep right than two editions. But "One IR" is not yet one IR. PatternStep/ReadShape is the old IROp/QueryIR under a new name, and place, FreshNames and scan_root are copied from the deleted lower.rs. The read path is now core → Facts → pipeline steps, which is two intermediate forms to keep in step.

Meaning is still carried in strings (__anon_, __temp_b, ^name, "_", the hard-coded "score"), and B4 and B6 are both name-string bugs. After five more changes of this kind, each new core node will touch read_shape's item() and key_columns separately; they have already drifted for Neg and Call. The by-construction fixes are: a typed binding identity (scope, var) instead of name strings, and requirements derived from the core (S7). Together they remove this class of bug instead of patching each case.

Write groups are the largest new semantic surface. B3 shows that lowering a group to today's per-statement writes is not the same relation. That needs one selection owner in the executor before more statements (returning, @if_commit) arrive.

Validation

All runs were in a clean detached worktree. I restored every temporary case (git status clean).

  • At 6d40bc42: cargo test -p omnigraph-compiler --lib --locked: 312 passed, 0 failed. cargo test -p omnigraph-gqt --locked: 536 of 536 cases passed in process, and the harness tests passed. cargo test -p omnigraph-planner --locked: 190 passed, 0 failed.
  • ae9c2a42 changes only one .gqt expectation (the B8 case), so these results apply to the head.
  • The B2–B7 and D1 reproductions are temporary GQT cases, each failing at this head as described.
  • B3: the second update left n unset on both drafts.
  • B4: kind: "Knows" for the Likes edge.
  • B5: 3 rows instead of 1.
  • B6: the case hit its 10 s wall-time budget.
  • B2: E2 in both cases.
  • B7: parameter 'n' not provided.
  • D1: T3 as quoted.
  • B8: exact-head CI at 6d40bc42: GQT (served) failed only on boolean_literal_refused_where_a_property_is_named_false ("served and in-process verdicts differ … error does not contain "Q008""). The GQT (dst) job was cancelled by fail-fast; its cases that ran passed.
  • Exact-head CI at ae9c2a42: still running when I posted.
  • Code inspection only: S6–S10, the rrf weights/recall/terms options, and the B1 CI routing.

Limitations:

  • The PR has no fix for B2–B7, so I could not run "before fix / after fix". The cases fail at this head for the stated reason, and each needs a fix and the case kept as its regression.
  • One temporary case (a filter clause 10 / $p.x > 1 behind $p.x != 0) failed in the GQT plan check ("malformed typed expression tree") before execution, so I could not test filter-clause guard order.
  • I did not run the served GQT locally, the engine integration tests or the workspace tests; the evidence for them is exact-head CI.
  • I make no performance claims. The delta does not touch Lance storage paths.

…of the core IR

Add `omnigraph-reference`, a test-only interpreter that evaluates a checked
read's core IR as the RFC's operator table states it: nested loops over a
branch's export, no index, planner, optimizer or cost model. It depends on
the compiler and serde_json alone, which `tests/dependencies.rs` holds. Every
operator the planner lowers has an evaluation and a unit test, and so do
`Union`, `Fix` (naive iteration to the fixpoint) and `Call`, which the
planner refuses with P006.

GQT's `--- expect same as reference` evaluates a query step's read with it
and compares the rows with the engine's, ordered or unordered as the step's
rows expect says. The comparison goes through an `ExecutionHost` hook, so
only the GQT runner depends on the interpreter; the served runner refuses
the section by name, the DST runner skips it, and other executors refuse the
case. The section is restored on the 15 cases that carried the old engine
comparison and added to 17 more, so every lowered operator, aggregate and
traversal form runs against the reference at least once.
… type

A parameter default was range-checked only as a scalar integer or F32, so
$xs: [I32] = [3000000000] compiled, and an assigned literal was not range
checked at all. check_literal_type, which every default and assignment
passes, now refuses an integer outside its type's range, a float that is
not a finite F32 for an F32, and each element of a list or a vector by its
element type, with T3 naming the value and the type.
…o test passes on a floor

fix_convergence passed a refused step whose fixes printed no edit and never
checked what the fixes wrote. Every corpus step whose refusal prints a
machine-applicable fix is now listed with the text its fixes are meant to
write, the repair must equal it, a listed step that prints no edit fails,
and an unlisted one that does fails too. Its corpus floors become walks:
every cases/diagnostics case holds a step the compiler refuses.

The typing-work test discarded its check's result and timed it; it now
asserts the result and the expressions typed. The plan-budget test read
the term size out of the message; the checker records the largest term it
charged. The numeric operand test counts every outcome exactly, and the
bounded-refusals test expects exactly 256.
…es by what they own

The engine test files engine_v2.rs, engine_v2_memory.rs and
engine_v2_plan_replay.rs become execution.rs, execution_memory.rs and
plan_replay.rs, and engine_v2_scrubbed_replay.rs becomes scrubbed_replay.rs.
The GQT corpus drops its v2/ directory: plan assertions move to
cases/planner/, the other cases to the corpus root (engine_v2_now_is_query_scoped
becomes now_is_query_scoped), and cases_slow/v2/ to cases_slow/. Every
reference follows.
Every read runs from its plan, and the session setting engine had one value,
v2, which changed nothing. The setting, its Engine enum and the
OMNIGRAPH_ENGINE variable are removed, with the engine field of a request's
settings, the CLI's mapping of it, the GQT runner's OMNIGRAPH_GQ_ENGINE
baseline and the engine field of a worker input. Every set engine = v2; line
leaves the GQT corpus, the input_engine_route inventory case goes, and the
tests that spelled the setting or listed it in show output drop it. The
settings guide, the GQT README and the developer guides follow, and a
breaking note says what callers delete.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of #1316 at 8e3a59a7

Recommendation: revise before merge. The three commits fix real defects, and the before/after runs below prove each fix. But the Q006 change adds a new blocking defect: an or chain written with is null, contains or starts_with now passes the depth bound, and a chain of 1000 such terms (about 30 KB of source) overflows the 16 MiB query stack and aborts the process. A second defect is in the same class as the guard fix: a block whose only failing part is its aggregate argument is still placed as if it cannot fail, so the core IR runs a where before the elements, and the reference interpreter refuses where the engine returns rows. The open findings from my earlier reviews (N1–N4, B8, D1, D2) are unchanged; this delta does not touch their code.

Reviewed head: 8e3a59a7. My last review covered aad3ca07 (posted at b429323f, review 5935982718). This review covers the three commits b429323f..8e3a59a7:

  • fa81835e fix(compiler): a guarded block reads its guard as a column instead of typing a copy of it;
  • 0445816e fix(compiler): Q006 measures each expression's depth, not the sum of its siblings;
  • 8e3a59a7 fix(compiler): printed fixes take their positions from parse tokens, and positions from a line index.

While I reviewed, the head moved to 11c4b379 (four commits, 10:16Z–10:31Z: nearest over an optional or a union, one literal range rule, a served null parameter default, fix-convergence tests). I did not review those commits. They do not change depth.rs, syntax/parse.rs or diagnostic.rs, and the three check/mod.rs lines cited in D4 are the same at 11c4b379 (each moves down 8 lines). No Decision needed comment exists on this PR, and no inline comment covers this delta.

Why, what, how, impact

Why. Three compiler defects. (1) A block that can fail, written under a guard, held a copy of the guard and was typed again. A guard that held blocks was copied again inside them, so typing work grew exponentially, a name the guard bound was bound again (T68), and an aggregate argument was not guarded. (2) The Q006 estimate added up the and/or chains of sibling expressions, so a write of about 250 groups was refused although no tree was deep. (3) A missing comma's fix rescanned text for a trailing comment and cut a multiline string that holds //, then inserted the comma inside the string.

What and how. (1) Each guard is now the earlier operand as typed (lazy_guard). A guarded block's guard is projected once as a column of the outer row, and condition_env filters the block's Env leaf on that column (check/mod.rs:5914-6012). The checker counts typed expressions, and a test holds that count linear in the guarded blocks. (2) measure ends an expression at a list separator, or where "a second operand follows" an operand, and keeps only the deepest expression per level (depth.rs:173-180, :241-250, :308-317). (3) token_end takes a construct's end from its last parse token (syntax/parse.rs:225), and a thread-local LineIndex positions refusals from line starts computed once per source.

Impact. Queries that the guard rules promise to evaluate safely now run, and typing a source with many guarded blocks is linear. Large write sources pass Q006. Printed repairs no longer corrupt string literals. D3 below reverses part of the Q006 protection.

Blocking

D3. Q006 now undercounts an or chain whose terms use an infix word, and a long chain aborts the process. The new rule ends the expression at "a word right after an operand that is not an operator" (depth.rs:241-250). Only and, or, not, in, is, when, then and else are exempt. But the grammar has more infix words (grammar.pest:379-389, :124): null after is ($x is null), contains, starts_with, and the habit spellings like, between, starts with/ends with, and case … end. Each of these words resets the chain count in the middle of one expression. So $p.nick is null or $p.nick is null or … measures one level, whatever its length. The tree it builds is one level per or.

Measured through the public check_source_all, on the 16 MiB on_query_stack thread, with RUST_MIN_STACK=16777216 (debug build). Schema node Person { handle: String @key, nick: String?, age: I32 }; the query is match { $p: Person } where <n terms joined by or> return { $p.handle }:

Term n b429323f (before) 8e3a59a7 (head)
$p.handle = "a{i}" 300, 1000, 5000 Q006 Q006
$p.nick is null 300 Q006 accepted
$p.nick is null 1000, 5000 Q006 stack overflow, process aborts
$p.handle contains "a{i}" 300 Q006 T7 (habit refusal; Q006 passed)
$p.handle contains "a{i}" 1000, 5000 Q006 stack overflow, process aborts
$p.handle starts_with "a{i}" 1000, 5000 Q006 stack overflow, process aborts

is null is valid GQ, so a 300-term chain is a valid query with a 300-level tree that the bound must refuse. A stack overflow is not a panic: it aborts the whole process. The server and the CLI give each query thread this same 16 MiB stack (depth.rs:39-45), so one ad-hoc read could stop a server. I did not run this against a server or a release build. Release frames are smaller, so the threshold is higher, but the undercount has no limit: 5000 terms also pass the bound.

  • Invariant: the estimate never undercounts the tree a source can build (depth.rs:23, and the Q006 row in diagnostics.md:183); failures are bounded and observable (invariants 8 and 11).
  • Class: every infix word in comparison, and every word added to it later. The rule is a deny-list of words that continue an expression, so each new infix word reopens the defect without a test failing.
  • Fix by design: invert the rule. Unknown words continue the expression. Only the words that start a clause or an element end one (match, where, return, order, limit, let, insert, update, delete, set, as, …), together with the list separators. The estimate then errs high, as its contract requires. A word you forget then costs a false Q006, never a crash. Sibling write groups still reset at match/update/insert/delete, so the 250-group write that 0445816e fixes still passes.
  • Regression: GQT can express it as an error. Please add a case whose steps are a 300-term is null chain and a 300-term contains chain, each with --- expect error: Q006. At 8e3a59a7 the first is accepted and the second is refused T7, so both fail for the predicted reason. Keep the depth.rs sibling test as it is.

D4. A block whose only failing part is its aggregate argument is placed as if it cannot fail. fa81835e makes the aggregate argument count toward whether a block can fail, but only for the guard decision (check/mod.rs:5157: term.can_fail() || arg…can_fail). Two other sites decide placement from a different answer:

  • a where conjunct is fenced after the elements when pred.can_fail || blocks.iter().any(|b| b.inner.can_fail()) (:2815). b.inner is the block's term without the Group, so the argument is not counted;
  • a let fences later conjuncts only when v.can_fail (:3289). The block's value is a column reference, so no block counts at all.

Observed at 8e3a59a7 with a temporary GQT case (persons ada age 1 and bob age 5, each working at a company; dan age 0 works nowhere, so the element $p worksAt $c drops dan):

Step Result
where count { $d: Company where 100 / $p.age > 0 } > 0 (inner where fails) ada, bob; same as reference
where sum($x) { $d: Company, let $x = 100 / $p.age } > 0 (same division, as an inner let) ada, bob; same as reference
where sum(100 / $p.age) { $d: Company } > 0 (same division, as the argument) the engine returns ada, bob; the reference refuses E2: a divisor is zero
match { $p: Person, let $s = 100 / $p.age } where $p.age != 0 E2, as traversal.md:237-245 states
match { $p: Person, let $s = count { $d: Company where 100 / $p.age > 0 } } where $p.age != 0 ada (engine and reference agree); the where ran below the let

Row 3: the core IR runs the conjunct before $p worksAt $c, against "where … is evaluated after all of them" (traversal.md:33). The engine happens to return the documented rows, so engine and reference disagree. AGENTS.md calls that a bug in one of the two, and here it is the IR. The same step fails the same way at b429323f, so this delta did not cause row 3, but fa81835e fixes the same cause through the guard path and leaves this path open. Row 5: a let whose value is a failing block behaves differently from the scalar let in row 4, against the documented let rule.

  • Invariant: an expression that can fail runs only on the rows the written order selects (A3), and the core IR states that order (the reference interpreter is its oracle).
  • Class: every site that asks "can this expression fail?" The answer has four owners today: SExpr::can_fail, Term::can_fail, the local can_fail at :5157, and the block clause at :2815. They disagree.
  • Fix by design: one owner. Let the block's value expression carry can_fail = term.can_fail() || arg.can_fail (that is Term::can_fail of the Group the block lowers to, which already counts aggregate arguments). Then pred.can_fail at :2815 and v.can_fail at :3289 cover blocks with no extra clause, the clause at :2815 and the local at :5157 go away, and the guard decision reads the same flag. Please check the other readers of SExpr::can_fail when you make this change. If row 5 is intended, change the let text in traversal.md instead. Either way, the doc and the IR must agree.
  • Regression: extend guarded_block_runs_only_where_its_guard_selects.gqt (or a sibling case) with rows 3 and 5. At 8e3a59a7, row 3 fails with the reference refusal and row 5 returns ada.

Optional

  • O13. condition_env drops the guard silently for a term shape it does not know. It walks Expand, Filter, Project, Group, Correlate (outer), Order and Cut, and returns any other variant unchanged (check/mod.rs:6010). I checked that today the Env leaf is always on that spine: finish starts from Env and every step in apply wraps it in one of those variants (Optional through Correlate's outer side). So this is not a defect now. But a future step that wraps the scope in a Union, Score or Call would make a guarded block run on every outer row without an error. Please make the fallback loud (a typed internal error, or a debug_assert! plus a test that walks each step kind), so the invariant holds by construction and not by today's step list.
  • O14. token_end visits every inner pair, not only the last. .into_inner().map(token_end).max() (syntax/parse.rs:235-238) recurses through the whole subtree on each call, and span_of/source_span call it for each construct. The old trim_end cost only the trailing whitespace. Total work is now the sum over the tree of subtree sizes, bounded by source size × depth. I did not measure it. Pest pairs are in source order, so the last non-empty inner pair is enough.

Earlier blocking findings

# Status at 8e3a59a7 Evidence
N1–N3 Open, unchanged The delta changes no planner, optimizer, shaping, engine/scan.rs or lexical_scan.rs code
N4 Open upgrade.md and lance.md rollback text unchanged
B8 Open, unchanged omnigraph-gqt-core unchanged
D1 Open Cargo.toml and the lance.md NGRAM row unchanged
D2 Open dataset_index.rs, error.rs and the troubleshooting text unchanged

Tradeoffs and liability

fa81835e removes liability. It replaces re-typing of copied source with one typed guard column, which makes typing linear, removes the T68 false refusal, and closes an escape from T39. The test that holds typing work linear is the right kind of guard. It adds condition_env, a second structural walk over the term with a silent fallback (O13), and it leaves the "can fail" answer spread over four places (D4). After five more changes of this kind, each new construct that can fail must remember to update each site. One flag on the value expression would make that hold by construction.

0445816e fixes a real false refusal, but it trades a safe overcount for an unsafe undercount (D3). For a safety bound, the cost of a missed word must be a false refusal, not a crash. That is why a list of words that end an expression is the right shape, and a list of words that continue one is not.

8e3a59a7 replaces a text rescan with parse facts, which is one source of truth for where a construct ends. string_lit is atomic, so its text never reaches the trivia scan, and lists hold only element and missing_comma_rec children, so last_end is always set before a gap. The LineIndex is keyed by the borrowed source's address and length, and it is removed on unwind. I found no defect. O14 is a cost note only.

Validation

Clean detached worktrees at b429323f (before) and 8e3a59a7 (head). GQT built in the shared target directory; compiler tests in a separate one. RUST_MIN_STACK=16777216 as in CI.

Proof Command b429323f 8e3a59a7
fa81835e: the PR's case, copied unchanged cargo test -p omnigraph-gqt --test gq_logic_tests --locked -- guarded_block_runs_only FAIL at step 2 (if guards the argument): E2: '/' has a zero divisor. Each other step alone: 1 (or) ok; 3 (coalesce) T68; 4 (and, inner let) E2; 5 (a guard holding a block) T68 1 passed (0.96 s)
0445816e: siblings_are_measured_by_the_deepest_not_their_sum, copied into depth.rs cargo test -p omnigraph-compiler --locked --lib FAILED: depth(write(1)) = (2, 6), depth(write(1000)) = (2, 2005) passed
8e3a59a7: the three new PARSE_REPAIRS rows, copied same FAILED: the repair reads let $s = "x, $p knows $q, the string cut and the comma inside it passed
Compiler lib at head same — 384 passed, 0 failed
D3 temporary integration test calling check_source_all inside on_query_stack table in D3 table in D3
D4 temporary GQT case, table in D4 row 3 fails the same way rows as in D4

Each "before" failure is a predicted behavioral reason, not a compile, setup or empty-run failure. I rebuilt the head GQT binary after the base build and ran the head cases again with the same results.

CI: at 8e3a59a7 every check passed (among them GQ Logic Tests, all GQT jobs, Test Workspace, Lint (clippy), DST pinned suite, Storage Upgrade Compatibility) except RustFS S3 Integration (default), which was cancelled when the head moved. At 11c4b379, at 10:59Z, GQ Logic Tests, GQT (dst-clippy), Lint (clippy), DST pinned suite, Dependency Guard (cargo deny) and Check AGENTS.md Links had failed. Those runs are outside this delta and I did not read their logs.

Limitations:

  • D3 was run in a debug build through the compiler API, not against a server or a release build.
  • I did not run case … end, like or between chains. The same word rule applies to them by inspection.
  • I did not run the workspace, DST or Clippy suites. I did not review 0efb4ca0..11c4b379.
  • All temporary test edits and probe files were removed, both worktrees are clean, and nothing was pushed to the PR.

…int the resolved analyzer

T2, T14, T21, T32, T33, T35, T36, T45, T46, T54, T58 and T59 had a catalog
entry and no emitter: no compiler, planner or engine source raises them.
They leave the catalog, and their numbers stay unallocated. T17, T19 and T20
describe what raises them now: a nearest call without its window, a
full-text call's field or terms operand, and a lexical query that is not
text or has no searchable terms.

schema plan printed an omitted analyzer as "edition 2 analysis"; it prints
the profile and scorer the index resolves to (FullTextAnalysis::resolved),
so an omitted analyzer reads analyzer = "english_v1".
Every ranked scan is a nearest scan, so RankKind had one variant. It goes,
with RankedAccess's kind: the score is the distance column (DISTANCE_COLUMN),
sorted ascending, and the code that matched on the kind uses it directly.
Explain keeps a ranked scan's documented kind: "nearest"; a saved plan no
longer carries the kind, so BOUND_PLAN_VERSION is 15. The full-text
validation probe's comment states what enters the validation now.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of #1316 at 11c4b379

Recommendation: revise before merge. The three fixes in this delta are real, and the before/after runs below prove each one. One new blocking defect: the new nearest path returns the right rows, but it drops the ranking object that the search page promises for every read that draws from a ranking. The CI failures that this delta caused (Clippy and the MCP page length) are already fixed at 88bd29c3. The findings from my earlier reviews (N1–N4, B8, D1–D4) are unchanged. This delta does not touch their code.

Reviewed head: 11c4b379. My last review (review 5937237232) covered b429323f..8e3a59a7. I posted it at 11c4b379, but it said that I did not review the four newer commits. This review covers those four commits, 8e3a59a7..11c4b379:

  • 0efb4ca0 fix(planner): nearest ranks a population that holds an optional or a union by its rows' exact distance;
  • 4abe9fe4 fix(api): a declared null parameter default survives the served catalog;
  • f9ea81ce fix(compiler): one range rule for every literal that meets a declared type;
  • 11c4b379 test(compiler,gqt): fix convergence checks the intended repair, and test floors become exact counts.

While I reviewed, the head moved to 88bd29c3 (seven commits, 10:36Z–11:20Z). I checked only what touches this review: f57db50d and 88bd29c3 fix the two CI failures below, and 221c4bf2 removes RankKind but leaves the logic of planned_ranking that D6 cites unchanged. I did not review a9bf35cf (test file renames), 5ed97b54 and be9d7172 (the engine setting removal) or 8ef2695a (diagnostic codes and the analyzer print). Line numbers below are at 88bd29c3. No Decision needed comment exists on this PR, and no inline comment or later review covers this delta.

Why, what, how, impact

Why. Three defects. (1) A nearest over a population that holds an optional or a union failed with a planner internal error. The population is read as a read of its own, so the ranked node is a column of its rows, but the shaper still scanned it first (the ranked binding $d has 2 scans). (2) The served catalog read a declared = null default as "no default". (3) The range check covered only a scalar integer or F32 default, so $xs: [I32] = [3000000000] and an assigned literal such as set { age: 3000000000 } passed type checking and failed later, at execution, with an untyped error.

What and how. (1) ranking_steps (shape/read.rs:1762) checks whether the population binds the ranked node itself. If it does not, it adds an exact-distance Score step and ranks by that column. The optimizer then always adds the same Sort and TargetCut (optimizer.rs:582-632). (2) ParamDescriptor.default deserializes through present_value, so a present null is Some(Null) and an absent key is None. (3) out_of_range in check_literal_type (literal.rs:95) is now the one range rule for defaults and assignments, element by element for lists and vectors.

Impact. Queries that ranked such populations now run, and the rows agree with the reference interpreter. MCP and queries list show a declared null default. Out-of-range literals are refused at type check with T3 before anything runs.

Blocking

D6. A nearest on the new exact-distance path returns no ranking object. The search page states: "A read that draws from a ranking carries a ranking object in its JSON envelope" (docs/user/search/index.md:265). planned_ranking (omnigraph-planner/src/ranking.rs:62) finds the ranking from a ranked Scan, a Fuse, or an exact bm25 Score. The new path has none of these. It has a distance Score, so planned_ranking returns None.

Observed with the CLI built at 11c4b379 (omnigraph query --json, direct local graph, Doc { slug @key, embedding: Vector(4) @index }, two docs, one tag):

Query Rows ranking
nearest over { match { $d: Doc } return { $d } }, k: 2 a, c {"by":"nearest","order":"ascending","exhaustive":true,"score_columns":["distance"]}
the same over { match { $d: Doc, optional { $t: Tag, $d tagged $t } } return { $d, $t } } a, c (correct) absent

A client that reads ranking to find the score column or to check exhaustive gets nothing for a query that the user wrote as nearest.

  • Invariant: the read reports the ranking it draws from, whatever physical path computes it. Query semantics are typed structures (invariant 9). The report must not depend on which operator happens to compute the score.
  • Class: every ranking that is computed without a ranked scan. Today that is this path. A future exact path for bm25 in some shape, or a nearest that the planner chooses to compute exactly, would hit the same problem. The reporting is inferred from physical operators, so each new physical path must remember to update planned_ranking.
  • Fix by design: the shaper already knows that this read draws from nearest (ranking.key is Nearest). Record the ranking function on the step (for example on PatternStep::Rank, or as the planned ranking of the read), and let planned_ranking read it. Exhaustiveness then comes from the scans only where scans exist. An exact path is exhaustive by construction. This removes the inference instead of adding a fourth case to it.
  • Regression: GQT has no ranking expectation, so the cheapest owner is crates/omnigraph/tests/search.rs, which already asserts the ranking object (result.ranking()). Add the optional population there and assert by: nearest, exhaustive: true, and the score column.

Optional

  • O15. A list literal for a scalar gets a range message instead of a type message. out_of_range runs before types_compatible (literal.rs:95 and :120), and for a Literal::List it checks the elements even when the declared type is not a list. So $x: I32 = [3000000000] is refused with T3: `x` has type I32, but 3000000000 is outside its range. The real error is that a list is not an I32. The code is still T3, so this is only a diagnostic issue. Check the type first, then the range.
  • O16. Two range owners remain. integer_range (literal.rs:399) is the new rule, but fits (check/mod.rs:6658) still decides integer ranges for negated literals (:6619, :6630). Let fits call integer_range, so one table holds the ranges.
  • O17. fix_convergence's INTENDED list is a second home for case data. The intended repair of each diagnostic step now lives in a Rust table keyed by "<file> step N" (fix_convergence.rs:224), not in the .gqt case. A change fails loudly, which is good: an unlisted step that prints a fix fails, and a listed step that prints none fails. But each new diagnostic case needs edits in two files, and inserting a step renumbers the keys. An --- expect fix section in the case would keep one owner.
  • O18. The typing measure is production state that only tests read. 11c4b379 made Clippy fail on fields `exprs` and `largest_term` are never read (Clippy, GQT dst-clippy and DST pinned were red at 11c4b379, then green at 88bd29c3). f57db50d fixes this with #[cfg_attr(not(test), allow(dead_code))] on Measured (check/mod.rs:311). That works, but largest_charged (:1161) is still a production Cell that every charge call updates, only so that a test can read it. The T70 refusal already holds that size in its message. If a test needs the size, a typed field on the refusal gives it one owner, and Measured could then be cfg(test) only. 88bd29c3 also brings docs/user/operations/mcp.md back to 350 lines, which clears Check AGENTS.md Links.
  • O19. Put the range regression in GQT. The regression for f9ea81ce is a compiler unit test. GQT can express this contract (expect error: T3), and parameter_defaults_bind_omitted_parameters.gqt already holds the scalar range steps. Please add the list-default and assignment steps there (see Validation for the exact steps that I ran).

What I checked that holds

  • 0efb4ca0 over ties, null vectors, a parameter window and rrf. A temporary GQT case agrees with the reference interpreter on all five steps at 11c4b379: k: 1 with two docs at the same distance keeps the smaller identity; a doc with a null embedding is left out (nearest skips null vectors); k: $k binds through fold_pipeline; a union population with a tie; and an rrf arm over an optional population where one target has two rows (each target is ranked once). At 8e3a59a7, every step fails with the planner internal error.
  • The Rank step had only Nearest keys before this delta, because search_node returned None for any other key. So making the TargetCut unconditional (optimizer.rs) and treating any Rank as the window (requirements.rs:177) changes no earlier plan.
  • The exact path scores with the same kernel as Lance's flat kNN (docs/dev/lance.md, "Vector distance"), so the F32 values equal an exhaustive nearest. This path calls no Lance API. Its cost is one exact distance for each population row with no index. That is acceptable, because before this delta these queries were refused.
  • 4abe9fe4: the other Option<Value> fields in omnigraph-api-types are params, where null and absent mean the same. F64 overflow such as 1.0e309 is already refused at T3, so a non-finite default cannot reach literal_json and appear as "default": null.

Earlier blocking findings

# Status at 11c4b379 Evidence
N1–N3 Open, unchanged The delta changes ranking shaping only for populations that do not bind the ranked node; it does not touch the code these findings cite
N4 Open upgrade.md and lance.md rollback text unchanged
B8 Open, unchanged omnigraph-gqt-core unchanged
D1, D2 Open Cargo.toml, dataset_index.rs and error.rs unchanged
D3 Open depth.rs unchanged
D4 Open The can_fail sites in check/mod.rs are unchanged in logic

Tradeoffs and liability

0efb4ca0 removes liability. It fixes the cause, which is "the shaper assumed that the ranked node is a binding", with one general rule rather than a special case for optional and union. Any population shape that yields the node as a column takes the exact path. The cost is D6: the ranking report is still inferred from physical operators, so the new path is invisible to it. 4abe9fe4 is small and correct, and present_value is the right shape. f9ea81ce moves a late, untyped execution error to a typed T3 at check time, and replaces one special case with one rule. O15 and O16 are what is left to make it fully one owner. 11c4b379 makes tests stricter and removes message parsing. The cost is test-only state in the production checker (O18) and a second home for case data (O17).

Validation

Clean detached worktrees at 8e3a59a7 (before) and 11c4b379 (head). The GQT runner was built with cargo test -p omnigraph-gqt -p omnigraph-gqt-core --locked --no-run at each commit, and each case was run with omnigraph-gqt <case> and RUST_MIN_STACK=16777216.

Proof Command 8e3a59a7 11c4b379
0efb4ca0: the PR's case, unchanged omnigraph-gqt cases/nearest_ranks_a_population_holding_an_optional_or_a_union.gqt FAIL step 1: planner internal error: the ranked binding $d has 2 scans under its search node; the union step alone: … has 3 scans … ok (1.55 s)
0efb4ca0: ties, null vector, k: $k, union tie, rrf arm (temporary case, each step expect same as reference) omnigraph-gqt /tmp/probes/nearest_over_rows_probe.gqt FAIL step 1, same planner error ok, all five agree with the reference
D6 head CLI: init, load --mode merge, query --json — (the query is refused) ranking absent for the optional population; present for the plain one
4abe9fe4: the PR's unit test, copied cargo test -p omnigraph-api-types --locked --lib a_null_default_survives FAILED: {"…","default":null} reads back None, expected Some(Null) 1 passed
f9ea81ce: the PR's unit test, copied cargo test -p omnigraph-compiler --locked --lib a_literal_meeting_a_declared_type FAILED at its first row on wording only (base refuses with the default … is outside the range of I32), so I used the GQT steps below for the behavior 1 passed
f9ea81ce: GQT steps added to a copy of parameter_defaults_bind_omitted_parameters.gqt, one step per run omnigraph-gqt <copy> [I32] = [1, 3000000000] default: param 'xs': value Integer(3000000000) exceeds I32 range (no T3); update … set { age: 3000000000 }: value Integer(3000000000) exceeds I32 range at execution; the same for insert; Vector(2) = [0.5, 1.0e39]: value Float(1e39) exceeds F32 range each refused with T3; the whole extended case passes
O15 same, step $x: I32 = [3000000000] — T3: `x` has type I32, but 3000000000 is outside its range
O18 exact-commit CI Clippy, DST and GQT passed at 8e3a59a7 Clippy dead fields and mcp.md at 351 lines at 11c4b379; at 88bd29c3 every check passes except Dependency Guard (cargo deny)

Each "before" failure has the predicted behavioral reason, except the unit test row marked as wording, which I replaced with GQT steps. Steps to add for f9ea81ce:

--- query
query out_of_range_list_default($xs: [I32] = [1, 3000000000]) { match { $p: Person } where $p.age in $xs return { $p.handle } }

--- expect error: T3

--- mutate
query out_of_range_assignment() { match { $p: Person } where $p.handle = "p1" update $p set { age: 3000000000 } }

--- expect error: T3

Limitations:

  • I did not run the workspace, DST or Clippy suites locally. I relied on exact-commit CI for O18.
  • Dependency Guard (cargo deny) fails at 88bd29c3 (chrono, chrono-tz, humantime unmaintained). It is a new advisory, not this PR: #1438 fails the same check at de2d9307, and main passed it at 07:46Z.
  • I ran D6 at 11c4b379. At 88bd29c3 I checked it by inspection only: shape/read.rs is unchanged and planned_ranking still finds a ranking only from a ranked scan, a Fuse or an exact bm25 score.
  • The D6 run used a direct local graph through the CLI, not a server. The server uses the same planned_ranking.
  • I did not measure the cost of the exact path on a large population.
  • All temporary cases were outside the repository, the copied unit tests were removed, both worktrees are clean, and nothing was pushed to the PR.

The GQ 3 RFC (section 2) makes `and` and `or` guards: `a and b`
evaluates `b` only where `a` is not false, `a or b` only where `a` is
not true. The interpreter evaluated both sides, so a right side that can
fail refused a row its left side had already decided, where the engine
answered.
`Score` refuses a non-finite input (the GQ 3 RFC, section 2), and a
distance from finite components can overflow `F32`. The interpreter
reported the infinity as the row's score, which renders as null, and kept
an overflowing target in a `nearest` window. `distance` now refuses any
non-finite score with E3 and `nearest` a non-finite one among the targets
it keeps, as the engine does.
The interpreter bound only the parameters a caller supplied, at whatever
value it held. It now binds as the engine's binding does: an omitted
parameter takes its declared default, an omitted nullable one without a
default is null, and an omitted required one, a null for a non-nullable
one, a number outside its declared type, a DateTime with a non-zero digit
past the millisecond and a value bound to now()'s reserved name are
refused. A window or constant read from a parameter is refused outside its
range (a limit below zero, a nearest window below one, rrf's constant
outside 1..=RRF_MAX_K), and so is a null lexical query, which the RFC
makes a non-null String (section 4). A DateTime is its instant: spellings
of one instant, another offset or zero padding, compare and render as one
value.
The interpreter refused every lexical call over an english_v1 field, the
default profile, because it carried no stemmer, and applied no NFC. The
GQ 3 RFC (section 4) fixes both: NFC before tokenization, and english_v1
stems with the English Snowball stemmer, then removes the stop list of
each word and its stem, then folds. The interpreter now runs the whole
pipeline in that order, with frostem's English algorithm and the
compiler's stop list; unicode-normalization supplies NFC.
Reading the entity behind a reference scanned its type's rows for the id,
so a read over n entities of one type took n² comparisons; a 16,384-edge
case spent seconds there. Each type now keeps its entities' positions by
id; scans still read the rows in export order.
Export read only a branch head. `Omnigraph::export_jsonl_at` exports at
any read target, a branch or a captured snapshot, through the same
coherent cut; GQT's reference comparison reads a snapshot step's rows
through it.
A step compared with the core reference interpreter only when its case
wrote `--- expect same as reference`, about a third of the rows steps, and
no error step. Every read step on the direct engine is now compared:

- rows against rows, ordered or as bags as the step's expect compares;
- an `E`-coded engine refusal against the same code from the reference;
- an engine refusal of input the query cannot read (a parameter value, a
  read target) against the reference refusing its input;
- an engine resource bound against the reference's rows, since the
  reference has no bound.

A step refused at compile time has no core term and is not compared. The
DST, served and benchmark executors do not compare: the comparison is a
property of the direct target, not of the case. A step opts out only with
`--- reference skipped: <reason>` after its last expect section; the
reason is required, the section is a new `reference_skip` rule, the runner
prints the reason under the case's result line, and a skip on a step the
comparison never reaches fails the step. The comparison reads the step's
read target, a snapshot included. `--- expect same as reference` is
removed, and the case format is version 2.
Drop every `--- expect same as reference` section, which the default
comparison replaces; the steps commit ce8af54 took out of comparison,
because the served suite ran their cases, compare again on the direct
engine. Steps that scan a catalogue type, which the export does not carry,
and the two scale cases whose data the reference's nested loops cannot
read within the case's budget opt out with that reason. The fix
convergence test names the typed_search_options step by its new section
index.
…ory bounded by the window

The exact lexical scan kept every matching document's identity and its
per-term alternatives until the corpus statistics were known, then a score
per identity, so its memory grew with the corpus: TREC-COVID query 48 over
171,000 documents exceeded the 150 MiB pool. The field corpus is now read
once for its statistics alone, keeping no document, and the Score operator
scores each row of its population from the row's own text, which its scan
projects; a sort with a limit then keeps the window. A corpus several times
the pool scores within it.
…ch is refused for an index

The upgrade guides described a stored-query upgrade procedure the owner
has not settled; they now say it is pending and keep only what a refused
stored query prints, the same error[CODE] line for a parse and a type
failure. An explicit l2 distance is stored as an omitted one, so only
another distance keeps an earlier binary out. A lone re-mention's filter is
written with coalesce when its test can be null, and an optional adds plan
nodes in proportion to its own pattern. No guide or skill line says this
binary refuses a search with FullTextIndexRebuildRequired: only an earlier
binary does, on an index this one built.
The parse-print-parse generator now builds whole source files (one to three
declarations, or declarations and an `explain`) and covers the header
(parameters of every type with list and negative defaults, annotations with
and without parentheses, `as commit` and `as unattached branch`), string
escapes, raw control characters and comment-like text inside strings,
negative numbers in lists, infix `contains`/`starts_with`, bare `@meta`,
keyword-named calls and catalogue edges in alternations. Each case also
requires printing to be a fixed point and reparses a respelling with
comments between tokens and raw newlines, tabs and carriage returns in
strings.

A coverage test parses the generated texts of a fixed seed and requires
every rule of grammar.pest except a commented exclusion list (silent rules,
rules inside atomic tokens, recognizers, the statement grammar).

A display round trip types generated predicates and return values, displays
the stored IRExpr, types the displayed text in the same query and requires
the same declaration. Membership, `contains(…)` and `starts_with(…)` are
left out: Display prints them infix, which the type checker refuses.

The gate runs 450 and 300 cases; the nightly twins run 100,000 and 20,000
and are enrolled in tests-slow-nightly.yml.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of #1316 at 9888b0e5

Recommendation: revise before merge. This delta adds no new blocking defect. The bm25 memory fix is correct: the rows and scores are the same as before, and its new test fails before the fix and passes after it. The GQT corpus now compares every direct-engine read with the reference interpreter, and all 613 cases pass at the head. The cost of the memory fix is that an exact bm25 now analyzes its population's text a second time. This doubles the work that it charges to read_work_limit, and the execution report does not show the second pass (O20). The earlier blocking findings (D6, N1–N4, B8, D1–D4) stay open. This delta does not change their code.

Reviewed head: 9888b0e5. My last review (review 5938070931) covered 8e3a59a7..11c4b379. It said that I did not review a9bf35cf, 5ed97b54, be9d7172 and 8ef2695a. This review covers those four commits and every commit after 88bd29c3, up to 9888b0e5:

  • a9bf35cf test file and GQT case directory renames; 5ed97b54 and be9d7172 remove the one-value engine setting; 8ef2695a removes twelve diagnostic codes that no stage raises and prints the resolved analyzer;
  • 2f3c724b, f7fbbf30, 070b06d3, 6d1fd000, ef87065c: reference interpreter fixes (and/or guards, non-finite distance, parameter binding, english_v1 and NFC, lookup by id);
  • c0dbc286 export at a read target; 3732684a and 9943820f compare every direct-engine read step with the reference;
  • cc3cb99c the exact bm25 scores each row from its own text; 37d04499 the exact scan's cost;
  • 4ed86485 upgrade and GQ 3 guide edits; 9888b0e5 the round-trip generator covers every grammar production.

No Decision needed comment exists on this PR. No inline comment or later review covers this delta.

Why, what, how, impact

Why. The exact bm25 kept every matching document's identity and term hits until it knew the corpus statistics. Its memory grew with the corpus, and a 171,000-document query exceeded the 150 MiB pool. Also, GQT compared only about a third of the read steps with the reference interpreter, and no error step.

What and how. cc3cb99c reads the field corpus once for its statistics only (Bm25Scorer, lexical_scan.rs:588). The Score operator then scores each population row from the text that its scan now projects. A sort with a limit keeps the window. 3732684a makes the reference comparison the default for every read step on the direct engine. Rows are compared with rows, and an E-coded refusal with the same code. A step opts out only with --- reference skipped: <reason>. The reference interpreter fixes make the interpreter follow the RFC where the engine already did.

Impact. A bm25 over a corpus larger than the query pool now answers within the pool. The GQT corpus is a much stronger oracle: 12 steps opt out, each with a reason (11 catalogue types that the export does not carry, 1 scale case).

Blocking

None new in this delta.

Optional

  • O20. The exact bm25 analyzes its population's text twice, and the report shows only one pass. The corpus pass analyzes every value of the field. Then Bm25Scorer::score analyzes each population row again and charges its tokens and comparisons to read_work_limit. For a whole-type population, the charged work doubles. A temporary GQT case (5 docs, bm25 over { match { $d: Doc } return { $d } }) needs read_work_limit 15 at 88bd29c3 and 30 at 9888b0e5. At 29 the head refuses with resource limit exceeded for read_work_limit: actual 30, limit 29. Under the default limit, the largest corpus that a bm25 can answer is therefore about half of what it was. Also, ScoreExec::execute creates the second pass's memory without the operator's metrics (score.rs:359; compare scan.set_metrics at :355). So lexical_scan_rows and lexical_scan_tokens count only the corpus pass, while the work charged is two passes. I checked the metrics by code inspection only.
    • Tradeoff: memory bounded by the window, for about twice the analysis time and work. I think that this is the right trade until stage L, because N3 already makes the cost linear in the field. But the cost must be visible.
    • Fix: attach the operator's metrics to the second pass (or report it under its own metric names), and say in "The exact scan's cost" (analyzers.md:89) that bm25 analyzes the field once and its population rows again. This is the same class as N3: each lexical call adds a full pass that the query's shape does not bound.
  • O21. The Score row of the operator table still describes the old bm25. docs/dev/execution.md:666 says that a bm25 row's "identity [is] looked up in the scores LexicalMembers::resolve computed". cc3cb99c updated the Filter row next to it, but not this row. It should say that each row is scored from its own text by Bm25Scorer::score, and that the analysis is charged as exact lexical scan.
  • O22. Any other engine error counts as agreement with a reference input refusal. EngineRefusal::of maps every error that is not a compiler diagnostic or a resource bound to Input (omnigraph-gqt-core/src/lib.rs:4537). That includes internal errors (Manifest), DataFusion, Storage and Io. judge_reference then passes the step when the reference also refuses its input (:4654). So an internal engine error on a step whose parameters the reference refuses is reported as "same as reference". Today this needs a case whose expect error: text matches the internal error, so the risk is small. A safer rule: Input only for the typed parameter-binding and read-target errors, and an internal error always fails the comparison.
  • O23. Display prints three forms that the type checker refuses. The new round-trip test leaves out list membership, contains(…) and starts_with(…), because Display prints them infix and the checker refuses that spelling (round_trip.rs:82). The same note says that explain output, GQT plan expectations and the docs print this spelling. A user who copies a filter from explain gets a refusal. One spelling for the printer and the checker would remove the exclusion. This is not new in this delta, but the delta makes it visible.
  • O24. Two GQT case notes cite codes that no longer exist. issue_640_search_score_projection.gqt:4 cites T33, and rank_key_asc_direction_refused.gqt:4 cites T54, but that case now expects T65. 8ef2695a removes both codes from the catalog, but these notes still describe them as refusals.

What I checked that holds

  • cc3cb99c gives the same rows and scores as before over every population shape that I tried. A temporary GQT case with five steps passes at 9888b0e5 (with the reference comparison) and gives the same rows at 88bd29c3: bm25 over a population that holds an optional (d1 keeps both tag rows with one score), over a union, over a traversal with fan-out, inside rrf with nearest, and a plain window with limit 2. The scored text reaches Score in each shape, so the lowering's "field … is not in its input" refusal does not occur.
  • The new Bm25Scorer uses the same analyzer profile, Corpus::score and IDF as the old code. A row's length and hits come from the same analysis of the same value at the same snapshot, so the score is the same.
  • 3732684a: the judge requires rows to match rows (ordered or as bags, as the step's expect does), an E code to match the same code, and a stale skip to fail. A static refusal has no core term and is correctly not compared. The DST, served and benchmark executors do not compare, which is right, because the comparison belongs to the direct target.
  • 5ed97b54: the engine setting had one value. The removal is complete in the code paths (settings.rs, api-types, CLI, GQT runner, bench). The breaking note says that a stale set engine = v2; is refused as an unknown setting.
  • 8ef2695a: no compiler, planner or engine source raises the twelve removed codes. Only GQT notes (O24), release notes and RFCs mention them.
  • a9bf35cf only renames (34 lines in, 34 out). c0dbc286 passes a ReadTarget through the existing export path, with no new read logic.
  • 9888b0e5: the round-trip tests pass locally (8 passed, the 2 nightly twins ignored).
  • 4ed86485 removes the stored-query upgrade procedure and says that it is pending. That is honest, but a release with this PR has no upgrade procedure for stored queries until it is written.

Earlier blocking findings

# Status at 9888b0e5 Evidence
D6 Open ranking.rs and shape/read.rs unchanged since 88bd29c3
N1–N3 Open, unchanged The delta does not touch their code. cc3cb99c adds a second pass in the N3 class (O20)
N4 Open 4ed86485 edits upgrade.md, but the in-place rollback paragraph (upgrade.md:151-157, "or rebuild the indexes with the earlier binary") is unchanged
B8 Open served.rs only drops the removed same as reference refusal; the served runner still reads only the error line
D1–D4 Open check/mod.rs, depth.rs, dataset_index.rs and Cargo.toml unchanged in this delta

Tradeoffs and liability

cc3cb99c removes a memory bound that grew with the corpus, at the cause: the scan no longer holds per-document state. It adds about one more analysis pass per bm25 (O20). That cost goes away when stage L shares one analyzer between the index and the scan. 3732684a removes liability. The reference comparison now has one rule for every step instead of an opt-in section, and an opt-out must say why. The main new cost is that each new form must be supported by the reference interpreter or carry a skip reason. The reference fixes add a second implementation of parameter binding and DateTime rendering (canonical_datetime). That is the purpose of a reference, and the default comparison catches any drift. 5ed97b54 and 8ef2695a remove state that did nothing.

Validation

Clean detached worktrees at 88bd29c3 (base) and 37d04499, moved to 9888b0e5 (head). The engine code is the same at 37d04499 and 9888b0e5. The later two commits change only docs, the nightly workflow and a compiler test file. GQT runners were built with cargo build -p omnigraph-gqt --locked --bins at each commit, and run with RUST_MIN_STACK=16777216.

Proof Command 88bd29c3 head
cc3cb99c: the PR's test, copied into the base cargo dev-nextest -E 'test(bm25_scores_a_corpus_several_times_the_pool_within_it)' FAIL: ResourceLimitExceeded { resource: "query_memory_bytes", limit: 4194304, actual: 4196930 } PASS (15.1 s)
cc3cb99c: rows over five population shapes (temporary case) omnigraph-gqt /tmp/probes/bm25_population_shapes_probe.gqt ok, same rows ok, and each step agrees with the reference
O20: work charged the same query with set read_work_limit = N; answers at 15; refused at 14 (actual 15) answers at 30; refused at 29 (actual 30)
Corpus with the default reference comparison omnigraph-gqt crates/omnigraph-gqt/cases — 613 of 613 cases passed; 12 reference skips, each with a reason
Workspace and failpoint test build cargo dev-build-tests — compiles (at 37d04499)
9888b0e5 round trip cargo test -p omnigraph-compiler --locked --lib round_trip — 8 passed, 2 ignored

The bm25 memory regression is a Rust test, not GQT. That is correct: GQT can set query_memory_bytes, but the case needs 30,000 generated documents and the memory probes that check release, which only the Rust harness has.

CI:

  • At 37d04499, Test Workspace failed in its compile step on an infrastructure error (sccache: error: Server startup failed … dns error: failed to lookup address information), not on the code. The same command compiles locally. The run was then cancelled by the push of 9888b0e5.
  • At 9888b0e5, at the time of this review: Clippy, GQT (ordinary, served, dst, dst-clippy), the DST pinned suite, the aws server test, Format and the other quick checks pass. Test Workspace, RustFS S3 (default) and Storage Upgrade Compatibility are still running. Dependency Guard (cargo deny) fails, but it is not this PR: #1440 (a CLI-only change) fails the same check at 5c74be76, and the last main run passed it at 07:45Z, before the new advisories.

Limitations:

  • I did not run the workspace, DST, S3 or served suites locally. I checked the O20 metrics claim by code inspection only.
  • I did not measure the time of the second bm25 pass on a large corpus.
  • All temporary cases were outside the repository. I removed the copied test from the base worktree, both worktrees are clean, and nothing was pushed to the PR.

… its own region

A region named a kind of scope (an optional over a binding), so two
sibling optionals over one binding shared a region and a plan that moved
a conjunct from one to the other was accepted, swapping which side is
null. Each optional and correlated read is now numbered in written order
within the region that holds it, on the requirement side by its step and
on the plan side by a walk that numbers a node after its input side, the
order the planner builds them in.

The broken-plan test moves a conjunct between two sibling optionals
(refused now, accepted before) and between two sibling correlated reads.
…hat read's pool and work

A write group's selection read ran under its own query pool and
read_work_limit, but the pool was dropped with the run: the rows the
write kept, the identity and value maps an update or delete built from
them and the value map an insert built per row (each a copy of the
constant assignments) were charged nowhere, so a selection whose read fit
could hold any amount beyond it.

The selection now returns its rows held on the read's own context
(engine::SelectionRead), whose pool charges the kept batches and every
map built from them a batch at a time, and whose read-work budget
charges one unit per row each statement reads. LoadCharge becomes
PoolCharge, the one named charge on a statement's pool, with work().

The test runs, under a 4 MiB pool, a read that fits and a 6,000-row
selected insert whose maps do not (refused as query_memory_bytes by
'write selection inserts', every charge released), and under
read_work_limit 15000 a read that fits and an update over the same rows
that does not. Both writes succeeded under those bounds before.
@ragnorc
ragnorc marked this pull request as draft November 17, 2026 14:07

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta review of #1316 at 00a88ce5

Recommendation: revise before merge. This delta adds no new blocking defect. Both fixes are correct, and each new test fails before its fix and passes after it. The earlier blocking findings (D6, N1–N4, B8, D1–D4) stay open, because this delta does not change their code. I ask for one GQT regression for the read_work_limit half of 87e65af4 (below). The PR is a draft again since 14:07Z, but its review requests are still open, so I reviewed the new commits.

Reviewed head: 00a88ce5. My last review (review 5939170541) covered everything up to 9888b0e5. This review covers the three commits after it:

  • 96e26075 plan acceptance gives each optional and correlated read its own region;
  • 87e65af4 what a write keeps of its selection read is charged to that read's pool and work;
  • 00a88ce5 registers export_jsonl_at as a read-only surface in forbidden_apis.rs.

No Decision needed comment exists on this PR. No inline comment or later review covers this delta.

Why, what, how, impact

Why. Plan acceptance named a region by its kind only: "the optional over $d". Two sibling optionals over one binding were one region. So a plan that moved a conjunct from one to the other was accepted, and the wrong side became null. Separately, a write group's selection read ran under its own pool and read_work_limit, but the pool was dropped when the run ended. The rows that the write kept, and the maps it built from them, were charged nowhere.

What and how. 96e26075 adds an ordinal to Region::Optional and Region::Correlate (requirements.rs:165). Both sides count them with one helper, RegionOrdinals (requirements.rs:177): the requirements in written step order, and the plan by a walk that numbers a node after its input side (invariants.rs:166). 87e65af4 returns the selection's rows as engine::SelectionRead (context.rs:426). It keeps the read's QueryContext, holds the rows on its pool, and gives each statement a PoolCharge for its maps and one unit of read_work_limit per row it reads.

Impact. A broken plan of this shape is now refused at acceptance, before it runs. A write whose selection fits the pool but whose derived maps do not is now refused with query_memory_bytes. A write whose statements read more rows than the work budget allows is now refused with read_work_limit. Under the default limits I found no write that the change newly refuses (see O26).

Blocking

None new in this delta.

GQT regression requested

The read_work_limit half of 87e65af4 is behavior that GQT can express with three nodes. The PR tests it only in the Rust test with 6,000 rows. Please add the minimal case below. I suggest extending read_work_limit_bounds_a_chain_of_hops.gqt, which already owns "every read runs under read_work_limit" (it needs note: String? on Person). A temporary copy fails at 9888b0e5 and passes at 00a88ce5:

--- mutate
set read_work_limit = 4;

--- expect ok

--- query
query picked() {
    match { $a: Person, $a knows $b } where $a.handle = "a"
    return { $b.handle }
}

--- expect unordered
{"b.handle":"b"}
{"b.handle":"c"}

--- expect shape
b.handle: String

--- mutate
query stamp() {
    match { $a: Person, $a knows $b } where $a.handle = "a"
    update $b set { note: $b.handle }
}

--- expect error: read_work_limit

--- mutate
reset read_work_limit;

--- expect ok

(seed: a, b, c; a knows b, a knows c.) The read costs 4 units. The update costs the read's 4 and 2 for its pass over the two rows, so at the head it is refused with resource limit exceeded for read_work_limit: actual 6, limit 4. At 9888b0e5 the same update succeeds (affected: nodes=2).

The memory half needs thousands of generated rows and the release probes, so the Rust test is the correct owner for it. The 96e26075 defect is acceptance of a plan that the planner never builds, so only a test that edits a plan can show it. The unit test in validate/tests.rs is the correct owner.

Optional

  • O25. A region's identity is its position, and two walks compute it. The requirements count optionals by step index, and the plan counts them by a post-order walk over input sides. The two agree only because resolve_pipeline (optimizer.rs:459) builds each element over the rows of the elements before it, in written order. If a later planner reorders independent optionals or correlated reads, acceptance refuses the valid plan as an internal error. (The index-facts RFC names join order as future planner input, 2026-10-08-index-facts-as-planner-input.md:660.) That failure is safe, because nothing wrong runs, but the rule is in two places.
    • Option: the shaper gives each optional and correlated read a scope id. The logical and physical nodes carry it, and acceptance compares ids instead of counting. This adds one field to two node kinds and the mirror. It removes the agreement between the two walks, and a reordered plan is still accepted when it is correct.
    • Tradeoff: the ordinal is cheaper today and fails closed. The id is worth its cost when the first reordering rule is written, not before.
  • O26. For insert and update, the write's own row bound already limits what the maps can usefully hold. inserted_entities (mutation.rs:1171) and selected_entities (mutation.rs:1094) build a map for every selected row. Only then does the staging refuse more than 8,192 rows per table (admit_keyed_rows, staging.rs:359): an insert as it stages each row, and an update after the build (mutation.rs:2594). With this delta the build is bounded by the pool. So a large selected insert or update gets one of two errors, depending on its row count and row width: query_memory_bytes from write selection inserts/entities, or keyed entities for node:T. A user who raises query_memory_bytes only gets the second error. delete has no row cap, so it needs the pool charge.
    • Fix: let the build ask the staging's admission as the maps grow (the number of distinct entities per batch), so the write's own bound refuses first and gives the actionable error. The pool charge then stays the bound for the kept rows and the delete ids. I checked this by code inspection only. I did not run a selection larger than 8,192 rows.
  • O27. The insert charge counts each value's inline size twice. charge.entries::<(String, Literal)> (mutation.rs:1268) already charges each map entry, Literal included. map_bytes (:1207) then adds literal_bytes, which adds size_of::<Literal>() again (:1085). selected_entities is correct, because there literal_bytes measures the Vec<Literal> storage. The effect is small: an insert is refused a little early, and its reported actual is too high. A helper that counts only heap bytes for map values would fix it.

What I checked that holds

  • The two numberings agree for every plan the planner builds. Sibling optionals and correlated reads wrap current in written order (optimizer.rs:459–861), and an uncorrelated read after other elements is the right input of a CrossJoin. The walk visits a node's first input first, so it reaches them in the same order. Correlated union arms and a nearest window's population are keyed by their own region path on both sides. searched_side_right swaps the sides of a CrossJoin only when both sides are plain table scans, so it cannot reorder these regions. AntiJoin and expression blocks are not numbered on either side.
  • No valid plan is refused. A temporary GQT case with six queries passes at the head, and each step agrees with the reference interpreter: three sibling optionals, a nested optional beside a sibling, optionals between two aggregate from reads, an uncorrelated from followed by a correlated one, two ordered from reads over one binding, and two from reads with a let … count {} and a where exists {} block. The CI GQT jobs (ordinary, served, dst) also pass the whole corpus at 00a88ce5.
  • SelectionRead holds the context only for the write. selected is local to execute_named_mutation (mutation.rs:2227), which stages and returns before its caller commits, so the pool is not held across the commit. The kept batches are charged once by hold, and each map is charged one batch at a time, before the next batch adds to it. The PR's test shows that every charge is released after a refusal.
  • The update and delete accounting is correct. Each id is counted twice (the order Vec and the map key), each value once, and the entries of both collections. The keyed insert's index counts its key tuple.
  • The docs match the code. writes.md, settings.md (read_work_limit, query_memory_bytes) and the changelog note describe the new charges. 00a88ce5 adds the read-only surface that c0dbc286 introduced, so forbidden_apis lists every public read.

Earlier findings

# Status at 00a88ce5 Evidence
D6, N1–N4, B8, D1–D4 Open, unchanged This delta changes only the planner's validate/ module, engine/context.rs, engine/mod.rs, exec/mutation.rs, two loader type names, tests and three docs. None of these findings' code is among them.
O20–O24 Open, unchanged score.rs, the Score row (execution.md:666), omnigraph-gqt-core, Display and the two GQT notes are unchanged.

Tradeoffs and liability

96e26075 removes a false acceptance at its cause: the region is now one scope, not a kind of scope. It adds one counter type, shared by both sides, and one assumption, that the plan keeps written order (O25). 87e65af4 removes a real gap: the derived state of a write had no bound except the staging limits, which an insert or update reaches only after it builds every map. The change reuses the read's pool and work budget instead of adding a new allowance, and LoadCharge becomes the one PoolCharge. This is fewer primitives, not more. Its cost is a set of hand-written size estimates in mutation.rs (literal_bytes, map_bytes) that must change when Literal or the maps change. O27 shows how easily they drift. Five more statements that keep derived state would each need such an estimate. O26 would keep that to the one place where no other bound exists.

Validation

Clean detached worktrees at 9888b0e5 (base) and 00a88ce5 (head), one shared Cargo target. Built with cargo dev-build-tests and cargo build -p omnigraph-gqt --locked --bins. GQT ran with RUST_MIN_STACK=16777216.

Proof Command 9888b0e5 00a88ce5
96e26075: the PR's test, copied into the base cargo dev-test … a_conjunct_tested_in_another_scope_is_refused FAIL: the broken plan was not refused by a check (tests.rs:146) PASS
87e65af4: the PR's test, copied into the base cargo dev-nextest -E 'test(write_selection_maps_are_charged_to_the_read_pool_and_work)' FAIL at execution_memory.rs:2422: the insert succeeds (affected_nodes: 6000) PASS (28.6 s)
GQT regression (temporary case above) omnigraph-gqt /tmp/…/write_selection_work_probe.gqt FAIL: the update succeeds at read_work_limit = 4 ok: refused, actual 6, limit 4
Valid plans (temporary case, six queries) omnigraph-gqt /tmp/…/planner_sibling_regions_probe.gqt ok ok, every step agrees with the reference

CI at 00a88ce5: 27 checks pass and 4 are skipped, including Test Workspace, Storage Upgrade Compatibility, the DST pinned suite, RustFS S3 (default and failpoints), Dependency Guard and the GQT jobs. At 9888b0e5, Test Workspace and Storage Upgrade Compatibility had failed. I did not read why, because they pass at the head.

Limitations:

  • I did not run the workspace, DST, S3 or served suites locally; the CI results above are for the exact head.
  • O26 and O27 are from code inspection. I did not measure the size of Literal or run a selection larger than 8,192 rows.
  • All temporary cases were outside the repository. I removed the copied tests from the base worktree, and nothing was pushed to the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Typed change with nothing users can see; waives the changelog.d note check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant