Repository navigation
Conversation
Phase 0 of the GQ 3 RFC (draft): the compiler's half of edition 3. The one front door, parse_query, reads the edition line: no line or `gq 2;` is edition 2, unchanged; `gq 3;` selects the edition-3 parser, whose refusals name the construct and carry byte-range suggestions; any other line is Q006 with `gq 3;` as the fix. An edition-3 declaration is typed into the core IR (ir::core3) for every form of sections 2 to 4 and 7, recursion typed to Fix, and the planner refuses it with P006 (plan_edition3) at each engine door before a snapshot is captured. Nothing in edition 3 executes. Tests: the rule-set pin, the red-team corpus with pinned repaired trees (4 listed mismatches), the parse-print-parse property (2,000 cases, and 100,000 as a hunt), the desugaring-uniqueness test and the closure table, the before-and-after asset type-checked, edition-2 texts lowered to the same IR as before, a zero-I/O engine test, and one GQT case per code. The GQT runner keeps a step the compiler refuses at parse under `expect error:` and renders parse diagnostics with their code.
Stage M0 of the GQ 3 RFC measures how well a headless agent writes GQ in each edition before the break ships. This commits that instrument as a rerunnable part of the repository, beside the GQT corpus: - crates/omnigraph-gqt/agent/tasks: the 45-task competence instrument (39 kinds from the RFC's before-and-after corpus and the six M0 composition tasks) over the RFC's section 8 schema and a new seed. - crates/omnigraph-gqt/agent/corpus: a 30-task agent corpus over three GQT fixtures (social, attribution, reading), each task with its intent, parameters, an edition-2 ground truth with the client glue between its queries stated, the edition-3 ground truth, and the rows the edition-2 truth returns on the seed; 10 tasks need several edition-2 queries. - cards: the edition-2 card generated from skills/omnigraph, the edition-3 card written from the RFC; both pinned by SHA-256 in manifest.json with the agent model. - seeds: one injected error per refusal class and edition. - omnigraph-gqt-agent: accepts or refuses an answer through the parser and the type checker, reports the diagnostic and the normalised typed IR, runs edition-2 answers on the seed and compares rows; records the ground truths' rows and regenerates the edition-2 card. - run.py: one fresh `claude -p` per task and condition (first try, repair with the card, repair without it, seeded repair), results to a dated directory as JSON lines and a summary table. - agent::tests in the GQT library: every edition-2 ground truth accepted and returning its recorded rows, every edition-2 seed refused with its code, the cards matching their pins and the skill. The edition-3 checks are written and ignored until the edition-3 front end lands, with a tripwire test that fails on that day.
The edition-3 front end is in: the judge accepts or refuses an edition-3 answer through the same parse_query_diagnostic and typecheck_query_decl the engine doors use, and compares its core IR with the ground truths' after normalisation. Every edition-3 ground truth and alternative is accepted and every edition-3 seed refused with the code the front end gives it, checked by the GQT library test that was ignored. - t30's truth drew a second ranking of one population into a scope that already held its column (T68); it renames the second population first, as the RFC's before-and-after corpus does, and the card says so. - The prompt states each parameter's declared type, taken from the canonical truth and checked equal to the edition-2 truth's, so typed-IR equality does not depend on a guessed integer width. - t15, t17 and m3 accept the unanchored triple edition 3 admits. - The normaliser renames the variable behind a compiler column's sigil. - One more edition-3 seed (T74); the runner's summary gains the spread over runs, the gate verdict, gains and losses by task and the codes behind every repair.
Three runs per edition with claude-sonnet-5-5, the 45 competence tasks and the 30-task corpus, every condition: first try, repair with the card, repair without it, seeded repairs. results/2026-11-09 holds the records, the generated summary and a hand-read analysis of the gate. The runner keeps each declaration's typed IR by its SHA-256, which the judge rebuilds from the stored answer, instead of the IR itself.
v0.15.0 is tagged; its release snapshot is main's. The edition-3 preview note stays an unreleased changelog.d fragment.
ragnorc
left a comment
There was a problem hiding this comment.
Review of the agent instrument (head fe8f5b47)
Why. Stage M0 of the GQ 3 RFC (#1302) needs evidence that agents write edition 3 at least as well as edition 2. This PR adds the instrument that measures it.
What. The three test(gqt) commits add 75 tasks with ground truths, seeds, two reference cards, a Rust judge (omnigraph-gqt-agent, src/agent/), a Python runner, in-source tests that keep the data valid, and the first results. The first two commits are #1316, and the merge keeps main's v0.15.0 snapshot. I reviewed only the three commits and the merge resolution.
How it works. The judge accepts or refuses an answer through the compiler's own parser, type checker and lowering. It runs edition-2 answers on a seed and compares rows. It compares edition-3 answers by typed IR after it renames the names an author chooses.
Impact. No production path changes. The cost is about 2.1k lines of Rust, 0.7k lines of Python, 3.3k lines of data and 0.9 MB of results. The ordinary GQT library job also runs 9 new tests (about 8 s). The liability is small and contained: nothing in the engine, server or CLI depends on it.
Outcome: one judge defect to fix before the next run (B1). It does not change any verdict in the committed run. There are also some optional items. The PR's CI explanation is correct.
Blocking
B1. The IR normaliser renames schema names when an alias or variable has the same spelling, so a correct edition-3 answer can be judged wrong.
normalize::rename (normalize.rs:92-130) replaces every quoted string in the IR's Debug text whose spelling is a chosen name. A property, type or edge with the same spelling is renamed too. The module comment (normalize.rs:6-15) says that the comparison keeps "the names the task fixes (parameters, types, properties, edges)", and that the renaming "can only make two different declarations compare unequal, never two equal ones". The first claim is false, and the second does not hold either: two equal declarations can compare unequal.
Demonstration at fe8f5b47 with task m3. I took the canonical edition-3 truth and changed only the spelling of its count alias (as ALIAS, $ALIAS >= $min, return { $c.name, $ALIAS }, order { $ALIAS desc }), then ran omnigraph-gqt-agent --root crates/omnigraph-gqt/agent judge --edition 3:
| alias | verdict |
|---|---|
n (the truth) |
correct |
headcount |
correct |
slug (a Company property the query does not read) |
correct |
name (a property the same query reads as $c.name) |
typed IR equals no ground truth |
With as name, the answer's "name" property in $c.name becomes v2, but the truth keeps "name". Agents often give a column the name of the property it holds, so this case will occur. The same function also feeds the edition-2 ir_equal measure (mod.rs:157), which gives the "both by typed IR" row.
Effect on this run: none. Three accepted edition-3 first answers (run 2 m5, r02; run 3 s03) use an alias spelled like a property or edge (slug, cites). I changed only those aliases to a neutral name and judged them again. All three still miss for other reasons. So the published numbers and the gate verdict do not change. But the judge is the measurement, and the error goes in one direction (false misses), on the half of the gate that already fails.
The violated invariant: the judge must decide equality from structure and from the names the task fixes, and an author's name choice must not change the result. Renaming by spelling on Debug text cannot keep this invariant, because the text does not say which quoted strings are bindings. A fix by design is to rename at the layer that knows: alpha-rename variables, aliases and the declaration name on the parsed AST (to v0, v1, … in order of first binding), then type-check and lower, and compare the IR text with no substitution. This removes source_names and rename and adds no new state. A smaller fix (skip a name when the catalog defines it) still fails when an alias and a property have the same spelling, as above.
Test: GQT cannot express this, because it is the instrument's judge and not query behavior. The cheapest existing owner is agent::tests::the_judge_runs_answers_and_compares_rows_and_typed_ir. Add the m3 truth with as name and require "correct". At this head it fails as shown above. I did not write a fix, so I cannot show it pass.
Optional
O1. The judge test needs RUST_MIN_STACK but the documented command does not set it. cargo test -p omnigraph-gqt --locked --lib at this head, without the variable, aborts with thread 'agent::tests::the_judge_runs_answers_and_compares_rows_and_typed_ir' has overflowed its stack, and the abort also loses the results of the other library tests. With RUST_MIN_STACK=16777216, which the GQT workflow sets, all 61 pass. The binary already sets its own 16 MiB runtime stack (omnigraph-gqt-agent.rs:228), and the other GQT library tests pass without the variable. Run the two #[tokio::test] bodies on the same runtime shape (one helper). Then the command in AGENTS.md and in this PR's test plan works as written.
O2. The runner adds gq 3; for the agent. extract (run.py:192-205) removes any edition line the agent wrote and puts exactly one at the top. A real front door reads an answer with no edition line as edition 2. In this run every one of the 225 edition-3 first answers wrote the line, so no number changes. But the instrument cannot see this failure class, and the edition-2 path has no equivalent help. Judge the text as written, or count a missing line separately.
O3. The edition-2 card is a committed copy of seven skill sections, held equal by a test. This is a second copy that the test keeps from drifting. Measured cost is small: of the 28 commits in the last month that touched SKILL.md or references/queries.md, 2 changed one of the seven sections and would now also need cards --write. The alternative is to generate the card when the runner starts and record its digest in meta.json (the runner already does this), and keep a committed pin only for the hand-written edition-3 card. The tradeoff: the committed copy is easy to read in a diff, but git history already reproduces it from the skill at any commit. I would keep it as is unless skill edits to those sections become frequent.
O4. Results size. records.jsonl holds 0.88 MB of raw agent replies. If each later run is committed the same way, the repository grows by about 1 MB per run. You could commit summary.*, analysis.md and meta.json with a digest of the records, and store the records as a CI or release artifact.
Notes and context
- CI at this head.
GQT (ordinary),GQT (served),GQT (dst),GQT (dst-clippy)andStorage Upgrade Compatibilityfailed from infrastructure: the sccache download returned 504 and the other GQT jobs were cancelled, so the GQT corpus has no CI evidence at this head. A rerun is needed.Check AGENTS.md Linksfails onrelease notes changed after snapshot generation. I confirmed your explanation: with a temporary commit that removes onlychangelog.d/gq-edition-3-preview.added.md,check-docs.pypasses (482 files). Main373834b4passes too. - Gate reading. The analysis is clear about its limits: one reader who is not blinded, and typed-IR equality is strict. The hand-read count compares 39 equivalent edition-3 answers (not counting the 4 with extra columns) with edition 2's 39 by rows, and edition 2's
t02extra column counts as a miss. So both sides use the same rule. - Acceptance comparison. The edition-3 front door does not lower to a plan, but the edition-2 one does. In this run all edition-2 refusals are parse or type errors (
T7,T1,Q001, and the no-queryt38), so this difference changes no acceptance number. - Other open work. #1244 proposes composable GQ without a language break. Its card could be a third condition on this same instrument if it gets an implementation. I treat this as context only, not as a requirement for this PR.
Validation
All at exact head fe8f5b47, in a clean detached worktree. I restored all temporary edits.
cargo test -p omnigraph-gqt --locked --lib: aborted by a stack overflow inagent::tests::the_judge_runs_answers_and_compares_rows_and_typed_ir(O1). WithRUST_MIN_STACK=16777216: 61 passed.agent::alone: 9 passed.cargo build --locked -p omnigraph-gqt --bin omnigraph-gqt-agent, thenjudge --edition 3for the B1 table and for the three committed answers.python3 scripts/check-docs.py: fails on the release-notes snapshot, and passes with a temporary commit that removes the edition-3 fragment.- Python over
results/2026-11-09/records.jsonlfor the edition-line, refusal-code and name-collision counts. - I read AGENTS.md, invariants.md, testing.md and lance.md (no Lance domain applies: no storage path), the agent README, analysis.md,
src/agent/*.rs, the binary, and the runner's agent-call and extraction code.
Not done: the GQT corpus (--test gq_logic_tests), clippy and typos locally; no agent run (I did not call claude); no performance claims.
|
Closing without merging. The competence instrument and its agent corpus stay outside the repository for now, as section 11 of the GQ 3 RFC now records. The branch |
Phase 0 of the GQ 3 RFC (#1302), stage M0: the measuring half. The RFC is a draft, so this PR stays a draft until it is accepted. It is stacked on #1316 (the edition-3 front end): its first two commits are that PR's; review the three
test(gqt)commits. Two merges keep it current without rewriting #1316's commits: one ofmain, and one of #1316's branch at75cb05fc, which takes main'sdocs/releases/v0.15.0.mdunchanged and retargetschangelog.d/release.jsonto v0.15.1.What changed
crates/omnigraph-gqt/agent/, beside the GQT corpus (itsREADME.mdsays what each number means and how to run it):tasks/: the 45-task competence instrument (39 kinds from the RFC's before-and-after corpus and the six M0 composition tasks) over the RFC's section 8 schema, with a new seed;corpus/: a 30-task agent corpus over three GQT fixtures (social, attribution, reading). Each task has its intent, its parameters, the edition-2 ground truth with the client glue between queries stated (bind,each), the edition-3 ground truth and its equally canonical alternatives, and the rows the edition-2 truth returns on the seed. 10 tasks need several edition-2 queries;cards/: the edition-2 card is generated fromskills/omnigraph, the edition-3 card hand-written from RFC sections 3, 4 and 8; both are pinned by SHA-256 inmanifest.jsonwith the modelclaude-sonnet-5-5;seeds/: 18 edition-2 and 29 edition-3 seeded answers, one injected error per refusal class;run.py(standard library only): one freshclaude -pper task and condition (first try; repair with the card; repair without it, task, answer and diagnostic only; seeded repair from the diagnostic alone), refusing a reply from any other model; results go to a dated directory as JSON lines with a generated summary.omnigraph-gqt-agent(src/agent/,src/bin/): accepts or refuses an answer through the engine's own front door (parse_query_diagnostic, thentypecheck_query_decl, which types edition 3 into the core IR), reports the serialized diagnostic and the normalised typed IR, runs edition-2 answers on the seed and compares rows. It also records rows (record) and regenerates and pins the cards (cards).agent::tests(the GQT library target, run by the ordinary GQT job): every edition-2 ground truth is accepted and returns its recorded rows; every edition-3 ground truth and alternative is accepted by the full front end; every seed is refused with its code; both cards match their pins; the edition-2 card regenerates byte-for-byte from the skill; both editions' truths declare the parameters the prompt states.results/2026-11-09/: three runs per edition, every condition, 631 agent calls. Generatedsummary.md, plusanalysis.mdread by hand.Numbers (
claude-sonnet-5-5, three runs per edition, 75 tasks)m4asks for a helper and its caller)Shared tasks are the 45 that edition 2 expresses in one query, where both editions are judged without a person.
Gate verdict. Acceptance: passes (44–45 against 43). Correctness as defined (edition 2 by rows, edition 3 by typed IR): fails (23–25 against 39–40), and judged alike by typed IR it is still 23–25 against 28. Read by hand, the 22 edition-3 misses of run 1 are 16 equivalent spellings, 4 extra columns and 2 defensible alternatives, with none wrong. That is 39 against 39: a tie, not a gap. RFC finding 7 says why a machine cannot decide this half yet.
Edition 3 gained composition (every composition task in one query), the forms edition 2 lacks (arithmetic, conditionals,
union, recursion, computed writes, the catalogue) and the reserved edge (quoted every time, where edition 2 was refused withT7in all 8 tries). It lost nothing to a refusal or to a wrong answer on the hand reading. The repair codes per task and per seed are insummary.mdandanalysis.md.How it was tested
cargo fmt --all --check,cargo dev-clippy,typos,python3 scripts/check-docs.py: clean.cargo test -p omnigraph-gqt --locked: library 61 passed (the agent tests among them), the corpus 533 of 533, integration 18, runner dispatch 29.cargo dev-testexcludesomnigraph-gqt, the only crate this PR's own commits change, so the GQT command above stands in for it.CI
Every check passes at
cb0f4634. Earlier attempts failed only at steps that download GitHub release assets (HTTP 504 on the runners, infrastructure) and at the release-notes check, which #1316's75cb05fcfixed and this branch now carries.RFC findings
edition2_queries == 1), where both editions are judged automatically; the several-query tasks are reported as query counts and left to hand adjudication.$headcount as n), binds aletbefore returning it, or declares an integer parameter at another width. The instrument states each parameter's declared type in the prompt (from the canonical truth, checked equal to the edition-2 truth's) and lists equally canonical spellings as alternatives; it does not hand-adjudicate. The RFC's earlier 38–39 of 45 was read by hand; this number is the strict one.bm25has no window. "The top 2 by BM25 across the corpus, then who knows them" (r04) needs an inline read withorder { $score desc } limit 2aroundbm25, whilenearesttakesk:. Not wrong, but asymmetric; the card says it explicitly.updateon an edge. Sections 3 and 5 sayupdatetakes "a bound node or edge variable"; edition 2 refuses updating an edge (T16) and the RFC keeps today's codes. The card keeps the refusal.@index(fulltext, analyzer = …),Vector(3, distance = "cosine")), so both editions' tasks run against its edition-2 spelling (an L2 vector,@indexfull text). When edition 3 executes, the cosine field the RFC writes would ranknearestdifferently from the recorded edition-2 rows.let $n = count { … }and a filter for the block in the filter, an unaskedorderordistinct, ordering by a return alias. Either the gate's correctness half waits for execution (both editions by rows), or the normalisation it names covers these, starting with the formatter's canonical clause order.reachablereturning$q.namefrom a recursive read (T74: a relation drawn from, here by the recursion itself, needs variable-shaped columns), fixed with a second declaration; and the weighted hybrid drawing a second ranking of one population into a scope that already held its column (T68), fixed by renaming the second population first, as the before-and-after corpus already does. The edition-3 card's fusion paragraph said the same wrong thing and is corrected.Front-end findings (for PR 1316)
unionbetween two reads (match { … } return { … } union match { … } return { … }, seede3-26) is refused with the genericQ001at the declaration's closing brace:expected an operator that continues the expression or a clause (…); found }. It names neitherunionnor the form; the habit table could recognise it and printfrom union({ … }, { … }).T74names the wrong column and prints a colliding fix. Forreturn { $c, count($p) }drawn byfrom(seede3-29):the column p is not variable-shaped and cannot be drawn into scope, fixalias it on the inner return: … as p. The column iscount($p);as pwould collide with nothing here but reads as aliasing$p. No byte-range suggestion either.T74,T8, and edition 2's type codes alike), so an agent repairing without the card has only the message.T8foravg(count($p))(seede3-23) carries no fix, no suggestion and no position; the RFC's form is an inner grouped read drawn withfrom.T65's fix is a template, and without the card it does not repair. Fororder { nearest($d.embedding, $q) }the fix readsfrom nearest(<population>, $x.field, $q, k: <limit>) and order { $distance }, with no suggestion edits. Repairing from the diagnostic alone, the agent wrotefrom nearest($d, $d.embedding, $q, k: 20)after thematchblock, with the node variable as the population: 0 of 6 seeded repairs (e3-04,e3-05). The rewriter already knows the population (every variable the pattern binds); the fix could print the rewritten read, asQ008does fordelete T where.