Repository navigation
feat(evals): add the L1 retrieval and digest eval - #50
Merged
Merged
Conversation
Nothing measures whether recall and the digest find the right notes: the only check is a legacy query set over private notes, tuned on the queries it scores. This change specifies the L1 eval: a frozen synthetic dataset, six arms, per-stratum metrics with paired statistics, and digest false-injection at the default threshold.
A Python harness under uv runs the L1 eval: it generates and freezes the synthetic dataset through subscription CLIs with no user memory loaded, runs the arms in temporary homes against bilbo's pinned local embedder, and reports metrics, statistics and comparisons. A small invented fixture lets the harness's own tests run offline in CI.
Two reviews found ways the data run could freeze a wrong or underpowered dataset, and a CI failure showed run folders keeping absolute temp paths. Freeze now requires finished pooling with a per-split audit, applied resolutions bound to the query text and the preregistered test counts. The pool judge sees the passages that ranked a note instead of its first 3,000 characters. Run records keep no expanded temporary path, and generation sessions now refuse any loaded user instructions.
A second review pass found that a typical pool item split into two calls, that a prompt could not be rebuilt once pool --apply changed a query, and that judgments survived edits to the notes they judged. Pool calls now record the inputs their prompt is built from, and a judgment goes back to pending when its candidate or evidence notes change.
A third review pass found that a reviewer's old verdict still covered a new judgment after a note edit, so freeze could pass on a wrong label, and that a cached output could stand in for a call whose prompt had changed. Re-judged items now lose their old resolutions, reuse no output built from another prompt, and keep every superseded output. Pool records name the harness that built them.
The last review pass found that a note added as gold by a pool resolution kept relevance 1 after a review edit, that a reviewer's reject of such a pair was ignored, and that a note edit dropped a pending rewrite request. Freeze now refuses until the pair is judged and resolved again, apply honors the reject, and rewrites survive.
bilbo library land needs --title when a capture has no level-1 heading, which is every SQLite C API page, so the generator skipped them and the library came out biased toward long prose pages. It now always passes a title, logs the real land error, and can redo the library step before any query depends on it.
A review of the dev labels found a fifth of the digest positives built from a superseded fact, multi-hop questions stitched from unrelated facts, and notes that mention their own generation brief. Positives now come from the newest fact of a family, multi-hop queries must carry a real chain, and fidelity re-renders any note that leaks its instructions. Redo flags fix the items already generated.
The leak pattern also matched "the manifest", a real term in Android and package notes, so a sound note was re-rendered until one of its facts was dropped, and dataset check still passed the queries built on that fact. The pattern now names only the brief and instructions, and check refuses any item whose fact was dropped.
A malformed multi-hop output left the test split one query short of its preregistered count, and nothing replaced it. generate queries now fills a short stratum from facts it has not used, under new ids that never reuse a dropped one, so the split meets its preregistration without lowering it.
The probe asks the query model which tools it has, and it sometimes lists Codex's built-in clock, image and web tools, which a clean CODEX_HOME still carries, so steps failed at random. Those exact names are now allowed; any other tool still refuses the session.
The test validity review found that every three-note multi-hop item carried a note answering no part of the question, while every two-note item was sound. Multi-hop queries now chain exactly two notes. The validity share leaves out the sampled rows of a class that a generator fix removed entirely, and prints the raw share beside it.
The judge confused 26-character note ids that share a prefix, repeating one and omitting another until an item failed every re-roll. It now sees c01, c02 and so on per item; the ids are mapped back before validation, and pool records keep the mapping so prompts still rebuild exactly.
The harness grew past what one eval needs. The L1 eval is now one script in evals/l1-retrieval with the frozen notes-synth/v1 dataset and a baseline for bilbo 0.19.0. The generator that built the dataset stays reachable at the tag evals/notes-synth-v1-generator, so the harness, its tests and the OpenSpec change go. CI runs the eval as a smoke run that fails only on its guards.
The previous commit removed the old harness but missed the new eval. This adds evals/l1-retrieval: eval.py, its pins, the frozen notes-synth/v1 export with its datasheet, and the 0.19.0 baseline, plus the CI smoke run and the AGENTS.md commands that use them.
The datasheet still described the generator's layout, the README's run example aborted, and releases had no step that records a baseline. The dataset id now hashes the data files only, so a datasheet edit no longer invalidates a baseline. The run reports the preregistered sign-flip test beside the bootstrap intervals, bm25 applies the kind filter before its cut, and a killed run stops the llama-server it started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Nothing measured whether
bilbo recalland the digest find the right notes. The only check was 39 legacy queries over private notes, tuned on the same queries, so no release could show it ranks better or worse than the last one, or than plain BM25.What changed
evals/l1-retrieval/eval.py: one script (~800 lines). It builds a store from the dataset in a temp HOME, startsllama-serverwith bilbo's pinned GGUF, runs six arms (oracle, random, ripgrep, bm25, bilbo keyword-only, bilbo full), scores them withir_measuresper stratum, and measures the digest at 0.55.runwrites a results file and a report;diffcompares two runs with paired intervals. Guards abort a run on anything suspect: the oracle not scoring 1.0, unparsed recall output, unknown ids, notes not indexed, home paths in the output.evals/l1-retrieval/dataset/: notes-synth/v1, frozen. 630 synthetic notes over 14 invented projects, 60 public-domain SQLite pages, 366 queries in 9 strata and 436 digest prompts, split dev/test by project.dataset/README.mdis the datasheet: how it was made, label quality, known limits. The generator that built it is at the tagevals/notes-synth-v1-generator.evals/l1-retrieval/baseline/0.19.0.{json,md}: the test split, run once on the 0.19.0 release build.evalsjob runs the eval as a smoke run (oracle, bm25, keyword-only, no embedder) that fails only on its guards.AGENTS.mdandCONTRIBUTING.md(releases) say how to run it and record a baseline.No Rust change.
Results on 0.19.0 (test split, 160 scored queries)
How the dataset was made
Facts first: Sonnet 5.5 (
claude -p) rendered notes from planted facts, gpt-6.1-sol (codex exec) wrote blind queries from the facts and checked fidelity, a leakage filter rejected queries that echo their note, and gold was completed by pooling every arm's top 10 through an LLM judge. Fresh Opus 5.5 agents reviewed the labels: 2,276 audited "no" judgments (2,274 confirmed) and a validity sample at 97.0% raw, 98.6% excluding a class removed by a generator fix. All generation sessions ran with no user memory, skills, hooks or MCP loaded. The datasheet lists every limit.For the owner
dataset/README.mdholds rounded aggregate numbers of your real note store (notes per kind, length percentiles, share in Portuguese). No content. You approved this.b4af629) followed by the add (99fd31f); squash on merge if you prefer.How to test
nix develop -c cargo build --lockednix develop -c uv --directory evals/l1-retrieval sync --lockednix develop -c uv --directory evals/l1-retrieval run eval.py run --bilbo "$PWD/target/debug/bilbo" --split dev --arms oracle,bm25,bilbo-keyword --out "${TMPDIR:-/tmp}/results.json"nix develop -c uv --directory evals/l1-retrieval run eval.py diff baseline/0.19.0.json baseline/0.19.0.jsonexits 0.