test(discovery): install the three no-installer families - #70
Closed
pengyuzhang wants to merge 1 commit into
Closed
Conversation
A recipe installs, verifies and locates in one call, because the three are inseparable: a recipe that installed without verifying would let the runner record a version it never saw, and the scorer would then compare the collector's answer against the manifest's intention rather than against the disk. Three families are implemented - declare-mcp (27 entries), artifact (24) and npm-global (9). Between them that is 60 of Linux's 105 entries with no vendor installer, no GUI session and no model downloads, and they are the two categories where the collector does the most inference, so the cheapest half of the manifest exercises the most interesting logic. The other eleven declare themselves and record `unimplemented`. That is a deliberate fourth status beside installed, unavailable and failed: it keeps those entries out of the denominator and keeps them loud. A harness that silently omitted them would report a recall computed over whatever happened to work. Each names the phase it belongs to rather than inheriting a default stub, so adding a family to the manifest without a recipe fails loudly. Every recipe verifies what it created. On the first live run against an Ubuntu guest, N-09's symlink was silently absent and recorded `installed` - a write that did not land, reported as a success. That is the worst failure this harness can have, because it manufactures a defect in the thing it is measuring: the scorer would have reported the collector missing a file that was never there.
lghupan
approved these changes
Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #69.
A recipe installs, verifies and locates in a single call, because the three are inseparable: one that installed without verifying would let the runner record a version it never saw, and the scorer would then compare the collector against the manifest's intention rather than against the disk.
Three families, chosen deliberately
declare-mcpartifactnpm-globalBetween them, 60 of Linux's 105 entries with no vendor installer, no GUI session, no network beyond a registry and no model downloads. They are also the two categories where the collector does the most inference, so the cheapest half of the manifest exercises the most interesting logic.
The other eleven record
unimplementedA deliberate fourth status beside
installed/unavailable/failed. It keeps those entries out of the denominator and keeps them loud — a harness that silently omitted them would report a recall computed over whatever happened to work.Each names the phase it belongs to rather than inheriting a default stub, so adding a family to the manifest without a recipe fails loudly instead of quietly no-op'ing.
The same status covers an entry whose vendor source is unresolved. A URL nobody has filled in is a harness gap, and must never be reported as a vendor that stopped shipping.
Every recipe verifies what it created
This came from a live run, not from review. On the first pass against an Ubuntu guest,
N-09's symlink was silently absent and recordedinstalled— the transport could not write to/usr/local/binand nothing checked.That is the worst failure available to this harness, because it manufactures a defect in the thing it measures: the scorer would have reported the collector missing a file that was never there. Both implemented file-writing recipes now check afterwards and record
failedwith the real reason.Verification
The live results these recipes produced are in the runner PR that follows.