Skip to content

test(discovery): install the three no-installer families - #70

Closed
pengyuzhang wants to merge 1 commit into
pr/13-writers-bodiesfrom
pr/14-recipes
Closed

test(discovery): install the three no-installer families#70
pengyuzhang wants to merge 1 commit into
pr/13-writers-bodiesfrom
pr/14-recipes

Conversation

@pengyuzhang

Copy link
Copy Markdown
Collaborator

Stacked on #69.

A recipe installs, verifies and locates in a single call, because the three are inseparable: one that installed without verifying would let the runner record a version it never saw, and the scorer would then compare the collector against the manifest's intention rather than against the disk.

Three families, chosen deliberately

family entries needs
declare-mcp 27 a JSON/TOML/YAML writer
artifact 24 file and symlink writes
npm-global 9 one command

Between them, 60 of Linux's 105 entries with no vendor installer, no GUI session, no network beyond a registry and no model downloads. They are also the two categories where the collector does the most inference, so the cheapest half of the manifest exercises the most interesting logic.

The other eleven record unimplemented

A deliberate fourth status beside installed / unavailable / failed. It keeps those entries out of the denominator and keeps them loud — a harness that silently omitted them would report a recall computed over whatever happened to work.

Each names the phase it belongs to rather than inheriting a default stub, so adding a family to the manifest without a recipe fails loudly instead of quietly no-op'ing.

The same status covers an entry whose vendor source is unresolved. A URL nobody has filled in is a harness gap, and must never be reported as a vendor that stopped shipping.

Every recipe verifies what it created

This came from a live run, not from review. On the first pass against an Ubuntu guest, N-09's symlink was silently absent and recorded installed — the transport could not write to /usr/local/bin and nothing checked.

That is the worst failure available to this harness, because it manufactures a defect in the thing it measures: the scorer would have reported the collector missing a file that was never there. Both implemented file-writing recipes now check afterwards and record failed with the real reason.

Verification

$ python3 -m unittest discover -s tests -t . -q
Ran 86 tests in 0.046s
OK

implemented: ['artifact', 'declare-mcp', 'npm-global']
pending:     11 families
every family resolves to a recipe or a stated gap: True

The live results these recipes produced are in the runner PR that follows.

A recipe installs, verifies and locates in one call, because the three are
inseparable: a recipe that installed without verifying would let the runner
record a version it never saw, and the scorer would then compare the
collector's answer against the manifest's intention rather than against the
disk.

Three families are implemented - declare-mcp (27 entries), artifact (24) and
npm-global (9). Between them that is 60 of Linux's 105 entries with no vendor
installer, no GUI session and no model downloads, and they are the two
categories where the collector does the most inference, so the cheapest half of
the manifest exercises the most interesting logic.

The other eleven declare themselves and record `unimplemented`. That is a
deliberate fourth status beside installed, unavailable and failed: it keeps
those entries out of the denominator and keeps them loud. A harness that
silently omitted them would report a recall computed over whatever happened to
work. Each names the phase it belongs to rather than inheriting a default stub,
so adding a family to the manifest without a recipe fails loudly.

Every recipe verifies what it created. On the first live run against an Ubuntu
guest, N-09's symlink was silently absent and recorded `installed` - a write
that did not land, reported as a success. That is the worst failure this
harness can have, because it manufactures a defect in the thing it is
measuring: the scorer would have reported the collector missing a file that was
never there.
@pengyuzhang
pengyuzhang deleted the pr/14-recipes branch August 23, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants