test(discovery): execute the manifest in dependency order

[pengyuzhang]

Stacked on #70.

Two jobs, and the second is what the scorer depends on. The runner installs what it can, then records per entry id what the machine really ended up with. manifest.actual.json, not the manifest, is what scoring compares against — an install that failed, or that a vendor no longer ships here, must never be scored as a miss.

Dependency order, not manifest order

1 baseline-prereq          already in the image, verified only
2 npm / pipx / bin         binaries on PATH
3 app-installer            VS Code must exist before its extensions
4 vscode-ext               depends on 3
5 service                  start, wait for port, pull model
6 declare-mcp              config sites depend on their host app
7 artifact                 files and links
8 channel-variant          second installs, after the first ones
9 scheduler / identity     states that persist without a process
10 runtime-state           last: processes must be alive at scan #2

Step 6 after step 3 is the subtle one. Several MCP declaration sites live inside an application's own config directory, and writing M-SITE-08 before JetBrains exists creates a path the collector may treat differently from one the application itself created.

Blocked entries are unimplemented, not failed

The entry did not fail; the thing it needed was never there. Scoring it as a miss would blame the collector for the harness's own gap — and a second install of a tool that was never installed once proves nothing about duplication.

Canaries are substituted in one place

Generated once per run, applied here rather than in each recipe, so a credential exists in exactly one place: the run directory. A recipe that built its own would plant a value the redaction check never searches for, and the run would report a clean check it never made.

Verification

Dry guest, all three platforms:

linux  applicable=105 installed=52 failed=0 unimplemented=53 files=40
mac    applicable=110 installed=51 failed=0 unimplemented=59 files=40
win    applicable=103 installed=51 failed=1 unimplemented=51 files=39

$ python3 -m unittest discover -s tests -t . -q
Ran 110 tests in 0.076s
OK

The one Windows failure is a real finding, left visible

N-09 | wrote /usr/local/bin/claude-old but it is not there afterwards

N-09 is specified as a dangling symlink at /usr/local/bin/claude-old and marked applicable on all three platforms, but that path is POSIX-only — the entry cannot exist on Windows as written. Every other per-OS path in the manifest is a three-way mapping; this one is a bare string, inherited from the way the row is written in tests/README.md.

It is left failing rather than papered over, because the failure is the harness correctly reporting that it could not do what it was asked. Fixing it means deciding what the Windows equivalent is — a broken shortcut on PATH, most likely — which is a manifest change and belongs with the Windows work, where it can actually be validated. Windows is unvalidated in this series either way; see #68 on why no QEMU driver ships here.

Live results

These recipes, this runner, against a real Ubuntu 24.04 aarch64 guest:

105 applicable · 52 installed · 0 failed · 53 unimplemented   (61s)

Verified on the guest rather than from the runner's own report: 8 CLI binaries on PATH at their pinned versions, 12/12 config sites written, /etc/claude-code/managed-settings.json owned root:root, N-09's link present and dangling, 28 artifact files.