docs(discovery): add the ADR Discovery test plan

[pengyuzhang]

Adds Discovery/tests/README.md, the specification for how the ADR Discovery endpoint collector is measured end to end. One file, no code.

The method

Install a known set of AI tools on a real machine, scan it, and check the collector reports exactly that set — no misses, no inventions, right facts about each one. The comparison between the install manifest and the snapshot is the test result.

Three virtual machines, one per operating system (macOS, Ubuntu LTS, Windows 11). VMs rather than containers because launchd, the Windows registry, Task Scheduler and GUI application bundles are precisely the surfaces several probes exist to read, and no container has any of them. Each run restores a clean snapshot, scans for a baseline, installs the manifest, scans again, and scores the delta — so every reported asset can be attributed to something deliberately installed.

The manifest is a complete, executable inventory

120 entries, each with a stable id. The id is the unit the runner executes, that a run records an outcome against, and that a scorecard reports a miss under. Nothing is left as "and a few others" — an entry not listed is not tested.

Category	Entries	ids
AI tools	50	T-CLI-01…12 · T-APP-01…16 · T-EXT-01…05 · T-RT-01…09 · T-CHAN-01…08
MCP servers	29	M-SITE-01…14 · M-PIN-01…09 · M-SP-01…06
Skills & programmable surface	19	S-01…19
Agents	12	AG-01…12
Negative controls	10	N-01…10

Grouped the way the collector reports, so each table maps onto a slice of the snapshot:

• AI tools split by evidence channel — CLI agents, desktop apps and AI browsers, IDE extensions, model runtimes — plus eight named install-channel variants. T-CHAN-04 is the usr-merge pair that duplicated every PATH-installed agent on mainstream Linux.
• MCP servers on two axes: all 14 declaration sites the collector reads, and nine launch forms driving the supply-chain verdict, plus six special cases (undeclared server, enterprise-over-user scope precedence, three credential shapes, malformed bundle).
• Skills as exact paths rather than directories — ~/.claude/skills/pdf-filler/SKILL.md, not "the skills directory".
• Agents as the states the machine is left in before the second scan: running, spawned child, launchd/cron/systemd/Task Scheduler, and four authenticated identities.
• Negative controls as specific software, because precision is unmeasurable without them and a collector that reported everything would score perfectly on every other table.

Verified, not asserted

• 120 ids, none duplicated, contiguous numbering per group
• per-category counts match the summary table exactly
• all 42 of 42 catalog entries covered by an AI-tool row

That last property is worth keeping: a catalog entry with no manifest row is a tool the collector claims to recognize but that nothing ever verifies.

Reviewer notes

This is a specification, not an implementation. There is no VM provisioning, no manifest runner and no scoring script. Nothing here runs.

It documents a module that is not on main yet. The collector lives on an unmerged branch, so the document references things this repository does not currently contain — FIXTURE_SUITE.md, the adr-discovery CLI, diff_snapshots, and the 42-entry catalog the manifest was written against. A reviewer cannot verify the coverage claim from main alone.

Merging this puts a test plan in the tree ahead of the thing it tests. That is a deliberate choice to review the methodology on its own; if you would rather it land with the collector, close this and it will come in with that branch instead.

🤖 Generated with Claude Code

[pengyuzhang]

Pushed a second commit (77ddb41) that reworks the manifest, in response to the observation that the first version named categories but not contents — an implementer could not tell which tools to put on the VMs.

The manifest is now a complete enumerated inventory: 120 entries, each with a stable id, replacing the vague parts. "At least three of the above are installed a second way" became eight named T-CHAN rows; skills went from directories to exact file paths; agents went from item types to the states the machine is left in; negative controls went from examples to specific software.

The PR description above has been updated to match. Diff is unchanged in shape — still one file, no code.