test(discovery): prove the scorer responds to known faults

[pengyuzhang]

Stacked on #64.

The unit tests check that each piece behaves. This checks the instrument as a whole, the way you would check any instrument: feed it a known input, change one known thing, and see whether the reading moves by the amount it should.

$ python3 -m tests.tools.faultcheck

control, no fault injected:  tp=80 fp=0 fn=0 dup=0  recall=1.0 precision=1.0  gate=passed
control is clean:            yes

injected fault                       expected signal                  detected  gate   totals that moved
--------------------------------------------------------------------------------------------------------
misses a declared server             fn +1, named M-SITE-08           yes       pass   fn,tp
reports one tool twice               dup on every entry keyed to it   yes       fail   dup,tp
invents an asset                     fp +1, unattributed              yes       pass   fp
falls for a lookalike                fp +1, blamed on N-07            yes       pass   fp
reports a wrong version              version accuracy < 1             yes       pass   none
reports a wrong config scope         config_scope accuracy < 1        yes       pass   none
calls a mutable tag pinned           pinned accuracy < 1              yes       pass   none
leaks a planted credential           canary leaked, gate fails        yes       fail   none
goes silent on the unknown tool      review queue fails               yes       fail   none
trips over something unexplained     errors.unexplained +1            yes       fail   none
recall below the last accepted run   gate fails                       yes       fail   fn,tp
--------------------------------------------------------------------------------------------------------
11/11 faults detected - PASS

Both halves matter

The control line is the half that is easy to forget. A scorer that misses a planted duplicate is broken; so is one that reports defects nobody planted, because every false alarm costs somebody the afternoon it takes to prove the collector was fine. So each case asserts the whole delta, not just the number it was looking for — the totals that moved column is part of the check.

The none rows are worth reading: a wrong version on a tool that was found is a field failure, not a miss, and the scorer keeps those separate.

Faults are real failure modes

Not arbitrary mutations. Each is something a collector has done or plausibly could: a missed declaration site, usr-merge double-counting, a lookalike believed, :latest read as pinned.

Runs for each OS: --os mac, --os win. --json for machine-readable output. Exit code is non-zero if any fault goes undetected, so it can gate.

Verification

Passes 11/11 on all three platforms.