test(discovery): compute TP/FP/FN/DUP and field accuracy - #63
Open
pengyuzhang wants to merge 1 commit into
Open
Conversation
The measurement itself: for each entry recorded installed, how many assets claimed it. DUP is its own outcome rather than a partial success folded into TP. A duplicate inflates a fleet inventory, and treating it as most of a success is what let it recur. An asset no entry claims is an FP, attributed to the negative control that explains it where one does - an FP naming the control that provoked it is a bug report, and one naming nothing is a mystery. Only entries recorded installed are scored. An entry a vendor no longer ships, or that failed to install, leaves the denominator - but visibly, because a silently shrinking denominator flatters every recall number computed after it. Field accuracy is over true positives only and reported per field rather than blended: a collector that always gets version right and always gets config_scope wrong has a specific bug, and an average hides it. Version and path are compared against what the runner recorded installing rather than against the manifest, because the manifest states intent and intent is not what is on the disk. Recall and precision are None where the denominator is empty rather than 1.0. A category with nothing installed has no recall, and recording a perfect score for it would flatter any average somebody later computes from the file.
lghupan
approved these changes
Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #62.
The measurement itself, and the shape of
score.json. The run verdict — canaries and the gate — is the next PR: what the numbers are should not change when someone argues about what ought to fail a release.Outcomes
DUPis tracked apart fromTPrather than folded into it. A duplicate inflates a fleet inventory, and treating it as mostly-a-success is what let it recur.Only installed entries are scored
An entry a vendor no longer ships (
unavailable), one that failed, or one no recipe implements yet leaves the denominator — but visibly.score.jsonreports each count, because a silently shrinking denominator flatters every recall number computed after it.Field accuracy is never blended
Per field, over true positives only. A collector that always gets
versionright and always getsconfig_scopewrong has a specific bug; an average hides which. Version and path are compared against what the runner recorded installing, not against the manifest — the manifest states intent, and intent is not what is on the disk.Empty denominators report
null, not 1.0A category with nothing installed has no recall. Recording a perfect score would flatter any average computed downstream.
Every finding is keyed by manifest id
So a regression reads as
M-SITE-08 went from TP to FN, not as "MCP recall dropped".Verification
Against the recorded run from #61, whose four defects are known:
The duplicate spanning five entries is correct and worth understanding: one tool reachable by two paths duplicates the base entry, both variants keyed to it, and the two states attached to it.