Skip to content

test(discovery): compute TP/FP/FN/DUP and field accuracy - #63

Open
pengyuzhang wants to merge 1 commit into
pr/06-matchfrom
pr/07-outcomes
Open

test(discovery): compute TP/FP/FN/DUP and field accuracy#63
pengyuzhang wants to merge 1 commit into
pr/06-matchfrom
pr/07-outcomes

Conversation

@pengyuzhang

Copy link
Copy Markdown
Collaborator

Stacked on #62.

The measurement itself, and the shape of score.json. The run verdict — canaries and the gate — is the next PR: what the numbers are should not change when someone argues about what ought to fail a release.

Outcomes

outcome condition why separate
TP installed, matched exactly once
FP asset matches no installed entry an invention; costs an operator real time
FN installed entry matches no asset a blind spot
DUP installed once, matched by ≥2 assets not a partial success

DUP is tracked apart from TP rather than folded into it. A duplicate inflates a fleet inventory, and treating it as mostly-a-success is what let it recur.

Only installed entries are scored

An entry a vendor no longer ships (unavailable), one that failed, or one no recipe implements yet leaves the denominator — but visibly. score.json reports each count, because a silently shrinking denominator flatters every recall number computed after it.

Field accuracy is never blended

Per field, over true positives only. A collector that always gets version right and always gets config_scope wrong has a specific bug; an average hides which. Version and path are compared against what the runner recorded installing, not against the manifest — the manifest states intent, and intent is not what is on the disk.

Empty denominators report null, not 1.0

A category with nothing installed has no recall. Recording a perfect score would flatter any average computed downstream.

Every finding is keyed by manifest id

So a regression reads as M-SITE-08 went from TP to FN, not as "MCP recall dropped".

Verification

Against the recorded run from #61, whose four defects are known:

$ python3 -m unittest discover -s tests -t . -q
Ran 48 tests in 0.023s
OK

totals:  tp=73 fp=2 fn=2 dup=5  recall=0.9733 precision=0.9733
misses:  M-SITE-08, T-RT-04                       (the two injected)
dups:    AG-01, AG-08, T-CHAN-01, T-CHAN-04, T-CLI-01   (one binary, two names)

The duplicate spanning five entries is correct and worth understanding: one tool reachable by two paths duplicates the base entry, both variants keyed to it, and the two states attached to it.

The measurement itself: for each entry recorded installed, how many assets
claimed it.

DUP is its own outcome rather than a partial success folded into TP. A
duplicate inflates a fleet inventory, and treating it as most of a success is
what let it recur. An asset no entry claims is an FP, attributed to the
negative control that explains it where one does - an FP naming the control
that provoked it is a bug report, and one naming nothing is a mystery.

Only entries recorded installed are scored. An entry a vendor no longer ships,
or that failed to install, leaves the denominator - but visibly, because a
silently shrinking denominator flatters every recall number computed after it.

Field accuracy is over true positives only and reported per field rather than
blended: a collector that always gets version right and always gets
config_scope wrong has a specific bug, and an average hides it. Version and
path are compared against what the runner recorded installing rather than
against the manifest, because the manifest states intent and intent is not what
is on the disk.

Recall and precision are None where the denominator is empty rather than 1.0. A
category with nothing installed has no recall, and recording a perfect score
for it would flatter any average somebody later computes from the file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants