test(discovery): check planted canaries and gate the run - #64
Closed
pengyuzhang wants to merge 1 commit into
Closed
Conversation
Two things that decide whether a run is allowed to pass, kept apart from the measurement they sit on top of. The canary check is deliberately the crudest test in the harness. It does not know how redaction is implemented, does not walk the asset model, and searches the serialized snapshot - every byte the collector would have written out - for each planted value. A value that leaked into a field nobody modelled is exactly the one a smarter check would miss. Any hit fails the run outright regardless of every other score: a collector that finds every tool and leaks one token has not had a good run. A canary with no recorded value is reported as unplanted rather than clean, because a check that never ran is not a check that passed. The context line around a hit masks the value it reports - printing it into a document that then gets shared would repeat exactly the mistake being reported. The gate fails on a leaked canary, a dirty baseline, an unexplained error, any duplicate, a missed review-queue entry, or recall below the previous accepted run for that OS. Recall is compared against history rather than an absolute threshold because a real endpoint is not perfectly reproducible - a background updater can change a version mid-run - while a leaked credential or an invented asset is not weather and fails immediately.
lghupan
approved these changes
Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #63.
Two things that decide whether a run may pass, kept apart from the measurement they sit on.
The canary check is deliberately crude
It does not know how redaction is implemented, does not walk the asset model, and does not care which field a value came from. It searches the serialized snapshot — every byte the collector would have written to a file or posted to a server — for each planted string.
That crudeness is the point. A credential that leaked into a field the harness does not model is exactly the one a smarter, model-aware check would miss.
Any hit fails the run outright, regardless of every other score. A collector that finds every tool and leaks one token has not had a good run.
Two details worth reviewing:
unplanted, not clean. A check that never ran is not a check that passed.The gate
Fails on: a leaked canary, a dirty baseline, an unexplained error, any
DUP, a missed review-queue entry, or recall below the last accepted run for that OS.Recall is compared against history, not an absolute threshold. A real endpoint is not perfectly reproducible — a background updater can change a version mid-run — so an absolute number would either be set so low it gates nothing or would flap. Everything else is binary, because a leaked credential or an invented asset is not weather.
Verification
Ten new tests, each injecting exactly one condition into the recorded run: a leaked canary, an unplanted one, a dirty baseline, an unexplained error, a repeated asset id, and a recall regression against a previous score. On the recorded run itself:
which is correct: that fixture carries an injected duplicate.