Skip to content

test(discovery): check planted canaries and gate the run - #64

Closed
pengyuzhang wants to merge 1 commit into
pr/07-outcomesfrom
pr/08-verdict
Closed

test(discovery): check planted canaries and gate the run#64
pengyuzhang wants to merge 1 commit into
pr/07-outcomesfrom
pr/08-verdict

Conversation

@pengyuzhang

Copy link
Copy Markdown
Collaborator

Stacked on #63.

Two things that decide whether a run may pass, kept apart from the measurement they sit on.

The canary check is deliberately crude

It does not know how redaction is implemented, does not walk the asset model, and does not care which field a value came from. It searches the serialized snapshot — every byte the collector would have written to a file or posted to a server — for each planted string.

That crudeness is the point. A credential that leaked into a field the harness does not model is exactly the one a smarter, model-aware check would miss.

Any hit fails the run outright, regardless of every other score. A collector that finds every tool and leaks one token has not had a good run.

Two details worth reviewing:

  • A canary with no recorded value is reported as unplanted, not clean. A check that never ran is not a check that passed.
  • The context line around a hit masks the value it reports. Printing it into a document that then gets shared would repeat exactly the mistake being reported.

The gate

Fails on: a leaked canary, a dirty baseline, an unexplained error, any DUP, a missed review-queue entry, or recall below the last accepted run for that OS.

Recall is compared against history, not an absolute threshold. A real endpoint is not perfectly reproducible — a background updater can change a version mid-run — so an absolute number would either be set so low it gates nothing or would flap. Everything else is binary, because a leaked credential or an invented asset is not weather.

Verification

$ python3 -m unittest discover -s tests -t . -q
Ran 58 tests in 0.039s
OK

Ten new tests, each injecting exactly one condition into the recorded run: a leaked canary, an unplanted one, a dirty baseline, an unexplained error, a repeated asset id, and a recall regression against a previous score. On the recorded run itself:

canaries: 6 planted, 0 leaked, clean
gate:     FAILED - duplicates

which is correct: that fixture carries an injected duplicate.

Two things that decide whether a run is allowed to pass, kept apart from the
measurement they sit on top of.

The canary check is deliberately the crudest test in the harness. It does not
know how redaction is implemented, does not walk the asset model, and searches
the serialized snapshot - every byte the collector would have written out - for
each planted value. A value that leaked into a field nobody modelled is exactly
the one a smarter check would miss. Any hit fails the run outright regardless
of every other score: a collector that finds every tool and leaks one token has
not had a good run.

A canary with no recorded value is reported as unplanted rather than clean,
because a check that never ran is not a check that passed. The context line
around a hit masks the value it reports - printing it into a document that then
gets shared would repeat exactly the mistake being reported.

The gate fails on a leaked canary, a dirty baseline, an unexplained error, any
duplicate, a missed review-queue entry, or recall below the previous accepted
run for that OS. Recall is compared against history rather than an absolute
threshold because a real endpoint is not perfectly reproducible - a background
updater can change a version mid-run - while a leaked credential or an invented
asset is not weather and fails immediately.
@pengyuzhang
pengyuzhang deleted the pr/08-verdict branch August 23, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants