transcript_refusals: count parsed transcripts before the agent filter, and name the filtered subset (#374)

[fdaviddpt]

scripts/transcript_refusals.py applied the --agent filter before computing transcripts_parsed, so a filtered run reported files as unparsed that were never offered to the parser.

Reproduced first

Three clean synthetic transcripts, one matching the filter, against the code at be36015:

FILTERED   {'transcripts_found': 3, 'transcripts_parsed': 1, 'unreadable_files': []}
UNFILTERED {'transcripts_found': 3, 'transcripts_parsed': 3, 'unreadable_files': []}

The issue was found with three synthetic transcripts and was not run against a real transcript directory. It was here, and the real figure is the argument for the fix: against this machine's own ~/.claude/projects/-Users-…-claude-oss tree, --agent oss:developer reported 160 of 489 parsed with unreadable_files: [] — 329 files rendered as parse failures, none of which failed.

The judgment call, stated

The issue offered two fixes and they are not equivalent. Moving the filter below the count makes found and parsed agree, but it destroys the information that a filter ran at all: a filtered run and an unfiltered one over the same directory then render identically apart from aggregates nobody compares. Both halves are taken here.

• transcripts_parsed is the count of files the parser actually read, taken before the filter, so transcripts_found - transcripts_parsed == len(unreadable_files) holds unconditionally. The gap can now only mean parse failures, and they are always named.
• The filtered subset is its own field, in three states rather than two: transcripts_matched_agent_filter is null when no filter was applied (there was no question), an integer when one was, and 0 is a real finding meaning it matched nothing. agent_filter echoes the filter itself.

After:

FILTERED   found 3, parsed 3, matched 1, agent_filter 'oss:developer', unreadable []
UNFILTERED found 3, parsed 3, matched null, agent_filter null

Tests

Red first, four failures on the counting (assert 1 == 3, assert 0 == 3). Green after: 37 passed in the module, 2615 passed, 2 skipped for the full suite.

The fixture is three transcripts with a filter matching one, with two controls in the same fixture as the issue asked: an unfiltered run over the same three proving found and parsed still agree, and a file that genuinely cannot be parsed proving unreadable_files still fills under a filter. A third control covers a file that cannot be opened — chmod 000, in its own test so its skip cannot swallow the deterministic assertions, with the deny confirmed by attempting Path.read_text (the exact call analyze_transcript makes) rather than trusting the mode bit, and skipping loudly with what went untested when it does not take.

Review

One reviewer finding, fixed in 12c88ae: the new field's prose claimed three states without saying they are the measured state's three. In no-transcripts-found the key is absent (as transcripts_parsed already was), so a consumer using .get() saw None — which the prose defined as no filter was applied — while agent_filter said one was. Corrected in the docstring and the README, and the shape pinned by a test with a populated control. Labelled as a contract pin rather than a regression: the code already behaved this way.

The auditor returned no findings across four classes.

Closes #374

Verified by the maintainer

Independent red re-run: this branch's tests/test_transcript_refusals.py on origin/main in a
detached worktree, product code untouched.

5 failed, 32 passed in 0.09s
assert 1 == 3   (transcripts_parsed under a filter)

Five, where the report says four — and the difference is not a disagreement. The author ran the
new tests targeted with the rest deselected; this is the whole module, so it also catches the
contract-pin test added during review. Recorded because two numbers for one run would otherwise read
as one of them being wrong later.

The chmod 000 case skipped here rather than asserting, and its skip names what went untested in
its own words, which is the behaviour that arm was written for.