Skip to content

Test check_query_denominator; record measured harness coverage (#819) - #822

Merged
joshfactorial merged 1 commit into
developfrom
test/819_audit_harness
Oct 7, 2026
Merged

joshfactorial merged 1 commit into
developfrom
test/819_audit_harness

Conversation

@joshfactorial

Copy link
Copy Markdown
Collaborator

Steps 1 and 2 of the #819 testing audit. Test and docs only; no change to eidolon or to what the pipeline does on Delta.

Step 1: #[ignore] tests (no change needed)

release-gates runs on every PR into develop. The log of run 37473380770 (#821) shows all 8 ignored tests executing and passing, each in 7–30 s: five realigned SV gates, the transition-matrix round trip, the golden-BAM NM check and the validate differential. Each one fails with a message if its tool is missing; none can skip silently. The workflow header said "six" and named only the gates and the differential. It now lists them without a count.

Step 2: harness coverage of sv_pipeline.sbatch

All 18 suites in scripts/delta/tests/ pass, and all 232 of their mutations are caught locally. CI already runs both modes and fails a suite that reports no mutation summary.

Measured: 25 of 29 functions are exercised, counting functions reached through a tested caller (run_truvari_both → run_truvari → comp_sample, check_denominator). This replaces CLAUDE.md's rule-6 example, which still said 2 of 14 and listed check_denominator as untested.

Untested: score_caller, build_bnd_spans, convert_manta_inversions, index_and_align. These are reported on #819, not fixed here.

The defect fixed

test_scorer_calibration.sh never extracted check_query_denominator. Every run_truvari in the suite therefore called an undefined command, and || QUERY_FILTERED+=(…) flagged each stratum. Output went to /dev/null and nothing asserted QUERY_FILTERED, so the suite reported 132 passed. Shown before the fix: check_query_denominator: command not found, then QUERY_FILTERED=regression regression_allcalls.

Fix:

  • The function is extracted.
  • The filter stub writes TP-comp, as truvari does. Without it, every case took the "UNVERIFIED" branch.
  • New tests:
    • a summary missing TP-comp/FP is reported UNVERIFIED and not flagged;
    • a query with none of its 4 records scored is flagged, counted, and blamed on --passonly (4 non-PASS);
    • a fully scored query is not flagged (must-not-fire);
    • through run_truvari_both, an all-non-PASS query flags exactly the PASS-only pass, and an all-PASS query flags nothing (must-not-fire).
  • Mutations, all caught: reading missing keys as dropped; run_truvari ignoring the result; counting no non-PASS records.
  • bash -x shows no command not found. 142 passed, 0 mutations survived.

Not verified

  • Only the stubs were exercised; no real truvari ran. The stub's TP-comp is derived the way truvari defines it, but it has not been compared against a real truvari summary.

Refs #819.

🤖 Generated with Claude Code

test_scorer_calibration.sh never extracted check_query_denominator, so
every run_truvari in the suite called it as an undefined command (rc 127)
and pushed each stratum onto QUERY_FILTERED with nothing asserting it.
Extract it, make the filter stub write TP-comp as truvari does, and test
its three outcomes plus the QUERY_FILTERED wiring, with three mutations.

Replace CLAUDE.md's stale rule-6 example with the measured state (25 of
29 sv_pipeline.sbatch functions exercised) and stop release-gates.yml
from counting its ignored tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joshfactorial
joshfactorial merged commit 534c818 into develop Oct 7, 2026
7 checks passed
@joshfactorial
joshfactorial deleted the test/819_audit_harness branch October 7, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant