Skip to content

test(probes,detectors): add unit tests for several uncovered modules#1853

Open
nuthalapativarun wants to merge 1 commit into
NVIDIA:mainfrom
nuthalapativarun:test/detector-probe-coverage-sweep
Open

test(probes,detectors): add unit tests for several uncovered modules#1853
nuthalapativarun wants to merge 1 commit into
NVIDIA:mainfrom
nuthalapativarun:test/detector-probe-coverage-sweep

Conversation

@nuthalapativarun

Copy link
Copy Markdown
Contributor

Tell us what this change does. If you're fixing a bug, please mention
the github issue number.

This adds unit test coverage for several probe and detector modules that
currently have no tests:

  • detectors.visual_jailbreak.FigStep
  • detectors.any.AnyOutput / AnyNonspaceOutput
  • detectors.lmrc.Anthro / QuackMedicine
  • detectors.leakreplay.StartsWith
  • detectors.goodside.RileyIsnt / PlainJSON / Glitch
  • probes.misleading.FalseAssertion
  • probes.av_spam_scanning.EICAR / GTUBE / GTphish
  • probes.realtoxicityprompts.RTPBlank / dynamically generated RTP* classes
  • probes.malwaregen.TopLevel / SubFunctions / Evasion / Payload

Please ensure you are submitting from a unique branch in your repository to main upstream.

Verification

List the steps needed to make sure this thing works

  • Supporting configuration such as generator configuration file - n/a, test-only change
  • garak -t <target_type> -n <model_name> - n/a, test-only change
  • Run the tests and ensure they pass python -m pytest tests/
  • Verify the thing does what it should - new tests assert specific prompt counts, substring lists, detector wiring, and detect() scoring behaviour (e.g. word-boundary matching, case sensitivity, JSON validity, inverted trigger-list scoring)
  • Verify the thing does not do what it should not - tests include negative cases (no-hit outputs, missing notes, None outputs)
  • Document the thing and how it works - test names and assertion messages describe the expected behaviour

Adds coverage for visual_jailbreak, any, lmrc, leakreplay, and goodside
detectors, and for misleading, av_spam_scanning, realtoxicityprompts,
and malwaregen probes.

Signed-off-by: Varun Nuthalapati <nuthalapativarun@gmail.com>
@nuthalapativarun

Copy link
Copy Markdown
Contributor Author

Friendly bump - this PR is mergeable with green CI and hasn't had a first review yet. Would appreciate a look when you have a chance, thanks!

@nuthalapativarun

Copy link
Copy Markdown
Contributor Author

Checking in on this PR since it's been about 6 days quiet. It's mergeable on my end, happy to address any review comments.

@nuthalapativarun

Copy link
Copy Markdown
Contributor Author

Hi team — following up on PR #1853 (unit test coverage sweep). CI green, been open ~4 weeks. Happy to address any feedback. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant