Skip to content

test(probes,detectors): add unit tests for dan, promptinject, web_injection#1863

Open
nuthalapativarun wants to merge 1 commit into
NVIDIA:mainfrom
nuthalapativarun:test/uncovered-modules-batch2
Open

test(probes,detectors): add unit tests for dan, promptinject, web_injection#1863
nuthalapativarun wants to merge 1 commit into
NVIDIA:mainfrom
nuthalapativarun:test/uncovered-modules-batch2

Conversation

@nuthalapativarun

Copy link
Copy Markdown
Contributor

What does this PR do

Adds unit test coverage for three modules that currently have no tests:

  • detectors.dan.* (DAN, AntiDAN, DANJailbreak, STAN, DUDE, DevMode, MarkdownLink) - verifies
    the DANDetectorMeta-generated StringDetector subclasses are wired with the correct
    substrings, hit on their marker strings, and pass on clean output.
  • probes.promptinject.Hijack* / Hijack*Full - verifies prompts are generated via the
    PromptInject framework, primary_detector is wired to promptinject.AttackRogueString,
    capped variants respect soft_probe_prompt_cap, full variants have at least as many prompts
    as capped, and _attempt_prestore_hook sets triggers/settings notes correctly.
  • probes.web_injection.* (MarkdownImageExfil, ColabAIDataLeakage, StringAssemblyDataExfil,
    PlaygroundMarkdownExfil, MarkdownURI*ExfilExtended, TaskXSS, MarkdownXSS) - verifies each
    probe loads, generates non-empty prompts, wires the correct primary_detector, and checks
    module-specific behaviour (markdown image markers, prestore hook notes, domain mutation).

This continues the uncovered-module test sweep from #1853.

AI assistance disclosure

Portions of this PR (test scaffolding and initial drafts) were written with AI assistance
(Claude). I reviewed every changed line, ran the new and existing test suites locally, and
confirm the assertions reflect the actual contract/behaviour of the modules under test.

Verification

  • Supporting configuration such as generator configuration file - n/a, test-only change
  • garak -t <target_type> -n <model_name> - n/a, test-only change
  • Run the tests and ensure they pass: python -m pytest tests/probes tests/detectors -k "dan or promptinject or web_injection" - 376 passed
  • Verify the thing does what it should - new tests assert prompt counts/caps, substring
    wiring, primary_detector linkage, and detect() scoring for hit/no-hit cases
  • Verify the thing does not do what it should not - tests include negative cases (clean
    output should not trigger DAN detectors)
  • Document the thing and how it works - test names and assertions describe expected
    behaviour

…ection

Signed-off-by: Varun Nuthalapati <nuthalapativarun@gmail.com>
@nuthalapativarun

Copy link
Copy Markdown
Contributor Author

Bumping this — tests are still passing and it's been about 10 days without a look. Let me know if anything needs adjusting.

@nuthalapativarun

Copy link
Copy Markdown
Contributor Author

Hi team — following up on PR #1863 (unit tests for dan, promptinject, web_injection). CI green, no reviews yet. Happy to make adjustments if needed. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant