feat(detectors): add Unicode normalisation to StringDetector

[PhinehasNarh]

What this changes

Adds a normalize parameter to StringDetector (default None, fully backward-compatible).

When set, both the trigger strings and the model output are normalised before any matching occurs:

Value	Behaviour
None	No change — current behaviour
"NFKC"	unicodedata.normalize("NFKC", …) — collapses fullwidth code points, combining marks, and most homoglyphs to their ASCII equivalents
"NFKC+strip_format"	NFKC then strip Unicode category Cf characters (zero-width joiner, ZWSP, ZWNJ, BOM, etc.)

Why this matters

Without normalisation, a term in the trigger list that is also present in a model output but written with non-ASCII code points is not byte-equal to the listed ASCII substring and is missed:

case                      score   caught after NFKC?
PLAIN "hello" (control)   1.0     True
FULLWIDTH "ｈｅｌｌｏ"     0.0     True     ← false negative, fixed by normalize="NFKC"
ZWJ "h‍e‍l‍l‍o"             0.0     True     ← false negative, fixed by normalize="NFKC+strip_format"

This affects every StringDetector-based detector (unsafe_content.LDNOOBW, lmrc.Anthro, goodside.RileyIsnt, and ~30 others). garak already recognises this evasion class on the probe side (probes.badcharacters generates homoglyph/invisible payloads), so leaving detectors blind to it was an asymmetry, not an intentional scope decision.

Closes #1867.

Why this is not a duplicate

No existing open PR addresses StringDetector Unicode normalisation. PR #1880 and PR #1882 address a separate bug (unescaped regex metacharacters in word-boundary matching). The normalize parameter introduced here is independent of that fix.

Test commands run and results

python -m pytest tests/detectors/test_detectors_base.py -v --noconftest

17 passed in 0.21s

Six new tests cover:

• baseline: fullwidth homoglyph bypasses detection without normalize (confirms the vulnerability)
• normalize="NFKC" catches fullwidth evasion
• normalize="NFKC" does not break plain ASCII detection
• normalize="NFKC" alone does not strip ZWJ (documents the two-tier design)
• normalize="NFKC+strip_format" catches ZWJ-interrupted evasion
• invalid normalize value raises ValueError

AI assistance disclosure

This PR was written with AI assistance (Claude). I reviewed every changed line, ran the tests, and can defend the implementation end-to-end. The design follows the two-tier approach discussed in issue #1867.

[jmartin-tech]

Raising a ValueError in a base detector can cause the the run to terminate. Looking at the changes in this PR, calls to this _apply_normalize() method need to be wrapped in a try block and the value error should cause the detection to report the appropriate value either 0.0 for no hit or None to indicate that detection could not be preformed.

[PhinehasNarh]

Good catch - a ValueError in a base detector would have killed the whole run, which is obviously wrong.

Fixed in the latest push: still raises internally (the error itself is valid), but now wraps the call in a try/except ValueError, logs a warning, and appends None to signal that detection could not be performed for that output. The run continues cleanly.

Updated test_stringdetector_normalize_invalid_value to match: it now asserts results == [None] instead of expecting the exception to propagate. All 17 tests pass.

[PhinehasNarh]

Circled back on this @jmartin-tech - the earlier fix only wrapped the output-text call. There was a second _apply_normalize call on the trigger substrings that was still bare. In practice it only ran when normalize was already valid (the output call fails first and continues), so it couldn't actually terminate a run, but it read as unguarded and re-normalized every substring per output.

Latest push hoists the substring normalization out of the loops so it happens once, wrapped in the same try/except: bad normalize config logs a warning and returns None for all outputs. Kept the per-output guard too so the empty-substrings case is still covered. 17/17 detector tests pass, and I checked the invalid-normalize path directly against the real detector (returns all None, no raise).

[feiiiiii5]

Hi @PhinehasNarh — I opened a duplicate PR (#1938) for this issue before finding yours; closing mine in favor of this one.

While implementing my version I hit a few edge cases that might be worth adding tests for here, if you agree they're useful:

1. Cross-script confusables limitation: NFKC does not collapse Cyrillic 'а' (U+0430) to ASCII 'a'. Issue StringDetector substring matchers do no Unicode normalization — toxic output in homoglyph/fullwidth/zero-width form scores clean (false-negative) #1867 comments suggested NFKC handles homoglyphs broadly; a test pinning this limitation keeps the docstring honest.

2. Combining marks limitation: 'a' + U+0301 (combining acute) normalizes to precomposed 'á' (U+00E1), still not matching ASCII 'a'. Same as above — NFKC doesn't help here.

3. End-to-end with a real subclass: testing against the real unsafe_content.LDNOOBW detector (named in StringDetector substring matchers do no Unicode normalization — toxic output in homoglyph/fullwidth/zero-width form scores clean (false-negative) #1867) with a fullwidth variant of a word from its live wordlist. Catches regressions if the wordlist or detector wiring changes.

4. matchtype interactions: normalize + matchtype="word" and ="startswith". Normalization order relative to \b regex matters for the word case.

5. case_sensitive=False composition: all current tests use case_sensitive=True; verifying lower-casing + NFKC compose correctly is useful since both run before matching.

Happy to push these test additions to a branch against your PR if you'd like — or feel free to cherry-pick. Either way, the core fix here is solid.

[PhinehasNarh]

Thanks @feiiiiii5, and appreciate you closing #1938 in favor of this one. All five were worth adding, so I pushed them in c2663ac:

• the two limitation tests (cross-script Cyrillic confusables + combining marks) - good call on keeping the docstring honest, NFKC genuinely doesn't fold those to ASCII
• matchtype='word'/'startswith' composition with normalize
• case_sensitive=False + NFKC (fullwidth uppercase variant)
• the end-to-end against the real unsafe_content.LDNOOBW - I pick the wordlist entry dynamically so there's no hardcoded term, and set lang_spec='*' since it's exercising normalization rather than language routing

22/22 pass via pytest tests/detectors/test_detectors_base.py --noconftest. Thanks for the careful list - the limitation ones especially are nice to have pinned.