fix(redaction): make sensitive-key, JWT, and private-key matching linear-time

[rng1995]

Problem

redact_sensitive_text() runs three assignment regexes (quoted, colon, equals) over skill and plugin files, hook scripts and LLM output. Each embeds \b plus _SENSITIVE_KEY_PATTERN, [a-z0-9_.-]*(?:api[_-]?key|secret|…|token|…)[a-z0-9_.-]*, from src/skillevaluator/utils/redaction.py. Trouble starts when a long run of key characters ([a-z0-9_.-]) is never followed by a complete assignment:

• Quadratic: the engine retries the key at every word boundary inside the run, and a-a-a-… has one at every offset. Each retry rescans to the end of the run.
• Cubic: for each start it also retries every way of splitting the run around a sensitive word (token-token-…), rescanning the rest of the run each time.
• Quoted form: it repeats its lazy [^\r\n]*?(?P=quote) scan to the end of the line on every one of those retries.

Review found that the JWT and private-key (PEM) regexes, which run on the same text, were quadratic too. Neither is new; both behave the same on main:

• JWT: \beyJ matches after every -, so in "eyJ-" * n each start scans to the end of the run looking for ..
• PEM label: a label word, [A-Z0-9][A-Z0-9-]*, can contain -----. On "-----BEGIN A" * n, each header's label runs on across every later header.
• PEM label pairs: (?:word )*PRIVATE KEY retries every later PRIVATE KEY pair and rescans the words after it ("-----BEGIN " + "PRIVATE KEY " * n).
• Unterminated PEM: the (?![\s\S]*-----END ) lookahead rescans the rest of the text from every header, so many headers before a single -----END near the end is quadratic.

I found the assignment problem while reviewing #171, which works around it in one caller by truncating the input first (_bounded in plugin_component_risk.py). Every other caller of redact_sensitive_text() on main is still exposed.

Timings

One redact_sensitive_text() call, Python 3.13.14, Apple Silicon. Each measurement ran in a fresh process with a 60 s cap (20 s for the JWT and PEM rows).

Input	Size	Before	After
"a-" * n	8 KB	6.5 s	3 ms
	16 KB	27.0 s	5 ms
	32 KB	> 60 s	11 ms
"token" * n	16 KB	1.8 s	2 ms
	32 KB	7.3 s	4 ms
"token-" * n	2 KB	5.4 s	< 1 ms
	4 KB	42.9 s	1 ms
	8 KB	> 60 s (each regex alone > 60 s)	1 ms
"token-" * n + "token='" + "x" * m (unterminated quote)	4 KB	8.9 s	1 ms
	8 KB	> 60 s (quoted regex alone 43.4 s)	2 ms
Any of the above	1 MB	n/a	0.12–0.35 s

The before time is split roughly evenly across the three regexes. For example, "a-" * 8000 (16 KB) takes 8.7 s, 8.9 s and 8.7 s.

JWT and PEM inputs. "Before" is this PR's first commit (53eeb81), whose JWT and PEM regexes are the same as main's.

Input	Before, 64 KB	Before, 256 KB	After, 256 KB	After, 1 MB
"eyJ-" * n	0.80 s	12.4 s	0.09 s	0.34 s
"-----BEGIN A" * n	1.9 s	> 20 s	0.10 s	0.39 s
"-----BEGIN PRIVATE KEY-----" * n	0.97 s	15.6 s	< 0.01 s	0.01 s
"-----BEGIN " + "PRIVATE KEY " * n	9.8 s	> 20 s	0.11 s	0.43 s
"-----BEGIN PRIVATE KEY-----\n" * n + "-----END x"	0.29 s	4.2 s	0.10 s	0.37 s
"-----BEGIN PRIVATE KEY X-" * n	> 20 s at 16 KB	—	0.02 s	0.06 s

Fix

The code change is confined to redaction.py. The three assignment regexes keep their names, and _redact_sensitive_assignment keeps its contract.

• Key starts at the run start. A key now only starts matching at the first character of a key-character run ((?<![a-z0-9_.-])). The old key always began at the run's first word boundary. The characters before that boundary go into a new lead group, and the replacement writes them back unchanged. In --api-key=… that is the --. When a run is glued to a non-ASCII word character, as in étoken.secret=…, it is the leading [a-z0-9_] part.
• No re-splitting. An atomic group plus a possessive quantifier, (?>[a-z0-9_.-]*?WORD)[a-z0-9_.-]*+, finds the first sensitive word and consumes the rest of the run without ever re-splitting it.
• Quoted values. The lazy value scan plus back-reference, (?P<quote>["'])(?P<value>[^\r\n]*?)(?P=quote), becomes explicit per-quote alternatives: "[^"\r\n]*+"|'[^'\r\n]*+'. Nothing used the quote and value groups.
• Whitespace. \s* is possessive wherever the next token can't match whitespace. The colon form keeps a backtracking \s* after :, because token: , has always matched its last blank as the value.
• JWT. Same idea as the key: a match starts only at the beginning of a run of JWT characters ((?<![A-Za-z0-9_-])). The part of the run before its first \beyJ goes into a lead group that the replacement writes back. So x-eyJ… is still redacted and éeyJ… is still left alone, which a plain (?<![A-Za-z0-9_-])eyJ would change. Later starts in the same run are never tried, because they reach the same . with a shorter first segment and could only fail where the first start failed.
• PEM labels. A label word is now [A-Z0-9]++(?:-[A-Z0-9]++)*+: letters and digits joined by single -, the RFC 7468 label shape. So a label can no longer run across -----. The label also commits, atomically, to its first PRIVATE KEY pair followed by a space or -----. A later pair can only reach the same label end.
• Unterminated PEM. The third PEM rule is now _redact_unterminated_private_key(). It redacts from the first private-key header that ends after the last -----END in the text, which is exactly where the old lookahead succeeded, without rescanning the text from every header.

requires-python is >=3.12, so atomic groups and possessive quantifiers (3.11+) are available.

Behaviour

Every assignment match yields the same key, sep and end as before, and the same start once lead is skipped, so the re.sub output is byte-for-byte identical. I checked this with a differential run against the previous patterns on two input sets:

• 840k random inputs, with about 520k matches;
• all 813k sequences of up to 5 units from a 15-unit alphabet: keywords, -, ., _, é, ſ, quotes, :, =, blanks, newlines and ,.

For each input, each regex returned the same match list and sub output, and redact_sensitive_text() returned the same string. I also checked every Unicode code point to confirm the case-insensitive key class splits exactly into word characters ([a-z0-9_]) and non-word characters ([.-]). Under re.I that class also contains İ ı ſ K.

For the JWT and PEM rewrite, a second differential run compared the new code with 53eeb81 and main on 11.6M inputs:

• 2M random JWT/PEM token soups and mutated blocks;

• 1M realistic texts with RFC 7468 labels only;

• all 5.4M sequences of up to 7 units from a 9-unit JWT alphabet;

• all 3.3M sequences of up to 6 units from a 12-unit PEM alphabet.

• JWT: same output as before on every input.

• PEM and redact_sensitive_text(): same output, at every stage, as a reference that differs from 53eeb81 only in the label word grammar. The atomic label and the new unterminated-key function change nothing else.

• Where the output changes: only where the old code read a label that the new word grammar rejects. This is the one intended behaviour change. A header no longer counts as a private-key header when:

  • a label word has -- or a trailing - (-----BEGIN X- PRIVATE KEY-----); or
  • the label only reaches PRIVATE KEY by running into the next delimiter on the same line (-----BEGIN CERTIFICATE----------END RSA PRIVATE KEY-----).

  RFC 7468 labels do neither. All standard labels are handled as before: bare, RSA, EC, DSA, OPENSSH, ENCRYPTED, PGP … BLOCK, and dashed words such as X-Y.

Tests

New file tests/test_redaction_assignments.py:

• Representative outputs, 15 cases. Covers quoted values in both quote styles, mixed quotes (' inside "…" and the reverse), colon and equals forms, CLI flags (--api-key=), leading dots, a key glued to a non-ASCII word, and token-count keys that must stay visible. Two multiline cases cover a colon value on the next line and a mixed block. The expected strings match what the previous implementation produced.
• JWT positions, 12 cases. Covers start of text, after a space, a quote, = or Bearer, glued to - or to a dashed run, and a run that starts with eyJ-. JWTs glued to _, an ASCII letter or é stay unredacted.
• Private-key labels, 8 cases. Covers dashed label words, words before and after PRIVATE KEY, a later PRIVATE KEY pair, a single-line block, a block glued to an earlier header, and truncated keys after an earlier -----END , running into -----END, or with overlapping headers. All outputs match the previous implementation.
• Timing bound on 13 crafted worst-case inputs. There are 7 assignment inputs and 6 JWT/PEM inputs of 60–200 KB. The bound is generous (< 1 s), and each test also asserts the expected output. With the previous implementation each input takes seconds (the JWT/PEM ones 2.4 s to more than 15 s), so these tests fail on main. They now take 0–80 ms.
• Long key followed by a value. One test checks that a value after a 10k-character key run is still redacted, within the same bound.

The existing redaction tests pass unchanged.

The JWT and PEM fixtures are assembled from split literals, like the existing redaction tests, so Gitleaks doesn't flag them. The PR check scans every commit in the branch, so .gitleaks.toml allowlists the literal fixtures that 97a0616 committed, for that commit and file only.

• uv run ruff check .: all checks passed.
• uv run pytest -q -n auto -p no:cacheprovider: 9155 passed, 18 skipped, 0 failed.

Relation to #171

#171's _bounded truncation in plugin_component_risk.py should stay. An earlier version of this description said it was no longer needed for performance. That was wrong, because the JWT and PEM regexes were still quadratic. With this PR, every quadratic shape found in review stays under 0.5 s at 1 MB. Still, redact_sensitive_text() makes 14 passes over its input, so capping report text is still worthwhile.

🤖 Generated with Claude Code

[rng1995]

Reviewed at 53eeb81. The rewrite checks out.

• Same output as main. My own differential fuzz found no differences between main and this branch, for each of the three regexes' finditer/sub and for redact_sensitive_text(). It used a different generator from yours:
  • 1M grammar-built and mutated config, CLI, env, JSON and YAML inputs, with about 1.19M matches. They cover \r\n, \r, tabs, Unicode word characters, escaped and unbalanced quotes, --api-key=, and ., -, _ or digit leads.
  • An exhaustive run over all 8.3M sequences of up to 5 units from a 24-unit alphabet.
• Python versions. The fuzz and the new tests also pass on 3.12.0, 3.12.11, 3.12.13, 3.13.0, 3.13.5 and 3.13.14.
• Linear time. All 30 adversarial shapes I tried against the new regexes stay linear up to 1 MB (≤ 0.7 s). That includes the colon form with long newline runs after :.
• Tests. On main, each of the 8 new timing tests takes 10–20 s and fails. On this branch each takes ≤ 10 ms against the 1 s bound, so they're unlikely to flake on slow runners.

One issue inline: redact_sensitive_text() as a whole is still quadratic through the unchanged PEM and JWT regexes.