Skip to content

Reframe traps around code-shape signatures so agents recognize them while reading#1

Merged
oribarilan merged 4 commits into
mainfrom
us-bench-driven-skill-improvements
May 8, 2026
Merged

Reframe traps around code-shape signatures so agents recognize them while reading#1
oribarilan merged 4 commits into
mainfrom
us-bench-driven-skill-improvements

Conversation

@oribarilan

Copy link
Copy Markdown
Owner

Summary

  • Bench data (97-bench v0.5.1, discipline-traps suite) showed hardcoded-credential, mutable-default, and TOCTOU cells scoring near 0/10 even with skills loaded. Two distinct failure modes hide behind that score: firing failure (the user prompt didn't keyword-match a trigger, so the skill never loaded) and biting failure (the skill loaded, but its principles were written in write-time author-voice — "never log secrets" — which an agent reading existing code doesn't recognize as a code property to flag).
  • Three coordinated edits attack both modes without adding a new skill, a new trigger, or a version bump.
  • npm test green (lint + format + smoke). 12 skills lint OK. Branch is +44 / -15 across four files.

Changes

skills/security-and-trust-boundaries/SKILL.md

  • Principles 8, 9, 11, 12, 13 reframed to lead with the code shape an agent would see in a file (a string literal that looks like an API key; sha256(password); Math.random for a token), not the author action.
  • Each reframed principle carries an explicit "surface, don't silently rewrite" carve-out so the new framing doesn't turn agents into well-meaning refactor-the-whole-file machines.
  • Overview gains a stage calibration (production vs prototype) plus an override-trio clause for the three traps that bite at any stage: committed credentials, string-built SQL/shell, pickle.loads on untrusted input.
  • Non-trigger "Reading code without modifying it" removed — it now contradicts the reframe.
  • Matching Red Flags rows show both author-voice and code-shape framing.

skills/pre-commit-self-review/SKILL.md

  • Step 1 replaced (not appended) with a re-read + ±20-line scan around every diff hunk for six landmine shapes:
    1. Hardcoded credentials
    2. String-built SQL / LDAP / shell
    3. Unsafe deserialization on untrusted input
    4. Swallowed exceptions
    5. TOCTOU patterns
    6. Mutable default arguments
  • Hand-off must contain a named Adjacent issues: line — either surfaced findings (file:line — shape) or none found after an actual scan. The named artifact is the verification: an agent that skipped the scan has nothing to write.
  • Test fixtures (tests/, *.spec.*, fixtures/, mock/fake/stub filenames) explicitly carved out.

skills/using-97/SKILL.md

  • Priority rule 4 amended (one clause, not a new paragraph) so the existing read-before-edit reminder also asks the agent to scan for landmines while reading.
  • Names 4 of the 6 canonical shapes as an illustrative subset; the full list lives in pre-commit-self-review.
  • Bootstrap line count unchanged (54).

CHANGELOG.md

  • One ### Changed entry under [Unreleased] covering all three skill edits as one user-visible change.

Verification

npm test
> lint-skills: 12 skill(s) OK
> All matched files use Prettier code style!
> smoke-load OK

Per-skill line counts vs. caps:

  • security-and-trust-boundaries: 131 / 250
  • pre-commit-self-review: 118 / 250
  • using-97: 54 / 100

Bench validation (separate work, after release)

After this lands and the next plugin version ships, re-run discipline-traps-v052 in the sibling 97-bench repo against the per-cell prediction table in the planning notes:

  • Hardcoded-cred bootstrap-followers (haiku-4.5): 0/10 → 4–6/10
  • Hardcoded-cred bootstrap-ignoring (gpt-5-mini): 0/10 → 1–2/10
  • Mutable-default and TOCTOU: 0/10 → 5–8/10

Numbers below those thresholds trigger a follow-up US (widen before-you-refactor trigger or add a new scan-before-edit skill).

oribarilan added 4 commits May 8, 2026 14:22
…cognize them while reading

Bench data (97-bench v0.5.1, discipline-traps suite) showed that the
hardcoded-credential, mutable-default, and TOCTOU cells score near
0/10 even with skills loaded. Two failure modes hide behind that
score: (1) the skill never fires because the user prompt does not
keyword-match a trigger ('refactor file X' loads no security
context); (2) the skill fires but its principles are written
write-time author-voice ('never log secrets'), so an agent reading
existing code does not recognize the same trap as a code property.

This commit attacks both modes:

- security-and-trust-boundaries: principles 8, 9, 11, 12, 13
  reframed to lead with the *code shape* an agent would see in a
  file (string literal that looks like an API key; sha256 over a
  password; Math.random for a token), not the author action. Each
  carries an explicit 'surface, don't silently rewrite' carve-out so
  the new framing does not turn agents into well-meaning
  refactor-the-whole-file machines. Overview gains a stage
  calibration (production vs prototype) plus an override-trio clause
  for the three traps that bite at any stage. The 'reading code
  without modifying it' non-trigger is removed since it now
  contradicts the reframe.

- pre-commit-self-review: step 1 is replaced (not appended) with a
  re-read + ±20-line scan around every diff hunk for six landmine
  shapes (hardcoded credentials, string-built SQL/shell, unsafe
  deserialization, swallowed exceptions, TOCTOU, mutable defaults).
  The hand-off must contain a named 'Adjacent issues:' line — either
  surfaced findings or 'none found' after an actual scan. The named
  artifact is the verification: an agent that skipped the scan has
  nothing to write. Test fixtures are explicitly carved out.

- using-97: Priority rule 4 amended (one clause, not a new
  paragraph) so the existing read-before-edit reminder also asks the
  agent to scan for landmines while reading. Names 4 of the 6
  canonical shapes as illustrative; the full list lives in
  pre-commit-self-review. Bootstrap line count unchanged.

Validation step (separate work, after release): re-run
discipline-traps in 97-bench against the per-cell prediction table
in the planning notes. Hardcoded-cred bootstrap-followers should
move from 0/10 to 4-6/10; mutable-default and TOCTOU to 5-8/10.

npm test green: lint + format + smoke.
Match the density of recent entries (v0.6.0 is 4 lines for a
similar multi-skill behavior change). Split the bootstrap edit
into its own bullet so each bullet covers one change.
Adds a target (3-5 lines) alongside the existing ceiling (6-7),
explicitly calls out the 'tighten 13 to 7 and call it tight'
pattern that produced the verbose Unreleased entry, and adds a
pre-flight step requiring agents to compare against the latest
2-3 entries before committing.
@oribarilan oribarilan merged commit 94dd5ae into main May 8, 2026
18 checks passed
@oribarilan oribarilan deleted the us-bench-driven-skill-improvements branch May 8, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant