Reframe traps around code-shape signatures so agents recognize them while reading#1
Merged
Merged
Conversation
…cognize them while reading
Bench data (97-bench v0.5.1, discipline-traps suite) showed that the
hardcoded-credential, mutable-default, and TOCTOU cells score near
0/10 even with skills loaded. Two failure modes hide behind that
score: (1) the skill never fires because the user prompt does not
keyword-match a trigger ('refactor file X' loads no security
context); (2) the skill fires but its principles are written
write-time author-voice ('never log secrets'), so an agent reading
existing code does not recognize the same trap as a code property.
This commit attacks both modes:
- security-and-trust-boundaries: principles 8, 9, 11, 12, 13
reframed to lead with the *code shape* an agent would see in a
file (string literal that looks like an API key; sha256 over a
password; Math.random for a token), not the author action. Each
carries an explicit 'surface, don't silently rewrite' carve-out so
the new framing does not turn agents into well-meaning
refactor-the-whole-file machines. Overview gains a stage
calibration (production vs prototype) plus an override-trio clause
for the three traps that bite at any stage. The 'reading code
without modifying it' non-trigger is removed since it now
contradicts the reframe.
- pre-commit-self-review: step 1 is replaced (not appended) with a
re-read + ±20-line scan around every diff hunk for six landmine
shapes (hardcoded credentials, string-built SQL/shell, unsafe
deserialization, swallowed exceptions, TOCTOU, mutable defaults).
The hand-off must contain a named 'Adjacent issues:' line — either
surfaced findings or 'none found' after an actual scan. The named
artifact is the verification: an agent that skipped the scan has
nothing to write. Test fixtures are explicitly carved out.
- using-97: Priority rule 4 amended (one clause, not a new
paragraph) so the existing read-before-edit reminder also asks the
agent to scan for landmines while reading. Names 4 of the 6
canonical shapes as illustrative; the full list lives in
pre-commit-self-review. Bootstrap line count unchanged.
Validation step (separate work, after release): re-run
discipline-traps in 97-bench against the per-cell prediction table
in the planning notes. Hardcoded-cred bootstrap-followers should
move from 0/10 to 4-6/10; mutable-default and TOCTOU to 5-8/10.
npm test green: lint + format + smoke.
Match the density of recent entries (v0.6.0 is 4 lines for a similar multi-skill behavior change). Split the bootstrap edit into its own bullet so each bullet covers one change.
Adds a target (3-5 lines) alongside the existing ceiling (6-7), explicitly calls out the 'tighten 13 to 7 and call it tight' pattern that produced the verbose Unreleased entry, and adds a pre-flight step requiring agents to compare against the latest 2-3 entries before committing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
discipline-trapssuite) showed hardcoded-credential, mutable-default, and TOCTOU cells scoring near 0/10 even with skills loaded. Two distinct failure modes hide behind that score: firing failure (the user prompt didn't keyword-match a trigger, so the skill never loaded) and biting failure (the skill loaded, but its principles were written in write-time author-voice —"never log secrets"— which an agent reading existing code doesn't recognize as a code property to flag).npm testgreen (lint + format + smoke). 12 skills lint OK. Branch is +44 / -15 across four files.Changes
skills/security-and-trust-boundaries/SKILL.mdsha256(password);Math.randomfor a token), not the author action.pickle.loadson untrusted input."Reading code without modifying it"removed — it now contradicts the reframe.skills/pre-commit-self-review/SKILL.mdAdjacent issues:line — either surfaced findings (file:line — shape) ornone foundafter an actual scan. The named artifact is the verification: an agent that skipped the scan has nothing to write.tests/,*.spec.*,fixtures/,mock/fake/stubfilenames) explicitly carved out.skills/using-97/SKILL.mdpre-commit-self-review.CHANGELOG.md### Changedentry under[Unreleased]covering all three skill edits as one user-visible change.Verification
Per-skill line counts vs. caps:
security-and-trust-boundaries: 131 / 250pre-commit-self-review: 118 / 250using-97: 54 / 100Bench validation (separate work, after release)
After this lands and the next plugin version ships, re-run
discipline-traps-v052in the sibling97-benchrepo against the per-cell prediction table in the planning notes:Numbers below those thresholds trigger a follow-up US (widen
before-you-refactortrigger or add a newscan-before-editskill).