feat(#192): add evidence cap to retro agent's before-proposing check

[fullsend-ai-coder]

Summary

• Add an evidence cap to the retro-analysis skill's "Before proposing" section that prevents filing redundant evidence issues for parent issues with >= 5 open data points
• When the cap is hit, the agent skips the evidence proposal and references the parent issue and current PR in the summary comment instead
• Fails open: if the count query fails, the proposal proceeds normally

This reduces tracking noise (64 open evidence issues, top 5 parents owning 32 of them) without losing signal — the PR is still mentioned as a data point in the summary.

Supersedes #211 (rebased on current main to resolve CI failures from stale base).

Closes #192

Test plan

• Verify the new "Evidence cap for tracked patterns" subsection is present in internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md
• Verify the section is positioned as a subsection under "Before proposing: check for existing issues"
• Verify the gh api query template uses the correct search syntax for title-matching evidence issues
• Confirm make lint passes (trailing whitespace, end-of-file, markdown links)
• Over next 30 retro runs: no new evidence issue for parents with >= 5 open evidence issues

Generated with Claude Code

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 9:16 AM UTC · Completed 9:28 AM UTC
Commit: a2afa8b · View workflow run →

[fullsend-ai-review]

Review

✅ Approve

Summary

This PR adds an evidence cap to the retro-analysis skill's "Before proposing" section, preventing the retro agent from filing redundant evidence issues when a parent issue already has ≥ 5 open data points. The change is well-scoped — a single skill definition file, a single concern — and directly addresses #192.

Analysis

Correctness. The logic flow is sound: classify as evidence-type → query count via gh api search → skip if ≥ 5 → note in summary. The fail-open behavior (proceed if the count query errors) is the right default for agent-facing guidance — it avoids silently suppressing legitimate proposals due to transient API failures. The gh api search query correctly uses the GitHub Search API with in:title and URL-encoded # (%23), and the --jq '.total_count' extraction is correct. The new subsection is structurally positioned as ### under the existing ## Before proposing: check for existing issues section, after the duplicate-check guidance it logically depends on.

Security. No code execution paths are changed. The skill instructs the agent to use gh api for a read-only search query, introducing no new attack surface. Template placeholders (<target_repo>, <parent_number>) are filled by the agent from its own runtime context — not from untrusted user input in a way that could lead to injection.

Intent & coherence. The change matches the authorized work in #192 exactly. The scope is appropriately narrow. The approach — skill-level prompt guidance rather than code enforcement — is consistent with how other guardrails in this same file are implemented (e.g., the duplicate-check section immediately above). The definition of "evidence-type proposal" at the top of the subsection provides clear classification criteria for the agent.

Style & conventions. The writing style, markdown structure, and heading levels match the surrounding content. The new subsection follows the same pattern as the duplicate-check section: explain the concept → provide a subagent prompt → show the query template → define skip/include behavior.

Findings

#	Severity	Category	Description
1	low	conventions	PR title uses feat prefix. Per COMMITS.md, "adjusting prompts, or tuning agent behavior that users don't directly control" should use refactor or fix. Since this corrects a noisy-filing defect (#192 describes 64 open evidence issues as tracking overhead), fix(#192) is more appropriate. The second commit already correctly uses fix. If this repo uses squash merges, the PR title becomes the commit message on main — consider updating it before merge.

Previous run

Review — comment

Reviewed: 0403fc90d7495801f84a9100f4609167a929f5cf
Linked issue: #192
Verdict: Comment — one medium finding worth noting; no blocking issues.

Summary

This PR adds an evidence cap subsection to the retro-analysis skill's "Before proposing" section. It addresses the noise problem documented in #192 (64 open evidence issues, top 5 parents owning 32 of them) by instructing the retro agent to skip evidence proposals when a parent issue already has ≥ 5 open data-point issues.

The change is well-scoped, correctly placed, and architecturally consistent. The fail-open behavior is appropriate (this is a quality-of-life deduplication feature, not a security boundary). No security, intent, style, or documentation currency concerns were found.

Findings

1. Undefined "evidence-type proposals" concept — medium / correctness

File: internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md (new lines ~128–130)

The new section references "evidence-type proposals" and the Evidence for #N: ... title convention, but neither the retro-analysis skill nor the retro agent definition (agents/retro.md) formally defines when the agent should create evidence-type proposals or what the canonical title format is. The section provides a working definition via the parenthetical explanation ("the proposal would be filed as 'Evidence for #N: ...' referencing an existing open issue"), but the categorization between "evidence proposals" and "genuinely new patterns" relies on the agent inferring a distinction that is never explicitly taught.

Risk: An LLM consuming this skill may misclassify a proposal that is similar to but not exactly an "evidence" issue, either over-suppressing new patterns or under-suppressing redundant evidence.

Remediation (non-blocking): Consider adding a one-sentence definition of evidence-type proposals earlier in the "Before proposing" section or as a preamble to the new subsection, e.g.: "An evidence-type proposal is one where the agent observes a new instance of a pattern already tracked by an existing open issue, and would title its proposal 'Evidence for #N: ' rather than proposing a new improvement."

2. Search query tightly coupled to exact title format — low / robustness

File: internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md (new line ~145)

The gh api search query includes a trailing colon in the phrase: "Evidence+for+%23<parent_number>:". If any evidence issues were filed with a variant format (e.g., without the colon), they would not be counted. This is mitigated by the existing convention consistently using the colon format per #192's data, and by the fail-open design.

Remediation (non-blocking): Consider dropping the trailing colon from the search phrase to match more broadly: "Evidence+for+%23<parent_number>". This would catch minor title variations without significantly increasing false positives.

Dimensions reviewed

Dimension	Model	Result
Correctness	opus	1 medium, 1 low
Security	opus	No findings
Intent & coherence	sonnet	No findings
Style & conventions	sonnet	No findings (info-level suppressed)
Documentation currency	sonnet	No findings
Cross-repo contracts	—	Skipped (no exported APIs modified)

Previous run (2)

Review

Findings

Low

• [API query correctness] internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md:147 — The prose says "title starts with" but the GitHub Search API's in:title qualifier matches substring containment, not prefix. Practically immaterial for a controlled naming convention, but the prose could be adjusted to say "contains" for accuracy.

• [undefined terminology] internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md:127 — The phrase "evidence-type proposals" is introduced without formal definition. The following paragraph provides a contextual definition via the parenthetical "(i.e., the proposal would be filed as 'Evidence for #N: ...')", which is sufficient for the LLM agent audience but could be made more explicit.

• [query correctness] internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md:147 — The %23 URL encoding for # should work correctly with gh api (which passes path strings as-is), but if it ever double-encoded, the fail-open policy would silently disable the cap.

• [fail-open risk] internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md:153 — The fail-open policy is reasonable for availability but creates a silent failure mode. Adding guidance to note query failures in the summary field would improve observability of whether the cap is functioning.

---

Labels: PR modifies a skill definition (retro-analysis SKILL.md), which is documentation-category content.

[guyoron1]

/fs-review

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 10:19 AM UTC · Completed 10:27 AM UTC
Commit: a2afa8b · View workflow run →

[guyoron1]

/fs-fix

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 11:12 AM UTC · Completed 11:22 AM UTC
Commit: a2afa8b · View workflow run →

[fullsend-ai-coder]

🔧 Fix agent — iteration 1 (human-triggered)

Addressed both review findings. (1) Added an explicit definition of evidence-type proposals at the start of the subsection. (2) Relaxed the search query by dropping the trailing colon for broader matching.

Fixed (2):

1. Undefined 'evidence-type proposals' concept (internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md): Added an explicit definition of 'evidence-type proposal' as a new paragraph at the start of the subsection, so the retro agent has clear classification criteria rather than inferring the distinction from a parenthetical
2. Search query tightly coupled to exact title format (internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md): Dropped trailing colon from the gh api search query to match evidence issues more broadly regardless of minor title variations. Also updated the subagent prompt to say 'contains' instead of 'starts with' to accurately describe the in:title search behavior

Tests: passed

_{Updated by fullsend fix agent}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 11:27 AM UTC · Completed 11:32 AM UTC
Commit: a2afa8b · View workflow run →