docs(EC-1999): document testing expectations for collection changes

[BohdanMar]

Summary

• Expand the "Add a new collection" entry in the Common Change Patterns section of AGENTS.md to document testing expectations
• Collections are metadata-only — no new tests needed for the collection itself
• make conventions-check validates dependency-collection superset constraints
• New rules added alongside a collection still need _test.rego coverage

Resolves EC-1999
Supersedes #1777

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 763c20fd-986d-4965-a2b0-e7dba2d4a076

📥 Commits

Reviewing files that changed from the base of the PR and between 57634e6 and 6ee1f77.

📒 Files selected for processing (1)

• AGENTS.md

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The collection creation guidance now specifies metadata-only definitions, removes collection-specific test requirements, identifies CLI and conventions-check validation, and retains test coverage for accompanying rules.

Changes

Collection guidance

Layer / File(s)	Summary
Metadata-only collection instructions AGENTS.md	The guidance now describes metadata-only collection definitions, CLI and conventions-check validation, and _test.rego coverage for accompanying rules.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 6ee1f

This is a localized documentation update with no actionable merge-blocking risk remaining after normal checks.

Suggested reviewers: cuipinghuo, joejstuart

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the documentation change and its testing-expectations focus.
Description check	✅ Passed	The description states what changed, why it matters, and links EC-1999, although it uses a Summary section instead of the template headings.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:14 AM UTC · Completed 8:28 AM UTC

Commit: 87c4a29 · View workflow run →

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
unit-tests	100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 21 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[fullsend-ai-review]

Review

Findings

Medium

• [protected-path] AGENTS.md — This PR modifies AGENTS.md, which is a protected path. The PR links to EC-1999 and provides a clear rationale for the change. Human approval is always required for protected-path changes, regardless of context.

Low

• [technical-accuracy] AGENTS.md:82 — The path pattern policy/*/collection/<name>/ is consistent with the Architecture section, which already uses policy/*/collection/. However, collections currently only exist under policy/release/collection/. The glob is aspirational but internally consistent within the document.

• [pattern-inconsistency] AGENTS.md:82 — The new entry uses a different structure from most neighboring entries (semicolons instead of em-dashes/parentheses). The existing section already has some format variation (the OCI blob entry on line 81 also deviates), making this a minor style nit.
  Remediation: Consider restructuring to use parenthetical notes instead of semicolons for consistency with surrounding entries.

---

Labels: PR is a documentation-only change to AGENTS.md

[fullsend-ai-review]

See the review comment for full details.

[cuipinghuo]

/lgtm
/approve

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 8:28 AM UTC · Completed 8:40 AM UTC

Commit: 87c4a29 · View workflow run →

[fullsend-ai-retro]

Retro: PR #1815 — docs(EC-1999): document testing expectations for collection changes

Verdict: The end-to-end pipeline worked. A retro agent identified a real documentation gap from PR #1766, filed issue #1776, and the system produced a technically correct fix via PR #1777. A human (BohdanMar) later created PR #1815 with minor editorial improvements and merged it, explicitly superseding the agent's PR. The review agent on PR #1815 performed well — correctly flagging the protected-path governance requirement, producing reasonable technical-accuracy and style findings, and using the challenger/adversarial verification pattern to appropriately downgrade findings that didn't hold up under scrutiny.

Timeline

1. Jul 15 09:24 — Retro agent files issue #1776 after analyzing PR Add redhat_security policy collection #1766.
2. Jul 15 09:25–09:34 — Triage and code agents process issue; PR #1777 created with correct documentation change.
3. Jul 21 13:53 — Human reviewer gives lukewarm approval and requests rebase. Fix agent resolves the merge conflict.
4. Jul 21–Aug 6 — Review raises a line-length style nit; /fs-fix is invoked but the PR stalls.
5. Aug 17 08:13 — Human creates PR #1815, superseding docs(#1776): add collection change pattern to AGENTS.md #1777 with improved glob pattern (policy/*/collection/<name>/) and cleaner sentence structure.
6. Aug 17 08:14–08:28 — Review agent reviews PR docs(EC-1999): document testing expectations for collection changes #1815 (run 32009343249). Three sub-agent calls fail due to sonnet alias resolving to unavailable Vertex model; all recover on retry.
7. Aug 17 19:06 — Human approves. Aug 18 08:27 — Merged.

Evidence for existing issues (no new proposals needed)

• Model availability retries — The sonnet alias resolved to claude-sonnet-4-5@20250929 (unavailable on Vertex), causing 3 sub-agent failures before successful retries. This wasted tokens and added ~5 min to the review. Evidence for agents#379 and fullsend#1771.
• Orphaned PR and issue — PR #1777 remains OPEN despite being superseded by merged PR docs(EC-1999): document testing expectations for collection changes #1815. Issue #1776 remains OPEN despite the work being complete. Evidence for fullsend#2292 and agents#586.
• Code agent editorial polish — The agent's PR docs(#1776): add collection change pattern to AGENTS.md #1777 kept the hardcoded path policy/release/collection/ while the Architecture section of the same file already used the glob policy/*/collection/. The human's version adopted the consistent glob pattern. Evidence for agents#458 (code agent should match peer-file formatting).

What went well

• The retro → triage → code pipeline correctly identified and addressed a real documentation gap.
• The review agent's challenger pass on PR docs(EC-1999): document testing expectations for collection changes #1815 worked as designed, correctly downgrading two findings that didn't hold up under adversarial scrutiny.
• The protected-path governance finding was appropriate — the review used comment verdict (not approve), correctly deferring to human judgment for AGENTS.md changes.
• The substantive content of both the agent's and human's versions was nearly identical, confirming the retro agent's original proposal was well-specified.