feat(readme): advisory MiniMax README-drift reviewer (Layer 2, PR suggestions) — keeps Layer 1 mechanical#178
Open
amirkhakshour wants to merge 1 commit into
Open
feat(readme): advisory MiniMax README-drift reviewer (Layer 2, PR suggestions) — keeps Layer 1 mechanical#178amirkhakshour wants to merge 1 commit into
amirkhakshour wants to merge 1 commit into
Conversation
Adds Layer 2 of the README-drift stack WITHOUT touching Layer 1: the mechanical `readme-claim-check` job in ci.yml stays the hard, exact, free gate. This new job is ADVISORY — it never fails the PR. - .github/workflows/readme-review.yml — on PRs touching README/lib/bin/db/ recipes/schemas, runs a MiniMax-M3 reviewer (key from the MINIMAX_API_KEY repo secret; self-skips if absent). permissions: contents:read, pull-requests:write; only runs on same-repo PRs. - scripts/readme-review-ci.sh — reuses the existing minimax gatekeeper (cheap $0.005 'is this worth a review' veto) and feeds readme-claim-check --json as EXACT ground truth so the LLM judges SEMANTIC drift, not counts. Posts each suggested_fix as a line-anchored PR `suggestion` block where the excerpt matches a README line, else one consolidated advisory comment. Fail-open at every stage; always exits 0. DRY + MOCK env knobs for local testing. Why advisory not a gate: LLM verdicts are non-deterministic, so a hard LLM gate flakes red; advisory suggestions turn the model's real strength (drafting the fix) into value without the flaky-gate downside. Mechanical exactness stays in Layer 1. Deliberately did NOT LLM-ify readme-claim-check.sh — an LLM is worse at exact counting than `git ls-files | wc -l`, and would add cost/latency/nondeterminism to the one check that is currently free and always right. Requires operator action: add repo secret MINIMAX_API_KEY. Tested locally via MOCK+DRY (line-anchored suggestion + summary fallback + no-key self-skip).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds Layer 2 of the README-drift stack — a MiniMax-M3 reviewer that posts non-blocking PR suggestions for semantic claim drift. It does not replace the mechanical checker.
Deliberate non-change: I did not convert
scripts/readme-claim-check.sh(Layer 1) into an LLM. It's the deterministic, exact, free, sub-second gate; an LLM counts worse thangit ls-files | wc -land would add cost + latency + a flaky red gate. Layer 1 stays the hard gate inci.yml.How it works
readme-review.ymlruns on PRs touchingREADME.md/lib/bin/db/migrations/recipes/schemas→readme-drift-gatekeeper.sh, MiniMax, ~$0.005) — vetoes trivial diffs.readme-claim-check.sh --jsonis fed to the reviewer so it trusts exact counts and only judges prose/capability/comparison/citation drift.{drifted_claims:[{excerpt, reason, suggested_fix}]}.suggestionblock (one-click accept) where the excerpt matches a README line, else one consolidated advisory comment.Advisory by design (the whole point of "suggest as PR review"): LLM verdicts are non-deterministic, so a hard LLM gate flakes; advisory suggestions capture the model's strength (drafting the fix) with none of the flaky-gate downside. The job always exits 0 — it never fails a PR.
Safety / least privilege
permissions: contents:read, pull-requests:write; same-repo PRs only.MINIMAX_API_KEY/ noclaude/ nojq/ LLM unreachable / unparseable output → self-skips, exit 0.MO_REVIEW_DRY=1+MO_REVIEW_MOCK=<json>for local testing (used below).Tested locally
gh api .../comments path=README.md line=Nwith asuggestionblockgh pr commentfallbacksource cl_*/&&…||logpatterns as the existingreadme-drift-*.sh); CI shellcheck blocks only atseverity=error.Add a repo secret
MINIMAX_API_KEY(Settings → Secrets → Actions). Until then the job self-skips cleanly.