feat(verification): add the subject_access_review verifier - #83
Conversation
… broken pass with the judge Two defects that together let a run whose verification never executed report a perfect score. First, verifiers carried no cluster identity. BaseVerifier.kubeconfig defaulted to None and nothing ever set it, so every check inherited the ambient kubeconfig's current-context, which is a mutable global that any run can rewrite. A run was observed getting an authoritative "namespace not found" for a namespace the agent had just written to, because the check queried a different cluster. ClusterInfo now carries the resolved context (kind-<cluster> for kind, the existing gke_<project>_<location>_<cluster> for GCP) and the harness pins both kubeconfig and context onto every verifier leaf, recursing through compound nodes, before either the chaos scenario or the post-run pass runs. kubectl gains a --context passthrough. Second, an all-errored pass was scored by the LLM judge. rollup drops error entries from both numerator and denominator, so when every objective errors the correctness signal is never emitted, and the composite's preference chain fell through to the judged OutcomeValidity. Two runs reported c=1.000 that way with all 11 entries errored. The composite is now withheld when deterministic verification was declared but produced no correctness, leaving outcomeScore null and VerificationCoverage as the diagnostic. The judged fallback still applies unchanged to tasks that declare no verification_spec, which is what it was for. Signed-off-by: Eric Hole <ehole@onixnet.com>
Adds kubectl auth can-i-backed verifiers for the subject_access_review and subject_access_review_sweep check types the factory's T-023 and T-029 blueprints demand. The point verifier checks one (subject, verb, resource) against an expected allowed/denied outcome; the sweep verifier checks a whole matrix of pairs and requires an exact match in both directions, so an unexpected allow fails a negative sweep just as a missing allow fails a positive one. A real command error (bad flags, unreachable API) reports as verification status error rather than a pass/fail outcome, since can-i itself exits non-zero on an ordinary denied check. Signed-off-by: Eric Hole <ehole@onixnet.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: geojaz The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Warning Review limit reached
Next review available in: 55 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (17)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
kubernetes-prow
Bot
added
the
cncf-cla: yes
|
Hi @geojaz. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kubernetes-prow
Bot
added
needs-ok-to-test
janetkuo
added
ok-to-test
2 participants
Stacked on #79, which adds the
contextfield this verifier reads. The first commit shown here belongs to that PR and will drop out of this diff once it merges. Review only the last commit.Adds a
subject_access_reviewverifier so a task can assert what a subject is and is not allowed to do, rather than inferring authorization indirectly from RBAC objects.The verifier supports two modes:
Both modes go through the cluster's own SubjectAccessReview API, so the answer reflects what the API server will actually do, including bindings the task did not create.
This is a peer to the
http_probe,external_http_probe, andgit_repo_syncverifiers, and follows the same registration and result-shape conventions.Unit tests cover both modes plus the failure and error paths.