feat(verification): make the check timeout and total budget tunable by environment - #82
feat(verification): make the check timeout and total budget tunable by environment#82geojaz wants to merge 4 commits into
Conversation
Defaults are unchanged at 120s per entry and 600s for the pass as a whole. The total budget is the one that bites: it bounds the whole pass, so if early entries consume it the later ones are starved and report "evaluation did not complete before the deadline" without having meaningfully run, which reads as a failed check rather than an unmeasured one. BENCH_VERIFY_TIMEOUT_SEC and BENCH_VERIFY_TOTAL_BUDGET_SEC allow tuning that without a code change. Signed-off-by: Eric Hole <ehole@onixnet.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: geojaz The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Warning Review limit reached
Next review available in: 28 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
kubernetes-prow
Bot
added
the
cncf-cla: yes
|
Hi @geojaz. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kubernetes-prow
Bot
added
needs-ok-to-test
janetkuo
added
ok-to-test
…port BENCH_VERIFY_TIMEOUT_SEC and BENCH_VERIFY_TOTAL_BUDGET_SEC were parsed with a bare int(os.environ.get(...)), so a malformed value raised an unlabeled ValueError from deep inside module import, and a zero or negative value was accepted silently, producing confusing downstream behavior. Both now go through the existing get_int helper, plus a small _positive_int_env wrapper that range-checks the result. A bad override now raises a ConfigError naming the offending variable and its value.
python -O strips assert statements, so the bare assert used only to narrow get_int's return type would fall through silently under -O. Use an explicit None check instead; it is still unreachable in practice since get_int only returns None for a None default.
2 participants
The per-check timeout and the total verification budget were hardcoded, which made them impossible to adjust for a slow cluster or a task with a genuinely long convergence window.
The practical failure was a task timing out during verification and reporting a failure that was really just an impatient budget. Tuning it required editing the source.
Both values are now readable from the environment, with the existing values as defaults so behavior is unchanged unless a run opts in.