Repository navigation
test(e2e): derive scale-to-zero reporter waits from the reporter's timings - #270
Merged
Merged
Conversation
rambohe-ch
requested review from
Fei-Guo,
techworldhello and
zhuangqh
as code owners
November 5, 2026 22:47
…mings A Deployment scaled to zero has no pods, so the status reporter debounces it by first observation: it sees the problem on the next resync and only publishes it once a later resync finds it has persisted for the startup grace. The cluster reporter specs waited a fixed 3 minutes, tuned for the E2E install's 30s grace. With the chart defaults (60s resync, 60s grace), tick jitter can add one more resync, and the scaled-down pod first has to terminate: clusterBBRNotReady took 189s in two of three measurements on a cluster running the chart defaults, so the spec timed out. - Add utils.GetReporterTimings, which reads --resync-seconds, --startup-grace-seconds and --event-refresh-seconds from the running reporter (chart defaults when absent), and ReporterTimings.DebouncedEmitTimeout: resync + grace rounded up to resyncs + one resync of jitter + 90s slack (270s for both settings). - Cluster Status Reporter and the EPP scale-to-zero spec use it. The recovery quiet window stays 3 minutes. - ReporterEventRefreshInterval now reads through GetReporterTimings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
rambohe-ch
force-pushed
the
reporter-emit-timeout
branch
from
November 5, 2026 23:16
ed157ee to
ac610a1
Compare
1 task
1 task done
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for Change:
Cluster Status Reporter emits clusterBBRNotReady when BBR is scaled to zerofails intermittently on a cluster where the status reporter runs with the chart defaults:A Deployment scaled to zero has no pods, so the reporter debounces it by first observation. It sees the problem on the next resync, and only publishes once a later resync finds it has lasted the startup grace. The specs waited a fixed 3 minutes, tuned for the E2E install's 30s grace:
--resync-seconds--startup-grace-seconds--event-refresh-secondsinstall-components.sh)values.yaml)With grace equal to resync, tick jitter can push the emit one resync later. The scaled-down pod also has to terminate first (
terminationGracePeriodSeconds: 30), so the worst case reaches the 180s limit.ReporterEventRefreshIntervalnow reads throughGetReporterTimings, with the same behaviour.Requirements
Issue Fixed:
Notes for Reviewers:
Evidence
--resync-seconds=60 --startup-grace-seconds=60), scalingbody-based-routerto zero producedclusterBBRNotReadyafter 78s, 189s, 189s. Two of three runs exceed the old 180s wait.go test -mod=mod -count=1 ./test/e2e/utils/→ passed. It includes the newTestReporterTimingsFromArgs(defaults, E2E flags, zero grace, invalid values) andTestDebouncedEmitTimeout(the chart-defaults regression case is 270s).go vet -mod=mod ./test/e2e/...→ passed.bin/golangci-lint run ./test/e2e/...→ passed, 0 issues.cluster_status_spec.goandcontrol_plane_error_spec.go(!StandardK8sOnly && !Nightly && !Perf) → passed,SUCCESS! -- 7 Passed | 0 Failed.Merge Danger
Door: two-way. Test-only: no product code changes.
Blast Radius: e2e-timing. On a real failure, the scale-to-zero specs now wait up to 270s instead of 180s. Passing runs are unchanged, because the wait returns as soon as the Event appears.