Skip to content

test(e2e): derive scale-to-zero reporter waits from the reporter's timings - #270

Merged
rambohe-ch merged 1 commit into
kaito-project:mainfrom
rambohe-ch:reporter-emit-timeout
Nov 6, 2026
Merged

rambohe-ch merged 1 commit into
kaito-project:mainfrom
rambohe-ch:reporter-emit-timeout

Conversation

@rambohe-ch

@rambohe-ch rambohe-ch commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Reason for Change:

Cluster Status Reporter emits clusterBBRNotReady when BBR is scaled to zero fails intermittently on a cluster where the status reporter runs with the chart defaults:

timed out waiting for reporter event reason=clusterBBRNotReady object="" newer than 0001-01-01 00:00:00 +0000 UTC

A Deployment scaled to zero has no pods, so the reporter debounces it by first observation. It sees the problem on the next resync, and only publishes once a later resync finds it has lasted the startup grace. The specs waited a fixed 3 minutes, tuned for the E2E install's 30s grace:

--resync-seconds --startup-grace-seconds --event-refresh-seconds
E2E install (install-components.sh) 60 30 120
Chart defaults (values.yaml) 60 60 300

With grace equal to resync, tick jitter can push the emit one resync later. The scaled-down pod also has to terminate first (terminationGracePeriodSeconds: 30), so the worst case reaches the 180s limit.

 Cluster Status Reporter / EPP scale-to-zero
-  emitTimeout = 3 * time.Minute                  // fixed
+  timings := utils.GetReporterTimings(ctx)       // flags from the running reporter, chart defaults otherwise
+  emitTimeout = timings.DebouncedEmitTimeout()
+    = resync + ceil(grace/resync)*resync + resync (jitter) + 90s slack
+    = 270s for both settings
   recovery quiet window: 3m (unchanged)

ReporterEventRefreshInterval now reads through GetReporterTimings, with the same behaviour.

Requirements

  • added unit tests and e2e tests (if applicable).

Issue Fixed:

Notes for Reviewers:

Evidence

  • Before: on a cluster running the reporter with the chart defaults (--resync-seconds=60 --startup-grace-seconds=60), scaling body-based-router to zero produced clusterBBRNotReady after 78s, 189s, 189s. Two of three runs exceed the old 180s wait.
  • After:
    • go test -mod=mod -count=1 ./test/e2e/utils/ → passed. It includes the new TestReporterTimingsFromArgs (defaults, E2E flags, zero grace, invalid values) and TestDebouncedEmitTimeout (the chart-defaults regression case is 270s).
    • go vet -mod=mod ./test/e2e/... → passed.
    • bin/golangci-lint run ./test/e2e/... → passed, 0 issues.
    • E2E on the same cluster, cluster_status_spec.go and control_plane_error_spec.go (!StandardK8sOnly && !Nightly && !Perf) → passed, SUCCESS! -- 7 Passed | 0 Failed.

Merge Danger

Door: two-way. Test-only: no product code changes.

Blast Radius: e2e-timing. On a real failure, the scale-to-zero specs now wait up to 270s instead of 180s. Passing runs are unchanged, because the wait returns as soon as the Event appears.

…mings

A Deployment scaled to zero has no pods, so the status reporter debounces
it by first observation: it sees the problem on the next resync and only
publishes it once a later resync finds it has persisted for the startup
grace. The cluster reporter specs waited a fixed 3 minutes, tuned for the
E2E install's 30s grace. With the chart defaults (60s resync, 60s grace),
tick jitter can add one more resync, and the scaled-down pod first has to
terminate: clusterBBRNotReady took 189s in two of three measurements on a
cluster running the chart defaults, so the spec timed out.

- Add utils.GetReporterTimings, which reads --resync-seconds,
  --startup-grace-seconds and --event-refresh-seconds from the running
  reporter (chart defaults when absent), and
  ReporterTimings.DebouncedEmitTimeout: resync + grace rounded up to
  resyncs + one resync of jitter + 90s slack (270s for both settings).
- Cluster Status Reporter and the EPP scale-to-zero spec use it. The
  recovery quiet window stays 3 minutes.
- ReporterEventRefreshInterval now reads through GetReporterTimings.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: rambohe-ch <rambohe.ch@gmail.com>
@rambohe-ch
rambohe-ch force-pushed the reporter-emit-timeout branch from ed157ee to ac610a1 Compare November 5, 2026 23:16
@rambohe-ch
rambohe-ch merged commit 1051801 into kaito-project:main Nov 6, 2026
13 checks passed

This branch was previously deployed

1 inactive deployment
e2e-test — ac610a13 Deployed Nov 5, 2026 by rambohe-ch via e2e-tests #693
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant