Repository navigation
ci(cli): Alert when nightly publishing stalls - #1530
Merged
Merged
Conversation
A darwin-arm64 Build job sat in an environment wait for ten days. Every newer main push queued behind it, so GHCR nightly stopped at Oct 6. Add an hourly watchdog that flags stuck or failed Build runs and a nightly that lags main, tracked in one auto-closing issue with optional Slack notification. Add job timeouts to the Build workflow. Co-Authored-By: Burak Yigit Kaya <burak.kaya@sentry.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
BYK
approved these changes
Oct 16, 2026
BYK
enabled auto-merge (squash)
October 16, 2026 22:21
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 62347de. Configure here.
Query unfinished Build runs by status so a stuck run is still found after more than 50 newer pushes. Measure nightly staleness against the newest run created before the grace period instead of main HEAD, so fresh pushes no longer restart the grace window.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CLI nightly on GHCR stopped at
0.48.0-dev.1791270977(Oct 6) and nobody found out for ten days. In Build run 37441622445, thedarwin-arm64job got stuck inwaitingon theproductionenvironment. That environment has no reviewers and no wait timer, so nothing could ever approve it. Build usescancel-in-progress: false, so every newer main push queued behind that run and was then replaced by the next push. That stuck run has been cancelled now, and the next main build published0.48.0-dev.1792188007.This adds a
CLI Nightly Watchdogworkflow. It runs hourly and after each main Build completes, and it flags:nightlyolder than the newest built main commit once a 3h grace period has passed, or a manifest it can't readWhile something is wrong, it keeps one open issue (
CLI nightly publishing is unhealthy) with the current report and fails its own run. Once healthy again, it closes the issue. If theCLI_NIGHTLY_ALERT_SLACK_WEBHOOK_URLsecret is set, it posts to Slack when the issue opens and when it closes. Without the secret it skips Slack.The Build jobs also get
timeout-minutes, so a hung job can't sit for the 6h default. Timeouts don't cover jobs still waiting on an environment gate, which is why the watchdog checks run age.via byk.
--
View Junior Session [Sentry]