Skip to content

fix(ci): correctly extract rollback app/chart versions in promotion PRs#3446

Closed
yodem wants to merge 6 commits into
masterfrom
fix/sc-promotion-rollback-versions
Closed

fix(ci): correctly extract rollback app/chart versions in promotion PRs#3446
yodem wants to merge 6 commits into
masterfrom
fix/sc-promotion-rollback-versions

Conversation

@yodem

@yodem yodem commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Problem

The Manual Promotion workflow's auto-generated PRs always show the Rollback Reference as app=unknown, chart=unknown``. See #3445, where the table reads:

| Current on preprod | app=unknown, chart=unknown |

This defeats the whole point of the section — there's no recorded version to roll back to.

Root cause

build/ci/generate-promotion-pr.sh extracted the versions by piping the helmrelease YAML through python3 -c "<regex>" where the Python (and the regex inside it) lived in a double-quoted bash string. Bash mangled the regex's backslashes and quotes (\s, [\"\''], etc.) before Python ever saw it, so the re.search never matched and both values fell back to unknown.

I reproduced this locally: the regexes match fine when handed to Python directly, but fail when routed through the exact python3 -c "..." bash-string path the script used.

Fix

  • Write the parser Python via a single-quoted heredoc so bash performs no substitution on the regex.
  • Pass the YAML through an environment variable instead of escaping it on the command line.
  • Extract both the app image tag and chart version in a single pass.
  • The heredoc writes to a temp file (kept out of $(...)) so it stays portable to bash 3.2 as well as the bash 5.x on CI runners.

Verification

Ran the real script end-to-end against origin/preprod with gh stubbed:

prev_app=v6.102.1-preprod.2
prev_chart=0.85.13-preprod.2

instead of unknown. bash -n passes on both bash 3.2 (macOS) and the CI bash. No behavior change to changelog/AI-summary/PR-creation logic.

🤖 Generated with Claude Code

yodem and others added 6 commits June 2, 2026 13:41
…L; retire slack-webhook secret [sc-44609]

SLACK_URL was injected twice: an explicit `env` from the dedicated
slack-webhook secret (sefaria.secrets.slackWebhook) and the SLACK_URL key
bundled in local-settings-secrets (via envFrom). In Kubernetes an explicit
`env` always wins over an envFrom key of the same name, so SLACK_URL overrides
set in an environment's / cauldron's local-settings-secrets were silently
ignored, leaving pods serving a stale webhook (invalid_token).

Make local-settings-secrets the single source of truth and retire the
dedicated slack-webhook secret entirely:

- Remove the redundant explicit SLACK_URL env from the pods that already mount
  local-settings-secrets (web, task, reindex-elasticsearch, mongo-backup,
  postgres-packup) so the envFrom value takes effect.
- Migrate the blue-green deploy notifier (analysistemplate/rollout-complete)
  off the slack-webhook secret: SLACK_URL now comes from local-settings-secrets
  and the target channel from a new SLACK_CHANNEL key there. Both are
  optional (the curl is also `|| /bin/true`) so the post-promotion
  notification never blocks a rollout.
- Delete the slack-webhook secret template, the sefaria.secrets.slackWebhook
  helper, and the secrets.slackWebhook values stanza.

Companion infra change (separate PR): add the SLACK_CHANNEL key to
local-settings-secrets (prod + dev) and remove the orphaned slack-webhook-*
secrets. SLACK_URL already exists in those secrets.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…localSettings ref [sc-44609]

Addresses Copilot review on PR #3369:

- postgres-packup: the pgdump-uploader container (which runs upload-dumps.sh →
  `curl ... ${SLACK_URL}` under `set -e`) has no envFrom, so dropping the explicit
  SLACK_URL env left it unset and would fail the backup job. Restore SLACK_URL from
  the configured local-settings secret.
- mongo-backup: upload-dumps.sh (mongo) does not use SLACK_URL at all — the previous
  env was dead. Keep it removed; correct the misleading comment.
- rollout-complete analysistemplate: reference {{ .Values.secrets.localSettings.ref }}
  (prod: local-settings-secrets-production, where SLACK_URL/SLACK_CHANNEL live) instead
  of the hardcoded local-settings-secrets-{{ deployEnv }} pattern.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nnel [sc-44609]

The deploy notifier uses a Slack incoming webhook (SLACK_URL), which is bound to
a channel at creation; the payload channel override is redundant (and ignored by
app-based webhooks). Remove the CHANNEL env and the channel field from the payload,
so no SLACK_CHANNEL key is needed in local-settings-secrets.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The explanatory comments read well in the PR diff but are noise in the
long-lived chart after merge. Remove them; keep two short present-tense
notes where the behavior is non-obvious (pg-backup uploader's explicit
SLACK_URL, and the deliberate absence of a channel override).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Manual Promotion workflow's "Rollback Reference" always showed
app=`unknown`, chart=`unknown` (e.g. PR #3445). The version extraction
passed the regex through `python3 -c "..."` inside a double-quoted bash
string, so bash mangled the backslashes and quotes in the regex before
Python ever saw it, and the match always failed.

Feed the parser Python via a single-quoted heredoc (written to a temp
file so it stays portable to bash 3.2 — heredocs nested in $(...) break
there) and pass the YAML through an env var instead of escaping it on the
command line. Both app tag and chart version are now extracted in one
pass and surface correctly in the rollback reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gitvelocity-reviewer

Copy link
Copy Markdown

📊 Code Quality Score: 2/100

16 × 0.1 = 1.6, rounded to 2

Category Score Factors
🔭 Scope 2/20 Single CI script file, localized bug fix, no new APIs or subsystems
🏗️ Architecture 1/20 No architectural changes; minor script restructuring from two Python calls to one
⚙️ Implementation 4/20 Demonstrates bash heredoc quoting knowledge, Python subprocess patterns, env var passing; non-trivial root cause diagnosis
⚠️ Risk 3/20 CI-only change, easily reversible, low blast radius; no production code affected
✅ Quality 5/15 Good explanatory comment documenting root cause; clean implementation; no tests needed for CI script fix
🔒 Perf / Security 1/5 Env var approach avoids potential command-line injection of arbitrary YAML content

Was this score accurate? 👍 Yes · 👎 No

Scored by GitVelocity · How are scores calculated?

@yodem

yodem commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #3382, which carries the same root-cause fix (the version extraction now uses a single-quoted heredoc instead of the bash-mangled python3 -c "...", so the rollback reference no longer renders app=unknown, chart=unknown). Verified against the prod helmrelease that was live when #3447 was generated. Closing as redundant.

@yodem yodem closed this Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant