Skip to content

feat(ci): reverse back-merge promotion (prod→preprod→master)#3382

Merged
yodem merged 4 commits into
masterfrom
feature/reverse-promotion-backmerge
Jul 1, 2026
Merged

feat(ci): reverse back-merge promotion (prod→preprod→master)#3382
yodem merged 4 commits into
masterfrom
feature/reverse-promotion-backmerge

Conversation

@yodem

@yodem yodem commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

Why

After a hotfix lands directly on prod (via hotfix/* → prod, outside the normal twice-weekly master → preprod → prod cadence), that fix must propagate back down the chain so the environments don't diverge. There was no first-class way to do prod → preprod then preprod → master, and the promotion gate falsely failed the prod → preprod PRs that are already being merged by hand.

What

  • promotion-gate.yamlpreprod now accepts prod (reverse back-merge), alongside master / hotfix/*. preprod → master stays ungated (master isn't in the gate's branch list; gating it would block every feature → master PR).
  • manual-promotion.yaml — adds prod source + master target; validation is now a 4-pair allowlist (master→preprod, preprod→prod, prod→preprod, preprod→master); new chain-backmerge-to-master job auto-opens the preprod → master PR when a prod → preprod PR merges. The chained PR is created with BYPASS_BRANCH_PROTECTION_TOKEN so its required checks (PyTest / Playwright / Sandbox Deploy) actually run — a GITHUB_TOKEN-created PR would not trigger them.
  • generate-promotion-pr.sh — maps master → staging env for the rollback-ref lookup so reverse PRs don't warn "unknown".

Rollback-reference fix (app=unknown, chart=unknown)

This branch also fixes the long-standing bug where the auto-generated Rollback Reference table rendered app=unknown, chart=unknown (e.g. promotion PR #3447). Supersedes #3446, which targeted the same bug.

Root cause: the version extraction used python3 -c "..." double-quoted inside $(...), so bash collapsed the [\"\''] quote class to ["\''] within the program text. Python then raised SyntaxError: closing parenthesis ']' does not match — which 2>&1 swallowed and || ="unknown" turned into unknown. Both fields failed identically, hence both showed unknown. (Reproduced against the exact prod helmrelease that was live when #3447 was generated.)

Fix: the Python is now written via a single-quoted heredoc ('PYEOF') so bash performs no substitution on the regex, and the YAML is passed through an env var rather than the command line. Verified: it now extracts the real values (app=v6.101.14-prod.3, chart=0.85.12-prod.1 for the #3447 snapshot).

Additional hardening folded in:

  • Fetch promotion refs with explicit forced refspecs (refs/heads/<b>:refs/remotes/origin/<b>) so origin/<target> is reliably the true remote tip for both the changelog and the rollback lookup.
  • Randomize the GITHUB_OUTPUT heredoc delimiter for ai_summary so a model-generated summary containing a bare EOF line can't inject extra output keys (Actions output injection).
  • Resolve the "already exists" PR URL by SOURCE→TARGET pair via gh pr list --base --head --jq '.[0].url // empty' instead of a bare gh pr view that returned the runner's current-branch PR.
  • Same-repo guard on chain-backmerge-to-master (head.repo.full_name == github.repository) so a fork PR from a branch coincidentally named prod can't trigger the PAT-backed leg.
  • Documented the required fine-grained PAT scope for BYPASS_BRANCH_PROTECTION_TOKEN (Contents: Read + Pull requests: Write, this repo only).

Flow

Trigger prod → preprod from the Actions tab → review/merge → the preprod → master PR opens automatically with checks running → review/merge → environments back in sync.

Reviewer notes / caveats

  • Token scope: the chain leg needs BYPASS_BRANCH_PROTECTION_TOKEN to be able to create PRs (fine-grained PAT with pull-requests: write, contents: read). It's currently proven only for git push in deploy-static.yaml. Please confirm scope.
  • Inert until on preprod: GitHub evaluates pull_request triggers from the base branch's copy of the workflow, so the chain job is dormant until this is promoted to preprod normally. Recommend one deliberate dry-run after it reaches preprod.
  • The chain fires on any merged prod → preprod PR (action-created or hand-made) — the correct invariant: prod commits preprod lacked must reach master.

🤖 Generated with Claude Code

@gitvelocity-reviewer

Copy link
Copy Markdown

📊 Code Quality Score: 20/100

50 × 0.4 (Small ESF) = 20

Category Score Factors
🔭 Scope 10/20 10 files across CI workflows and Helm templates; two distinct subsystems (promotion workflow + secret consolidation); no new public APIs but meaningful cross-cutting infra changes
🏗️ Architecture 11/20 New reverse back-merge pattern with auto-chaining job; secret consolidation from dedicated secret to localSettings; meaningful deployment pipeline architecture change
⚙️ Implementation 9/20 Bash case statement refactor; GitHub Actions conditional job with pull_request event trigger; PR chaining logic; Helm template modifications; moderate complexity overall
⚠️ Risk 12/20 Deployment pipeline changes are high-risk; silent SLACK_URL removal in mongo-backup, reindex-elasticsearch, task, and web rollout templates without replacement; secret consolidation could break existing deployments; no feature flag or rollback mechanism
✅ Quality 5/15 Good inline comments explaining design decisions (injection surface, token rationale, channel behavior); no tests (appropriate for infra); missing operator migration notes for secret consolidation; no NOTES.txt update
🔒 Perf / Security 3/5 Explicit injection surface comment is good defensive practice; optional: true on SLACK_URL prevents hard failures; PAT usage documented with rationale; hardcoded SOURCE/TARGET in chain job prevents payload injection

Was this score accurate? 👍 Yes · 👎 No

Scored by GitVelocity · How are scores calculated?

@yodem yodem requested a review from Copilot June 15, 2026 06:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@yodem yodem requested a review from Copilot June 30, 2026 06:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

yodem and others added 3 commits June 30, 2026 13:41
After a hotfix lands directly on prod, the fix must propagate back down
the chain so environments don't diverge. This adds the reverse paths:

- promotion-gate: allow prod → preprod (reverse back-merge). preprod →
  master stays ungated (master is not in the gate's branch list; gating
  it would block every feature → master PR).
- manual-promotion: add prod source / master target, switch validation to
  a 4-pair allowlist (master→preprod, preprod→prod, prod→preprod,
  preprod→master), and add a chain-backmerge job that auto-opens the
  preprod → master PR when a prod → preprod PR merges. The chained PR is
  created with BYPASS_BRANCH_PROTECTION_TOKEN so its required checks run
  (a GITHUB_TOKEN-created PR would not trigger them).
- generate-promotion-pr.sh: map master → staging env for the rollback-ref
  lookup so reverse PRs don't warn 'unknown'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add 'head.repo.full_name == github.repository' so a fork PR from a branch
  coincidentally named 'prod' can't trigger the PAT-backed back-merge leg.
- Document the required fine-grained PAT scope for BYPASS_BRANCH_PROTECTION_TOKEN.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v1.27.0 runs on the deprecated Node 20 runtime. v3.0.3 runs on Node 24.
The v2+ webhook API replaced the SLACK_WEBHOOK_URL env var with the
'webhook' + 'webhook-type: incoming-webhook' inputs; the payload blocks
are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@yodem yodem force-pushed the feature/reverse-promotion-backmerge branch from f46f6c7 to fa513f9 Compare June 30, 2026 10:41
…n direct -> prod

Hotfixes should ship via hotfix/* -> preprod -> prod (the same gate as
everything else). hotfix/* -> prod stays allowed as a break-glass path for
true emergencies, but the gate now prints a warning and a reminder to run the
reverse back-merge afterwards. Documents the policy in the workflow header.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@yodem yodem merged commit a30936b into master Jul 1, 2026
18 checks passed
@gitvelocity-reviewer

Copy link
Copy Markdown

📊 Code Quality Score: 22/100

55 × 0.4 (Small ESF) = 22

Category Score Factors
🔭 Scope 10/20 2 critical CI/CD files, new automation trigger, new workflow job, affects entire deployment pipeline
🏗️ Architecture 12/20 New chained automation pattern, event-driven job addition, new PAT secret with documented scope, Slack action API migration
⚙️ Implementation 10/20 case statement refactor, same-repo security guard, 5-condition chain job guard, injection surface elimination via hardcoded values
⚠️ Risk 11/20 Deployment pipeline changes carry inherent risk, PAT-backed automation bypasses branch protection, pull_request trigger requires careful guarding; mitigated by same-repo guard and reversibility
✅ Quality 9/15 Excellent inline documentation of design rationale and security considerations, PAT scope requirements documented, no automated tests (expected for CI/CD YAML)
🔒 Perf / Security 3/5 Same-repo fork guard (+2), documented minimal PAT scope (+1), hardcoded values to prevent injection (+1); no formal threat model

Was this score accurate? 👍 Yes · 👎 No

Scored by GitVelocity · How are scores calculated?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants