fix(toolkit-lib): NoStack masks the real error when a new stack fails to deploy - #1845
Merged
Merged
Conversation
svozza
had a problem deploying
to
integ-approval
August 18, 2026 11:50 — with
GitHub Actions
Failure
aws-cdk-automation
enabled auto-merge
August 18, 2026 11:50
Closed
1 task
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1845 +/- ##
=======================================
Coverage 90.35% 90.35%
=======================================
Files 80 80
Lines 12159 12159
Branches 1727 1727
=======================================
Hits 10986 10986
Misses 1139 1139
Partials 34 34
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… to deploy
`monitorDeployment` passed `finalState.wrapped` to the diagnoser from inside
its catch block. `finalState` is still the pre-deploy lookup at that point,
which holds no stack when the deployment was creating one from scratch, so
the getter threw `ToolkitError('NoStack')`. Because that happened while
evaluating an argument, it replaced the deployment error being reported, and
users saw
NoStack: CloudFormationStack object does not hold a stack
Every failed deployment of a new stack was affected: a resource failure that
rolled the stack back reported `NoStack` rather than naming the resource.
Diagnosis is an enrichment of the deployment error, so it can only ever
replace that error with something that says more. Move it into a helper that
holds to this:
- describe the stack that was actually deployed, rather than a pre-deploy
description that is absent when creating and stale when updating;
- return without throwing if that lookup fails or the stack is gone, leaving
the original error to propagate;
- otherwise diagnose, and let `throwOnError` replace the error only when a
cause was established.
Fixes aws#1802
mrgrain
force-pushed
the
fix/nostack-masks-real-error
branch
from
August 18, 2026 12:25
e3601c1 to
b8d4ee0
Compare
mrgrain
approved these changes
Aug 18, 2026
1 task
9pace
pushed a commit
to 9pace/aws-cdk-cli
that referenced
this pull request
Aug 19, 2026
…yment (aws#1848) Fixes aws#1802 Second of two PRs, split per review feedback on aws#1803. aws#1845 fixed the error masking; this is the stale `DescribeStacks` read that caused the failure in the first place. ## The bug `stabilizeStack` polls `DescribeStacks` until the stack is no longer `*_IN_PROGRESS`, and treated `REVIEW_IN_PROGRESS` as a stable, terminal status. Because that read is eventually consistent, a poll issued after `ExecuteChangeSet` can still report the pre-execution status — so `waitForStackDeploy` rejected the deployment with `StackDeployFailed` while CloudFormation went on to reach `CREATE_COMPLETE`. `REVIEW_IN_PROGRESS` is ambiguous, and `stabilizeStack` cannot tell its two meanings apart from what it observes: - a `CREATE` ChangeSet was created and never executed — nothing will move the stack, so waiting hangs forever; - a ChangeSet *was* executed and this read is stale — treating it as terminal fails a running deployment. The caller knows which. `waitForStackDeploy` is reached immediately after `ExecuteChangeSet`/`CreateStack`; the delete, rollback and bootstrap-lookup callers have executed nothing. So that fact is now passed down rather than guessed at: ```ts const stack = await stabilizeStack(cfn, ioHelper, stackNameOrArn, { pollingInterval: stabilizationPollingInterval, changeSetExecuted: true, }); ``` ## What `REVIEW_IN_PROGRESS` actually does The first version of this bounded every stale read by a retry budget, which was a guess. To replace the guess I ran the case directly against CloudFormation — a `CREATE` ChangeSet whose only resource fails immediately (an SNS topic with an invalid name), executed, polling `DescribeStacks` at 0.5s: ``` 13:57:35 REVIEW_IN_PROGRESS User Initiated 13:58:00 CREATE_IN_PROGRESS User Initiated 13:58:02 Bad CREATE_IN_PROGRESS 13:58:03 Bad CREATE_FAILED Invalid parameter: Topic Name 13:58:03 ROLLBACK_IN_PROGRESS The following resource(s) failed to create: [Bad] 14:00:27 ROLLBACK_COMPLETE ``` `REVIEW_IN_PROGRESS` appears exactly once, as the stack's initial state, and is never re-entered — including on the path where execution fails before touching a single resource, which was the case I had assumed could return to it. That gives a rule rather than a guess: - **once the stack has been seen with an operation in progress**, a `REVIEW_IN_PROGRESS` read is *provably* stale, so keep polling with no budget at all; - **before that**, the two meanings are genuinely indistinguishable, so allow a bounded number of re-reads (`STALE_REVIEW_READ_ATTEMPTS`) and then believe the status, because nothing moves an unexecuted ChangeSet on its own. The distinction matters: a budget spent across the whole wait would exhaust itself on a long deployment with several scattered stale reads and reinstate the original bug. There is a test for that case. ## Also here **Polling is pinned to the stack's ARN** after the first successful read. Polling by name can otherwise observe a *different* stack that a concurrent operation created under the same name, and mistake it for the one being deployed. `monitorDeployment` and `rollbackStack` now pass the ARN they already have, per the review suggestion on aws#1803. **A stack deleted mid-wait is reported as gone.** This falls out of the ARN pinning: `DescribeStacks` keeps answering for a `DELETE_COMPLETE` stack when asked by ARN, where a by-name read stops finding it. Without this, `cdk rollback` returned `success: true` for a stack that had been deleted out from under it — `stabilizeStack` handed back the deleted stack as a stable status, skipping the `StackDisappeared` error, and a clean delete leaves no monitor errors to fail on. Caught by review and fixed with a test that reproduced the false success first. **Deployment errors name the stack** rather than echoing the caller's argument, which would now put a full ARN in front of the user. ## Testing `cfn-api-stabilization-polling-interval.test.ts` is renamed to `cfn-api-stabilization.test.ts` and the new cases join it, rather than adding a second file with a near-identical harness and a confusingly similar name. 12 tests, covering: a stale read mid-wait; more stale reads than the leading budget across a long deployment; a leading review status that never resolves; a stack deleted mid-wait; name-to-ARN narrowing; and the delete/rollback/bootstrap callers keeping today's behaviour. Every new test was confirmed to fail against the unfixed code and pass after; each guard was re-checked by reverting it and watching the specific test fail. Full `toolkit-lib` suite passes (1915 tests). `deploy-stack-error-surfacing.test.ts` needed one narrowing: its throttle mock keyed on "any ARN", which was a fine proxy when only the diagnosis read used one. Now that stabilization does too, it throttles reads after the deployment settles instead. ## Not here `deployments.ts` reads `cloudFormationStack.stackId` before checking `.exists`, and that getter throws when the stack is absent — a pre-existing latent crash on the rollback path, unrelated to this change. Happy to fix separately. ### Checklist - [ ] This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed - Release notes for the new version: --- By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates to #1802
First of two PRs, split per review feedback on #1803. This is the error-masking half; the stale
DescribeStacksread follows separately.The bug
monitorDeploymentpassedfinalState.wrappedto the diagnoser from inside its catch block.finalStateis still the pre-deploy lookup at that point, which holds no stack when the deployment was creating one from scratch — so thewrappedgetter threwNoStack. Because that happened while evaluating an argument, it replaced the deployment error being reported:Every failed deployment of a new stack was affected, not only the ones from the stale-read bug in #1802. A plain resource failure that rolled the stack back reported
NoStackinstead of naming the resource, and the diagnoser's CloudTrail enrichment was discarded.What changed
Diagnosis is an enrichment of the deployment error — it can only ever replace that error with something that says more. The old code broke that in two ways, and the guard for a third was missing. It now lives in one helper that holds to the invariant:
stack-diagnoser.ts:163— a throttled or access-denied diagnosis was previously invisible even under-v.Two consequences worth calling out, both of which shrink the surface rather than grow it:
finalState's initialiser is now provably dead — the try block either assigns or throws, and the catch always rethrows — so the mutable local that meant two different things at two points in the method no longer starts life holding a pre-deploy value. That's what made this bug representable in the first place.CloudFormationStack.lookup(...).wrapped, thewrappedgetter has no callers left anywhere in the repo, so it's removed. The class whose throwing getter caused this no longer has one.Testing
New
deploy-stack-error-surfacing.test.ts, 6 cases: a failing create via change-set and direct, each with rollback on and off; a failing update of an existing stack (guards against a fix that only works when the stack is missing); and a describe that fails mid-diagnosis, asserting the originalROLLBACK_COMPLETEerror survives and the swallowed failure is still visible at debug level.Written before the fix — 5 of the 6 failed at
stack-helpers.ts:75via the catch block, and the existing-stack case passed both before and after. Each subsequent cleanup was verified load-bearing by reverting it and confirming a specific test failed.Full
toolkit-libsuite passes (1905 tests). No integ test: no new resource types or cross-service interactions.Follow-ups, deliberately not here
waitForStackDeployalready holds the settledCloudFormationStackfromstabilizeStackand discards it when throwing, so this helper re-describes the same stack. Worth fixing — the CFN client is configured with 7 retries capped at 15s, so on a throttled account that redundant read can add up to ~60s per failed stack, on precisely the path where throttling is the likely cause. It needs acfn-api.tssignature change, so it's out of scope for this PR.CloudFormationStackDiagnoser(so callers never hold a rawStack) needsDiagnosis.throwOnError()split intothrowOnError/throwOnProblemfirst. Otherwise a diagnoser-owned lookup failure surfaces asErrorDiagnosisFailedand masks the real error — this same bug class through a different door.Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license