Skip to content

fix(toolkit-lib): NoStack masks the real error when a new stack fails to deploy - #1845

Merged
aws-cdk-automation merged 1 commit into
aws:mainfrom
svozza:fix/nostack-masks-real-error
Aug 18, 2026
Merged

fix(toolkit-lib): NoStack masks the real error when a new stack fails to deploy#1845
aws-cdk-automation merged 1 commit into
aws:mainfrom
svozza:fix/nostack-masks-real-error

Conversation

@svozza

@svozza svozza commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Relates to #1802

First of two PRs, split per review feedback on #1803. This is the error-masking half; the stale DescribeStacks read follows separately.

The bug

monitorDeployment passed finalState.wrapped to the diagnoser from inside its catch block. finalState is still the pre-deploy lookup at that point, which holds no stack when the deployment was creating one from scratch — so the wrapped getter threw NoStack. Because that happened while evaluating an argument, it replaced the deployment error being reported:

NoStack: CloudFormationStack object does not hold a stack

Every failed deployment of a new stack was affected, not only the ones from the stale-read bug in #1802. A plain resource failure that rolled the stack back reported NoStack instead of naming the resource, and the diagnoser's CloudTrail enrichment was discarded.

What changed

Diagnosis is an enrichment of the deployment error — it can only ever replace that error with something that says more. The old code broke that in two ways, and the guard for a third was missing. It now lives in one helper that holds to the invariant:

} catch (e: any) {
  // Deployment errors get replaced by a diagnosis of the underlying resource failures, which says more.
  // Any other error, and any failure to diagnose, leaves `e` to propagate as it is.
  if (ToolkitError.isDeploymentError(e)) {
    await this.diagnoseDeploymentFailure(stackArn, monitor.errors);
  }

  throw e;
}
  • Describe the stack that was actually deployed. The pre-deploy description is absent when creating and stale when updating, and the diagnoser reads the status off it.
  • A failed describe returns without throwing, leaving the original error to propagate rather than replacing it with something less useful. This answers the "but this can also fail... what then?" comment.
  • The lookup failure is logged at debug, matching stack-diagnoser.ts:163 — a throttled or access-denied diagnosis was previously invisible even under -v.

Two consequences worth calling out, both of which shrink the surface rather than grow it:

  • finalState's initialiser is now provably dead — the try block either assigns or throws, and the catch always rethrows — so the mutable local that meant two different things at two points in the method no longer starts life holding a pre-deploy value. That's what made this bug representable in the first place.
  • With the helper describing the stack directly rather than through CloudFormationStack.lookup(...).wrapped, the wrapped getter has no callers left anywhere in the repo, so it's removed. The class whose throwing getter caused this no longer has one.

Testing

New deploy-stack-error-surfacing.test.ts, 6 cases: a failing create via change-set and direct, each with rollback on and off; a failing update of an existing stack (guards against a fix that only works when the stack is missing); and a describe that fails mid-diagnosis, asserting the original ROLLBACK_COMPLETE error survives and the swallowed failure is still visible at debug level.

Written before the fix — 5 of the 6 failed at stack-helpers.ts:75 via the catch block, and the existing-stack case passed both before and after. Each subsequent cleanup was verified load-bearing by reverting it and confirming a specific test failed.

Full toolkit-lib suite passes (1905 tests). No integ test: no new resource types or cross-service interactions.

Follow-ups, deliberately not here

  • waitForStackDeploy already holds the settled CloudFormationStack from stabilizeStack and discards it when throwing, so this helper re-describes the same stack. Worth fixing — the CFN client is configured with 7 retries capped at 15s, so on a throttled account that redundant read can add up to ~60s per failed stack, on precisely the path where throttling is the likely cause. It needs a cfn-api.ts signature change, so it's out of scope for this PR.
  • Moving diagnosis into CloudFormationStackDiagnoser (so callers never hold a raw Stack) needs Diagnosis.throwOnError() split into throwOnError/throwOnProblem first. Otherwise a diagnoser-owned lookup failure surfaces as ErrorDiagnosisFailed and masks the real error — this same bug class through a different door.

Checklist

  • This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
    • Release notes for the new version:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@codecov-commenter

codecov-commenter commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.35%. Comparing base (5498a1a) to head (b8d4ee0).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1845   +/-   ##
=======================================
  Coverage   90.35%   90.35%           
=======================================
  Files          80       80           
  Lines       12159    12159           
  Branches     1727     1727           
=======================================
  Hits        10986    10986           
  Misses       1139     1139           
  Partials       34       34           
Flag Coverage Δ
suite.unit 90.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

… to deploy

`monitorDeployment` passed `finalState.wrapped` to the diagnoser from inside
its catch block. `finalState` is still the pre-deploy lookup at that point,
which holds no stack when the deployment was creating one from scratch, so
the getter threw `ToolkitError('NoStack')`. Because that happened while
evaluating an argument, it replaced the deployment error being reported, and
users saw

    NoStack: CloudFormationStack object does not hold a stack

Every failed deployment of a new stack was affected: a resource failure that
rolled the stack back reported `NoStack` rather than naming the resource.

Diagnosis is an enrichment of the deployment error, so it can only ever
replace that error with something that says more. Move it into a helper that
holds to this:

- describe the stack that was actually deployed, rather than a pre-deploy
  description that is absent when creating and stale when updating;
- return without throwing if that lookup fails or the stack is gone, leaving
  the original error to propagate;
- otherwise diagnose, and let `throwOnError` replace the error only when a
  cause was established.

Fixes aws#1802
@mrgrain
mrgrain force-pushed the fix/nostack-masks-real-error branch from e3601c1 to b8d4ee0 Compare August 18, 2026 12:25
@mrgrain
mrgrain deployed to integ-approval August 18, 2026 12:25 — with GitHub Actions Active
@aws-cdk-automation
aws-cdk-automation added this pull request to the merge queue Aug 18, 2026
Merged via the queue into aws:main with commit a9a23b4 Aug 18, 2026
46 checks passed
9pace pushed a commit to 9pace/aws-cdk-cli that referenced this pull request Aug 19, 2026
…yment (aws#1848)

Fixes aws#1802

Second of two PRs, split per review feedback on aws#1803. aws#1845 fixed the
error masking; this is the stale `DescribeStacks` read that caused the
failure in the first place.

## The bug

`stabilizeStack` polls `DescribeStacks` until the stack is no longer
`*_IN_PROGRESS`, and treated `REVIEW_IN_PROGRESS` as a stable, terminal
status. Because that read is eventually consistent, a poll issued after
`ExecuteChangeSet` can still report the pre-execution status — so
`waitForStackDeploy` rejected the deployment with `StackDeployFailed`
while CloudFormation went on to reach `CREATE_COMPLETE`.

`REVIEW_IN_PROGRESS` is ambiguous, and `stabilizeStack` cannot tell its
two meanings apart from what it observes:

- a `CREATE` ChangeSet was created and never executed — nothing will
move the stack, so waiting hangs forever;
- a ChangeSet *was* executed and this read is stale — treating it as
terminal fails a running deployment.

The caller knows which. `waitForStackDeploy` is reached immediately
after `ExecuteChangeSet`/`CreateStack`; the delete, rollback and
bootstrap-lookup callers have executed nothing. So that fact is now
passed down rather than guessed at:

```ts
const stack = await stabilizeStack(cfn, ioHelper, stackNameOrArn, {
  pollingInterval: stabilizationPollingInterval,
  changeSetExecuted: true,
});
```

## What `REVIEW_IN_PROGRESS` actually does

The first version of this bounded every stale read by a retry budget,
which was a guess. To replace the guess I ran the case directly against
CloudFormation — a `CREATE` ChangeSet whose only resource fails
immediately (an SNS topic with an invalid name), executed, polling
`DescribeStacks` at 0.5s:

```
13:57:35  REVIEW_IN_PROGRESS    User Initiated
13:58:00  CREATE_IN_PROGRESS    User Initiated
13:58:02  Bad  CREATE_IN_PROGRESS
13:58:03  Bad  CREATE_FAILED     Invalid parameter: Topic Name
13:58:03  ROLLBACK_IN_PROGRESS  The following resource(s) failed to create: [Bad]
14:00:27  ROLLBACK_COMPLETE
```

`REVIEW_IN_PROGRESS` appears exactly once, as the stack's initial state,
and is never re-entered — including on the path where execution fails
before touching a single resource, which was the case I had assumed
could return to it.

That gives a rule rather than a guess:

- **once the stack has been seen with an operation in progress**, a
`REVIEW_IN_PROGRESS` read is *provably* stale, so keep polling with no
budget at all;
- **before that**, the two meanings are genuinely indistinguishable, so
allow a bounded number of re-reads (`STALE_REVIEW_READ_ATTEMPTS`) and
then believe the status, because nothing moves an unexecuted ChangeSet
on its own.

The distinction matters: a budget spent across the whole wait would
exhaust itself on a long deployment with several scattered stale reads
and reinstate the original bug. There is a test for that case.

## Also here

**Polling is pinned to the stack's ARN** after the first successful
read. Polling by name can otherwise observe a *different* stack that a
concurrent operation created under the same name, and mistake it for the
one being deployed. `monitorDeployment` and `rollbackStack` now pass the
ARN they already have, per the review suggestion on aws#1803.

**A stack deleted mid-wait is reported as gone.** This falls out of the
ARN pinning: `DescribeStacks` keeps answering for a `DELETE_COMPLETE`
stack when asked by ARN, where a by-name read stops finding it. Without
this, `cdk rollback` returned `success: true` for a stack that had been
deleted out from under it — `stabilizeStack` handed back the deleted
stack as a stable status, skipping the `StackDisappeared` error, and a
clean delete leaves no monitor errors to fail on. Caught by review and
fixed with a test that reproduced the false success first.

**Deployment errors name the stack** rather than echoing the caller's
argument, which would now put a full ARN in front of the user.

## Testing

`cfn-api-stabilization-polling-interval.test.ts` is renamed to
`cfn-api-stabilization.test.ts` and the new cases join it, rather than
adding a second file with a near-identical harness and a confusingly
similar name. 12 tests, covering: a stale read mid-wait; more stale
reads than the leading budget across a long deployment; a leading review
status that never resolves; a stack deleted mid-wait; name-to-ARN
narrowing; and the delete/rollback/bootstrap callers keeping today's
behaviour.

Every new test was confirmed to fail against the unfixed code and pass
after; each guard was re-checked by reverting it and watching the
specific test fail. Full `toolkit-lib` suite passes (1915 tests).

`deploy-stack-error-surfacing.test.ts` needed one narrowing: its
throttle mock keyed on "any ARN", which was a fine proxy when only the
diagnosis read used one. Now that stabilization does too, it throttles
reads after the deployment settles instead.

## Not here

`deployments.ts` reads `cloudFormationStack.stackId` before checking
`.exists`, and that getter throws when the stack is absent — a
pre-existing latent crash on the rollback path, unrelated to this
change. Happy to fix separately.

### Checklist
- [ ] This change contains a major version upgrade for a dependency and
I confirm all breaking changes are addressed
  - Release notes for the new version:

---
By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache-2.0 license
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants