Fix: UPDATE_ROLLBACK_FAILED

Your CloudFormation update failed, and then the automatic rollback that was supposed to restore the previous working state failed too, leaving the stack in UPDATE_ROLLBACK_FAILED. This is a genuinely stuck state, but unlike ROLLBACK_COMPLETE it is recoverable in place. You do not delete the stack. Instead you tell CloudFormation to continue the rollback, skipping only the resources that truly cannot be reverted, and then you reconcile those resources by hand so the template and reality agree again.

UPDATE_ROLLBACK_FAILED

What this error means

When you update a healthy stack and something in the update fails, CloudFormation automatically tries to roll every resource back to its previous state. UPDATE_ROLLBACK_FAILED means that rollback itself could not finish: at least one resource could not be returned to how it was before the update started. The stack is now in a state where it cannot accept a new update until the rollback is completed or explicitly forced to finish. The recovery command is continue-update-rollback, which retries the rollback. If a specific resource simply cannot be reverted (for example it was changed outside CloudFormation, or its prior configuration is no longer valid), you can pass --resources-to-skip to make CloudFormation stop trying to roll those resources back and mark the rollback complete. That unblocks the stack, but it leaves the skipped resources drifted from the template, which you must then fix manually.

Common causes

• A resource was deleted or modified outside CloudFormation, so there is no way to restore its recorded previous state.
• The previous configuration of a resource is no longer valid (an AMI that was deregistered, a subnet that was removed, an instance type no longer offered).
• A dependency the rollback needs no longer exists.
• CloudFormation lacks a permission required to perform the revert on some resource.
• A custom resource or Lambda-backed resource failed or timed out during the rollback.
• A database or stateful resource cannot be reverted because the change was not reversible.

How to fix it

STABILIZE

1. Confirm the state and identify which resources are stuck. Read the stack events and find the rollback failures and their reasons:

aws cloudformation describe-stack-events \
  --stack-name my-stack \
  --query 'StackEvents[?ResourceStatus==`UPDATE_ROLLBACK_FAILED`].[LogicalResourceId,ResourceStatusReason]' \
  --output table

DIAGNOSE

1. Read the ResourceStatusReason for each stuck resource. This tells you whether the resource can be reverted at all. If the reason is a transient issue (a permission that has since been fixed, a dependency now available), a plain retry may succeed. If the reason is that the resource cannot return to its prior state, you will need to skip it.
2. Try the rollback again with no skips first. Often the original blocker was transient and a clean retry finishes the rollback:

aws cloudformation continue-update-rollback --stack-name my-stack

3. If the same resource fails again, note its exact logical ID. That is the resource you will skip, and the one you will have to reconcile by hand afterward.

FIX

1. Continue the rollback, skipping only the logical IDs that genuinely cannot be reverted. Skip as few as possible:

aws cloudformation continue-update-rollback \
  --stack-name my-stack \
  --resources-to-skip StuckResourceLogicalId AnotherStuckId

2. Wait for the rollback to reach UPDATE_ROLLBACK_COMPLETE, which is a healthy, updatable state:

aws cloudformation wait stack-rollback-complete --stack-name my-stack

3. Reconcile every skipped resource. This is the step people forget, and skipping without it is what causes the next deploy to break. For each skipped logical ID, inspect the real resource, then either manually restore it to match the template or update the template to match reality. Detect the gap with drift detection:

aws cloudformation detect-stack-drift --stack-name my-stack
aws cloudformation describe-stack-resource-drifts \
  --stack-name my-stack \
  --stack-resource-drift-status-filters MODIFIED DELETED

4. Only once the skipped resources are reconciled should you attempt the intended update again.

How to prevent it

• Never modify CloudFormation-managed resources outside CloudFormation, since out-of-band changes are the most common reason a rollback cannot revert.
• Deploy updates through change sets so you review exactly what will change before executing.
• Give the CloudFormation execution role enough permission to revert every resource type it manages, so rollbacks do not fail on access.
• Run drift detection regularly so a stack does not silently diverge from its template before an update forces a rollback.
• Keep stateful resources (databases, buckets) with careful update and deletion policies so a failed update cannot corner you into an unrevertable rollback.

When to call a senior engineer

Call for help when a rollback keeps failing on a stateful or production-critical resource, when skipping would drift resources you cannot safely reconcile alone, or when the stack holds databases or data stores where a wrong recovery step risks data loss. Erzon engineers can complete the rollback with the minimum safe set of skips, reconcile the drifted resources so the template and reality match again, and get the stack back to a state your team can deploy to without fear.