How to roll back a bad deploy safely

TL;DR

• Decide fast: if users are impacted and the last version was healthy, roll back first and diagnose afterward. Fixing forward under pressure is how one incident becomes two.
• Rollback restores your code, not your data. The number one rollback killer is a schema migration the old code cannot run against.
• Make every migration backward compatible by one release (expand and contract), and never bundle “migrate” and “deploy” as one irreversible unit.
• Blue/green gives you instant, boring rollback; canaries shrink the blast radius so most rollbacks affect 5 percent of traffic instead of 100.
• Before the lever: confirm the target version, its config, and schema compatibility. After: watch error rate, latency, and one real user flow, then say clearly that the incident is mitigated, not solved.

Step 1: Make the rollback decision quickly

The most expensive part of most bad deploys is not the bug, it is the twenty minutes spent debating. Use a default:

Roll back when users are impacted, the previous version was healthy, and you do not know the root cause yet. All three are usually true.

Fix forward when rollback is unsafe (a destructive migration already ran, data written in a new format the old code cannot read) or when the fix is trivially small and verifiable, like reverting one config value or flipping a feature flag.

Situation	Default action
Error rate or latency spiked right after deploy	Roll back now
Bug is behind a feature flag	Turn the flag off, keep the deploy
New release already ran a destructive migration	Fix forward, carefully
Bad config or secret, not bad code	Fix the config, not the deploy
”It’s probably the deploy but we’re not sure”	Roll back anyway; it is the cheapest experiment you can run

One person owns the call. During an incident a deploy is guilty until proven innocent, and rolling back a deploy that turns out to be innocent costs you almost nothing.

Step 2: Understand the trap before you need to

Rollback replaces the application binary. It does not replace the database, queues, caches, or object storage. So the real question is never “can we redeploy the old version” but “can the old version operate against the current state of the world?”

The classic failure: release 42 renames a column as part of the deploy, release 42 has a bug, you roll back to release 41, and release 41 crashes on boot because the column it queries no longer exists. Now you have two incidents and the second one is worse.

The fix is a standing rule, not incident-time heroics:

1. Migrations are backward compatible by one release. The previous app version must run correctly against the current schema. Additive changes (new nullable columns, new tables) are safe. Renames, drops, and type changes are split across releases using expand and contract: add the new thing, dual-write, switch reads, and only drop the old thing after the release that reads it can never come back.
2. Migrations ship separately from the code that needs them, and the migration deploy is verified boring before the feature deploy goes out. “Forward-only migrations” is a fine policy exactly because each migration is trivial and compatible; it is a disaster as a policy layered on destructive migrations.
3. Data written by the new version must be readable by the old one for at least one release. New enum values, new serialization formats, and new queue message shapes all count as schema.

If you inherit a bad deploy whose migration was already destructive, do not roll the app back blindly. That is the moment to fix forward, or to involve someone who does database recovery for a living.

Step 3: Execute the rollback

The mechanics depend on your platform, but the shape is the same.

1. Freeze the pipeline. Stop other deploys and disable auto-deploy on merge, so the environment stops moving while you operate on it.
2. Identify the exact target. “The previous version” means a specific immutable artifact: an image digest, not a mutable tag like latest. On Kubernetes: kubectl rollout undo deployment/api or, better, kubectl set image to a pinned digest, then kubectl rollout status deployment/api.
3. Check the three compatibility questions before pulling the lever: Does the old version boot against the current schema? Does it have the config, secrets, and env vars it expects (config changes ride along with deploys more often than people think)? Are its API contracts still compatible with clients and dependent services that may have moved?
4. Roll back the same way you deploy. Through the pipeline or platform command, with health checks active. SSH-and-copy-files rollbacks create a snowflake state that the next deploy destroys.
5. Watch it land. Old pods healthy, new pods gone, no crash loops (kubectl get pods -w, kubectl logs --previous if anything restarts).

Blue/green: switch the router back to the still-warm previous environment. Rollback time is seconds, which is the entire argument for paying blue/green’s cost.

Canary: if the deploy is at 5 or 25 percent, “rollback” is routing the canary’s traffic back to stable. This is why canaries exist: most rollbacks stop being incidents and become non-events nobody outside the team hears about.

Step 4: Verify, then communicate

A rollback is a hypothesis (“the deploy caused it”) until the graphs agree.

• Error rate and latency back to the pre-deploy baseline, not just “improving”.
• One real user flow exercised end to end: log in, load the dashboard, complete a checkout. Health endpoints lie by omission.
• Background systems: queue depth draining, cron and consumers on the old version too (rollbacks that only cover the web tier are half rollbacks).
• If metrics do not recover, the deploy was innocent. Un-freeze your assumptions and widen the investigation: config, data, a dependency, a certificate.

Communicate in plain language at three moments: when you decide (“Rolling back the 14:02 deploy, expect recovery within 10 minutes”), when metrics confirm (“Error rates back to baseline as of 14:19, incident mitigated”), and afterward, with the distinction intact: mitigated is not resolved. The bad release is still bad, and it does not go out again until the root cause is understood, which is what the post-incident review is for.

When to call for help

Two situations justify outside help fast: a rollback that is blocked by an incompatible or destructive migration, and a rollback that completed but did not fix the symptoms. Both mean the incident is deeper than the deploy. Erzon takes production incidents from triage with a first response within one business hour, works inside your own repos and infrastructure with least-privilege access, and closes every incident with a written root-cause report and a prevention list, which for deploy incidents usually means making the next rollback a one-command non-event.