The complete guide to production incident response

TL;DR

• Incidents are decided before they start: teams with a runbook, tested restores, and rehearsed rollback resolve in minutes what costs unprepared teams a day.
• The sequence is fixed: detect, triage, mitigate, resolve, review. Mitigation (stop the impact) always precedes hunting the true cause.
• One person commands, one operates, one communicates. Updates on a stated cadence beat perfect information delivered late.
• Never destroy state to fix state: snapshot before restores, failovers, or anything irreversible.
• Every incident ends with a blameless written review and action items with owners, or you have paid for a lesson and not collected it.

What actually happens when production breaks

Every production incident, whatever the technology, moves through the same five stages: detection, triage, mitigation, resolution, and review. The technology varies (a corrupted database, a wedged deploy, a memory leak), but the shape of a well-handled incident does not. This guide walks the full arc, from the preparation that happens weeks before to the prevention work that happens after, and is written for the people accountable for the outcome: founders, CTOs, and the engineers holding the pager.

One framing up front. The cost of an incident is mostly determined by two numbers: how long until you notice (detection time) and how long until users stop feeling it (mitigation time). Almost everything below is in service of shrinking those two numbers.

Preparation: the incident before the incident

The best incident response happens weeks earlier, in calm conditions. Four things separate teams that resolve incidents in minutes from teams that lose a day:

A written runbook. Roles, severity levels, comms cadence, and the mitigation options for your specific system, decided in advance so nobody is inventing process at 3 a.m. If you do not have one, our incident response runbook template (/resources/incident-response-runbook-template) is a fill-in version you can complete in an hour.

Tested recovery paths. A backup you have restored, a rollback you have executed, a failover you have watched work. Untested recovery paths fail at a remarkable rate, and they fail at the worst moment by definition. The full pre-flight list is in our production readiness checklist (/resources/production-readiness-checklist).

Monitoring on symptoms users feel. Error rate, latency, and the one transaction that means your business works. CPU graphs tell you a machine is busy; a checkout success metric tells you the company is losing money. Alert on the second kind.

Decided ownership. Who is on call, who can approve a risky action, who talks to customers. In a five-person team this takes ten minutes to decide and it still needs deciding.

Detection: finding out before your customers do

There are only three ways you learn about an incident: your monitoring tells you, an employee stumbles on it, or a customer reports it. The third is the most expensive, because it means the impact predates your awareness by however long customers hesitated before complaining, which is usually longer than you think.

Two practical rules. First, check uptime from outside your own infrastructure, so the incident that takes down your network does not also take down your awareness of it. Second, make declaring an incident cheap and blameless. The engineer who says “this looks wrong, declaring an incident” and is wrong has cost five minutes. The engineer who hesitates for an hour because declaring feels dramatic has cost an hour of impact. Reward the first behavior explicitly.

Triage: thirty seconds of discipline

When the incident is declared, resist the reflex to immediately start fixing. Spend the first minutes answering three questions:

1. What is the user-visible impact? Down, degraded, or at risk? Which users, which functionality?
2. Is it getting worse? A static failure and a spreading one (disk filling, corruption propagating to replicas, a queue backing up) call for different urgency.
3. What changed recently? Deploys, config changes, migrations, dependency updates, traffic shifts. Most incidents trace to a recent change, and asking now saves hours of undirected log spelunking.

Then assign roles: one Incident Commander who runs the incident and keeps their hands off the keyboard, one or more operators who investigate and act, one communicator. In a two-person company these collapse, but the IC function (someone tracking the whole, making the risk calls) must exist even if it is the same brain.

Set the severity, because severity sets the tempo:

Severity	Typical definition	Response	Customer comms
SEV1	Outage, active data loss, or security breach	Immediate, all hands as needed	Within 15 to 30 minutes, updates every 30
SEV2	Major feature degraded, significant user subset affected	Within 30 minutes	Within an hour, updates every 60 minutes
SEV3	Minor degradation, workaround exists	Business hours	Usually none, or release notes

When torn between two levels, choose the higher. Downgrading is free.

Mitigation versus fix: stop the bleeding first

This is the single most consequential decision discipline in incident response. Mitigation stops the impact; the fix addresses the cause. They are different actions on different clocks, and mitigation comes first.

Rolling back the deploy, failing over to a replica, turning off the feature flag, putting the site in read-only mode, scaling up the tier that is drowning: none of these fix anything, and all of them can end user impact in minutes. The genuine fix (the code change, the corruption repair, the schema correction) can then proceed without a burning clock, which also makes it a better fix. Engineers under outage pressure write worse code than engineers who mitigated first.

Two safety rules while mitigating:

Never destroy state to fix state. Before a restore, a failover, or any irreversible action, snapshot what you have, even the broken version. A corrupted database that you preserved can be repaired or mined for the missing rows; one you overwrote with a restore cannot. This rule is the difference between a bad day and an unrecoverable one, and it is the most common mistake we see teams make under pressure. (If you are facing that specific situation, see our guide to recovering a corrupted production database at /resources/recover-corrupted-production-database.)

Prefer reversible actions. Between two mitigations of similar effect, take the one you can undo. A feature flag beats a hotfix; a rollback beats a forward patch written in a panic.

If mitigation is not working after a defined interval (30 minutes on a SEV1 is a reasonable default), escalate: to the internal expert, the cloud provider, or an outside specialist. The expensive failure mode is the team that grinds solo for six hours out of pride before making the call they should have made in the first thirty minutes.

Communication: the part customers actually remember

Customers forgive outages far more readily than they forgive silence. The communication bar is low and most companies still miss it:

• Acknowledge early. A status page or email within the first 30 minutes of a SEV1, saying what is affected and that you are actively on it. You do not need the cause to say this.
• Commit to a cadence and keep it. “Next update by 14:30” and then an update at 14:30, even if the update is “still investigating, mitigation in progress.” A kept cadence reads as competence; a broken one reads as chaos.
• State impact and remedy, not guesses about cause. Early cause theories are wrong often enough that publishing them creates corrections later. Say what users experience and what to do meanwhile.
• Close the loop. When resolved, say so, and for significant incidents follow up with a plain-language summary of what happened and what you changed. This is where a good written review pays for itself twice.

Internally: one channel, everything in it, no side DMs. The channel becomes the timeline for the review, for free.

Resolution: confirming it is actually over

The incident ends when the user-facing symptom is gone, verified directly, not when the internal graph looks better. Load the page, run the transaction, check the error tracker for new entries. Then watch for regression: a fault that was mitigated by a restart or a rollback can return, and the second occurrence should be caught by you, not customers. If the resolution involved a workaround rather than a true fix, say so explicitly and track the real fix as an open item; workarounds that quietly become permanent are how systems accumulate mystery.

Note the resolution timestamp. Time to detect, time to mitigate, and time to resolve are the three numbers worth tracking across incidents, because they tell you where to invest: long detection means monitoring work, long mitigation means recovery rehearsal, long resolution means debugging capability (for that last one, our guide to debugging production bugs at /resources/debugging-production-bugs covers the diagnostic side).

The review: collecting the lesson you paid for

Within a few business days, hold a blameless review and write it down. The format matters less than the discipline, but a consistent structure (summary, impact, timeline, root cause, contributing factors, action items with owners and dates) makes reports comparable and findable; our root cause analysis report template (/resources/root-cause-analysis-report-template) is a ready-made one.

Two things make reviews actually work. First, blamelessness as policy, not politeness: if the conclusion is a person’s name, dig one layer deeper to the safeguard that was missing, because people-shaped conclusions produce hidden failures next time, not fewer failures. Second, action items that are real work: owner, due date, checked at 30 days. Teams with binders of thoughtful, unexecuted RCAs have the same incidents on rotation, annually.

Prevention: turning incidents into reliability

Each incident, reviewed properly, retires one failure mode. Over a year that compounds into a visibly calmer system, but only if the action items favor structural fixes over vigilance. “Be more careful with migrations” prevents nothing; “migrations run expand/contract and are rehearsed in staging” prevents a category.

Where to aim the prevention budget: most teams’ incidents cluster in a handful of well-known patterns (expired certs, half-finished deploys, full disks, untested failover), and we cataloged the usual suspects in the most common DevOps failures (/resources/common-devops-failures). Run the production readiness checklist annually, because safeguards decay: the backup job that silently broke in March is discovered in November, one way or the other.

The honest summary: incident response is a small, learnable discipline, and almost all of it is decided before anything breaks. A one-page runbook, a restore you have actually performed, a rollback you have rehearsed, and a comms cadence you have agreed on will carry a team through incidents that would otherwise cost days.

And when an incident outgrows the team, whether that is a corrupted database, an infrastructure failure nobody can trace, or a bug that only exists in production, Erzon exists for exactly that moment: send the symptoms, get a senior engineer’s first response within one business hour, a flat quote at triage, and a written root cause report with every fix.