Incident response runbook template
By the Erzon engineering team · Last updated July 13, 2026
TL;DR
- A runbook’s purpose is to remove decision-making from the middle of an incident: roles, severity, and comms are decided in advance, in writing.
- The core sequence is always the same: detect, triage, mitigate, resolve, review. Mitigation (stopping the bleeding) comes before diagnosis of the true fix.
- One person is Incident Commander and does not touch the keyboard; someone else fixes, someone else communicates.
- Severity levels exist to set response speed and comms cadence, not to assign blame or drama.
- Copy the template below, fill in the bracketed fields, and rehearse it once before you need it.
When to use this
Use this template if your team’s current incident process is “whoever notices it, panics first.” That works until the first incident that lasts more than an hour, involves a customer-facing outage, or hits while the one person who knows the system is asleep. Fill it in once (an hour of work), store it somewhere reachable when your infrastructure is down (not only in the wiki that runs on that infrastructure), and walk the team through it once.
The template covers roles, severity, communication, the step sequence, and escalation. A filled example follows so you can see what “done” looks like.
The template
1. Scope
This runbook covers production incidents affecting [service or product name]. It applies when [define your trigger, for example: “any user-facing functionality is degraded or down, or data integrity is at risk”].
2. Roles
Assign these at the start of every incident. One person can hold two roles in a small team, but the Incident Commander should not be the person typing fixes.
| Role | Responsibility | Default person | Backup |
|---|---|---|---|
| Incident Commander (IC) | Runs the incident: assigns tasks, tracks state, makes the call on risky actions, decides severity | [name] | [name] |
| Operator | Hands on keyboard: investigates and applies changes | [name] | [name] |
| Communicator | Posts updates internally and to customers on the agreed cadence | [name] | [name] |
3. Severity levels
| Level | Definition | Response time | Update cadence |
|---|---|---|---|
| SEV1 | [e.g. full outage, data loss in progress, or security breach] | [e.g. immediate, page on-call] | [e.g. every 30 min] |
| SEV2 | [e.g. major feature degraded, significant subset of users affected] | [e.g. within 30 min, business hours page after hours] | [e.g. every 60 min] |
| SEV3 | [e.g. minor degradation, workaround exists] | [e.g. next business day] | [e.g. daily until resolved] |
Rule: when unsure between two levels, pick the higher one. Downgrading later is free; upgrading late is not.
4. Communication
- Internal channel: [e.g. #incident-response in Slack]. All incident discussion happens here, not in DMs. The channel is the record.
- Status page / customer comms: [tool or page]. The Communicator posts within [X minutes] of a SEV1/SEV2 being declared. Say what is affected, that you are on it, and when the next update comes. Do not speculate on cause.
- Who may speak to customers: [role or names].
5. The sequence
Step 1: Detect and declare. Whoever notices, posts in [internal channel]: what they saw, when, and any error text. Declaring an incident is always the right call; a false alarm costs five minutes.
Step 2: Triage. IC assigns roles, sets severity, and states the current known impact in one sentence. Capture the start time.
Step 3: Mitigate. Stop the bleeding before finding the cause. Standard mitigations for this system, in order of preference:
- [e.g. roll back the most recent deploy]
- [e.g. fail over to replica / secondary region]
- [e.g. enable maintenance mode / feature flag off]
- [e.g. scale up the affected tier]
Before any risky action (restores, failovers, schema changes): take a snapshot or backup of the current state, even the broken one. Broken state you preserved can be repaired; broken state you overwrote cannot.
Step 4: Resolve. Once impact is contained, find and apply the real fix. Confirm resolution against the user-facing symptom, not just the internal metric. IC declares the incident resolved with a timestamp.
Step 5: Review. Within [e.g. 3 business days], hold a blameless review and write the report. Use a structured format such as our root cause analysis report template (/resources/root-cause-analysis-report-template).
6. Escalation
Escalate when: [e.g. no mitigation after 30 minutes on a SEV1, data integrity in question, or the fix requires expertise the team lacks].
| Situation | Escalate to | Contact |
|---|---|---|
| [e.g. database corruption, restore needed] | [internal expert or external partner] | [phone/email] |
| [e.g. infrastructure/cloud provider issue] | [provider support tier] | [support link, account ID] |
| [e.g. security incident] | [person/firm] | [contact] |
7. After the incident
- RCA report written and shared
- Action items filed with owners and due dates
- This runbook updated with anything learned
Filled example
A compressed example for a small SaaS team, so the fields feel concrete:
Scope: Production incidents affecting the Acme app (app.acme.example) and its API. Roles: IC: on-call engineer. Operator: second engineer (or IC if solo, in which case comms cadence drops to hourly). Communicator: CTO. SEV1: app down or data loss in progress. Respond immediately, update customers every 30 minutes. Mitigations in order: roll back last deploy via
deploy rollback; flip read-only mode flag; fail over Postgres to replica (requires IC approval, snapshot first). Escalation: database corruption or a failed restore goes to our external database partner within 30 minutes, not after 4 hours of solo attempts.
How to use this
Copy the template section into your wiki and a printable doc, replace every bracketed field, delete rows that do not apply, and add your system’s real mitigations to Step 3. Then spend 30 minutes running a tabletop drill: pick a plausible failure and walk the sequence out loud. Every gap you find in the drill is one you will not find at 3 a.m.
If you are mid-incident right now and the runbook does not exist yet, that is the situation Erzon exists for: send us the symptoms and you will have a senior engineer’s first response within one business hour, with a flat quote at triage.
Questions on this
What is an incident response runbook?
It is a short written procedure your team follows when production breaks: who does what, how severity is decided, who gets told and when, and the sequence from detection to review. Its job is to remove decisions from the worst possible moment to make them.
How detailed should a runbook be?
Detailed enough that a mid-level engineer can run an incident at 3 a.m. without asking anyone what to do next, and short enough that people actually read it. One to two pages for the core procedure is the right target; system-specific recovery steps live in separate per-service runbooks.
How often should we update the runbook?
After every real incident, as part of the post-incident review. If the team improvised something during the incident that worked, it goes in. If a step in the runbook was wrong or skipped, fix it or delete it. A runbook nobody has touched in a year is a runbook nobody trusts.