Production readiness checklist

TL;DR

• Production readiness is testable, not aspirational: every item below has a pass condition you can demonstrate, not just assert.
• The highest-stakes items are the recovery ones: a restore you have performed, a rollback you have rehearsed, a failover you have watched work.
• Monitor user-facing symptoms (errors, latency, checkout success), not just host metrics; alert only on things a human should act on.
• Run the checklist before launch, before big traffic events, and annually; readiness decays quietly.
• Copy the checklist, check what you can prove today, and treat every unchecked box as a queued incident.

When to use this

Run this before you trust a system with real users or real money: a launch, a migration, a big marketing push, or an inherited system you are now responsible for. It is grouped into database, infrastructure, and application, because those are the three places production actually breaks, and each item is phrased so that “done” means demonstrated, not intended.

Honest scoring matters more than a perfect score. “We have backups” and “we restored one last month in 40 minutes” are different claims; only the second one helps you during an incident.

The checklist

Database and data

• Backups run automatically on a schedule matching how much data you can afford to lose ([e.g. hourly incrementals, daily full]), and a failed backup pages someone.
• A restore has been performed in the last 90 days to a scratch environment, and it produced a working database, not just a completed job. Time to restore: [X minutes/hours], and it fits inside your downtime tolerance.
• Backups are stored off the primary infrastructure (separate account/region), so the incident that takes production down cannot take the backups with it.
• Point-in-time recovery is enabled (WAL/binlog archiving) if losing a full day of data is unacceptable.
• A replica or failover target exists, and failover has been exercised at least once, by [name], with steps written down.
• Disk space and connection counts are alerted on trend, not just at 95%.
• Destructive access is limited: production credentials are not the same as staging, and ad-hoc write access to prod data goes through a review step or a break-glass process.

Infrastructure and DevOps

• A deploy can be rolled back in under [10] minutes, and someone has actually done it, including with a database migration in play (expand/contract pattern or a written exception).
• Deploys are health-gated: a bad deploy stops itself before reaching all instances.
• All TLS certificates are inventoried (including internal, service-to-service ones) and renewal is automated or alerted at 21 days.
• Secrets live in a secrets manager, not in code, dotfiles, or a shared doc; rotating one credential does not require a code deploy; the last person who left the company triggered a rotation.
• Infrastructure is reproducible: you could rebuild the environment from code/config in a known number of hours, not archaeology.
• Capacity has been load-tested at [2 to 3]x your best day, and the first bottleneck under load is known and written down.
• Autoscaling (if any) has been observed actually scaling, including whether the database and connection pools survive it.
• Log retention and rotation are enforced, and logs from all services are searchable in one place during an incident.

Application and monitoring

• User-facing symptoms are monitored: error rate, latency, and the one transaction that means the business works ([e.g. checkout success, signup completion]).
• Every alert that pages is actionable: it has a linked runbook or an obvious next step, and the on-call person has not muted it. Alerts nobody acts on have been deleted or downgraded.
• Uptime is checked from outside your infrastructure, so you find out before customers tweet.
• Errors are aggregated with stack traces (error tracker, not just logs), and new error types are surfaced within minutes of a deploy.
• On-call exists and is survivable: a named rotation (even if it is two people), reachable escalation, and an incident runbook they have read. If you do not have one, start from our incident response runbook template (/resources/incident-response-runbook-template).
• Dependencies fail gracefully: the app degrades rather than dies when a third-party API, cache, or queue is down, verified by turning one off in staging.
• Rate limits and timeouts are set on inbound and outbound calls, so one slow dependency cannot exhaust every worker.
• A staging environment exists that resembles production closely enough that a migration or deploy rehearsed there means something.

Example: a filled line, honestly scored

The difference between checked and actually checked:

• A restore has been performed in the last 90 days. Evidence: J.M. restored the June 3 nightly to db-scratch on June 5; 34 minutes to a queryable database; row counts verified against prod for the 4 largest tables. Gap found: restore docs referenced a bucket that no longer exists (fixed).

That “gap found” line is the entire point of the exercise. Nearly every first run of this checklist finds one safeguard that silently stopped working.

How to use this

Copy the checklist into your wiki. For each item, either check it with a line of evidence (who demonstrated it, when) or leave it unchecked and add it to a remediation list, ranked by blast radius: unrecoverable data loss first, extended downtime second, slow response third. Re-run it after major changes and at least annually. Expect the first pass on a system that has never been audited to leave a third of the boxes unchecked; that is normal, and much cheaper to discover now than mid-incident.

If the unchecked boxes are piling up faster than the team can clear them, that is what Erzon’s reliability retainer is for, and if one of them has already turned into a live incident, send us the symptoms: triage is free and you will have a first response within one business hour.