The 7 most common DevOps failures (and how to fix them)

TL;DR

Most infrastructure incidents aren’t exotic, they’re the same seven failures on rotation: expired certificates, half-finished deploys, disks quietly filling, pipelines nobody trusts, autoscaling that can’t actually scale, cloud bills growing unattended, and monitoring that watches the wrong things. Each has a known fix and a cheap prevention. If your production system exhibits three or more of these patterns, you don’t have bad luck, you have deferred maintenance.

1. The expired certificate

The failure: The site, an API, or an internal service stops cold. Browsers warn, integrations 500, and the cause is a TLS certificate that expired at the least convenient hour. Special mention: the internal cert (service-to-service, database TLS, webhook endpoint) nobody remembered existed.

The fix: Renew and deploy the cert; restart or reload whatever pinned the old one. Then inventory every certificate in the system, including the internal ones, because they were probably issued the same week.

The prevention: Automate renewal (Let’s Encrypt/ACME or your cloud’s managed certs) and alert at 21 days to expiry on everything with a cert. Expiry is the single most preventable outage in this list.

2. The half-finished deploy

The failure: A deploy fails partway. Some instances run new code, some run old; a migration applied but the rollback didn’t account for it. Production enters a state that is neither version, and the errors look like three unrelated bugs.

The fix: Don’t push forward blind and don’t blind-rollback either. Establish which components are on which version, check whether the database schema moved, then converge deliberately, usually completing the migration while patching the app to tolerate both states.

The prevention: Deploys must be atomic-ish and reversible: health-gated rollouts that halt on failure, backward-compatible migrations (expand → migrate → contract), and a rollback path you have actually rehearsed.

3. The disk that filled up

The failure: Everything degrades at once, the database won’t write, logs stop, sometimes the box won’t boot. Cause: logs, temp files, images, or database bloat quietly consuming disk for months, finishing at 3 a.m.

The fix: Free space carefully, compress or move logs off-box; do not delete database WAL/binlog files to make room, that’s how disk-full incidents become corruption incidents. Then find the growth source, not just the symptom.

The prevention: Log rotation with enforced retention, and disk alerts on trend (“full within 14 days at current rate”), not just threshold (“95% now”, which fires at 3 a.m., by tradition).

4. The pipeline nobody trusts

The failure: CI has been red or flaky for weeks, so the team ships around it, deploys by hand, tests skipped “just this once.” The pipeline’s whole purpose (catching the bad deploy) is gone precisely when it’s needed.

The fix: Triage the failures into real bugs, flaky tests, and broken infrastructure. Fix the infra, quarantine (don’t delete) the flaky tests with owners and deadlines, and get to a trustworthy green, a smaller honest pipeline beats a bigger lying one.

The prevention: Treat a red main branch as an incident, not wallpaper. Track flake rate; a pipeline the team routinely overrides is training everyone to ignore the alarm that matters.

5. Autoscaling that can’t actually scale

The failure: Traffic spikes, a launch, a feature, a good day, and the “elastic” infrastructure buckles. New instances crash-loop on boot, or every added web server just deepens the pile-on against a fixed-size database connection pool. Scaling the web tier scaled the pressure, not the capacity.

The fix: Find the real bottleneck under load, it’s usually the database, a connection pool, or a boot-time dependency, and fix that: cache the hot reads, size the pools, remove runtime dependencies from the boot path.

The prevention: Load-test at a multiple of your best day, not your average one. Autoscaling you’ve never watched work is a hypothesis, not a capability.

6. The runaway cloud bill

The failure: The invoice is 2 to 3x last quarter and nobody can say why. Usual suspects: forgotten environments running 24/7, unbounded log ingestion, NAT gateway and egress charges, over-provisioned databases “to be safe,” and orphaned volumes and snapshots accreting for years.

The fix: Get cost visibility per service (tagging, cost explorer), rank the top ten line items, and act on them: kill the zombies, right-size the instances, cap the log retention, put lifecycle policies on storage. The first pass on a never-audited account routinely finds 20 to 40% of the bill.

The prevention: A monthly 30-minute cost review with a budget alert on anomalies. Cost is a metric like latency, unwatched, it only trends one way.

7. Monitoring that watches the wrong things

The failure: The pager screams about CPU blips and disk-of-a-dev-box, but the actual outage, checkout failing, sync silently stopped, arrives via a customer email. Alert fatigue means the one page that mattered was swiped away with the noise.

The fix: Rebuild alerting around user-visible symptoms: can users log in, check out, load the dashboard? Synthetic checks on the money paths, alerts on error-rate and latency budgets, and delete every alert that has never led to an action.

The prevention: One rule: every page must be actionable, every critical user journey must page. Audit quarterly against the incidents you actually had.

The pattern behind all seven

None of these failures is exotic, and that’s the point: most downtime isn’t caused by hard problems, it’s caused by known problems left unattended, then discovered at the worst hour. The fix for each fits in an afternoon. The incident each one causes rarely does.