The kinds of fires we put out.

The situation. An e-commerce company ran a schema migration on a Friday deploy. It locked their orders table mid-write on a 40-million-row alter. Checkouts started timing out; the on-call developer's rollback attempt left the schema half-migrated and the ORM confused about reality.

What we found. The migration ran an ALTER TABLE that takes a full table lock on their MySQL version, fine in staging with 10k rows, fatal in production with 40 million. The rollback had dropped a column the new code still read.

The fix. Restored write availability first by patching the application to tolerate both schema states, then completed the migration online with a ghost-table approach during low traffic.

The outcome. Checkout restored, zero data lost, and a migration playbook in the root-cause report: online-DDL tooling, staging data at production scale, and a rule that schema changes never ride Friday deploys.

The situation. A SaaS team landed a press feature they'd waited a year for. Traffic went 12x in an hour. The site went down, and stayed down, because every autoscaling event was OOM-killed within minutes of starting.

What we found. Autoscaling was configured, but new instances pulled config from a service that was itself overloaded, so replacements booted into a crash loop. Meanwhile the database connection pool was sized for the old fleet, so surviving instances starved each other.

The fix. Broke the boot-time dependency (baked config into the image), right-sized the connection pool, put a cache in front of the two endpoints taking 80% of read traffic, and brought the fleet up in controlled waves.

The outcome. Serving traffic again the same day, stable at the new load. The prevention list turned the near-miss into capacity: load tests at 20x, a boot path with no runtime dependencies, and alerts on pool saturation.

The situation. A logistics company's orders stopped flowing into their fulfillment system. No errors, no alerts, the first signal was a customer asking where their delivery was. Three days of orders sat unfulfilled.

What we found. An upstream API had deprecated an auth flow; the integration's token refresh began failing, and a catch-all error handler swallowed the failure and reported the sync "complete." The classic silent failure: code that lies politely.

The fix. Restored the token flow against the new auth scheme, then replayed the missed orders from the event log, recovering all three days. Replaced the catch-all with failures that alert, and added a heartbeat check that measures the sync by results.

The outcome. Backlog cleared, customers notified before they noticed, and a monitoring principle in the report: never trust a system that reports success without evidence.

The situation. A founder-led business, one production system, no in-house senior engineer. Nothing was on fire, but the founder was the on-call, the backups were assumed rather than tested, and every deploy was a small act of courage.

What we found. First month: a backup job failing silently for six weeks. A disk trending toward full within the quarter. Two certificates expiring inside 30 days. A slow-query pattern that would have crossed the timeout threshold at roughly twice the data volume.

The fix. Fixed the backups and proved them with a restore drill. Set up alerting on symptoms, not noise. Handled patching and renewals monthly. Tuned the queries before users ever felt them.

The outcome. The pattern retainers are for, every finding was a future emergency, found on a Tuesday afternoon instead of a Saturday night. The founder's monthly reliability summary replaced the low-grade dread.