The quiet months

The situation

This one is different from the emergencies, because nothing was on fire. A founder-led business ran a single production system that customers depended on. There was no senior engineer on staff. The founder was the on-call, the escalation path, and the person who would be woken up if it broke.

On the surface everything was fine. Underneath, the founder carried a low-grade dread that never quite went away: the backups were assumed to be working rather than known to be, nobody had ever tried to restore one, and every deploy felt like a small act of courage. That is the state a reliability retainer exists for, the calm before a first serious incident that has not happened yet.

What we found

The first month of the retainer was a baseline review, and it surfaced four latent emergencies, every one of which was quietly on a path to becoming a real outage.

A backup job had been failing silently for six weeks. The founder believed there were backups; in reality there were none from the last month and a half, and the day that mattered would have been the day they learned it.

A disk was trending toward full within the quarter. Left alone, it would have filled at some unpredictable moment and taken the system down hard, most likely under load.

Two TLS certificates were set to expire inside thirty days. An expired certificate is one of the most common and most preventable outages there is, and these had nobody watching them.

A slow-query pattern was sitting just under the timeout threshold. At roughly twice the current data volume, which the business was on track to reach, it would have started timing out and degrading the product for everyone.

The fix

None of these were dramatic to fix. That is the point. Each one is a manageable Tuesday-afternoon task and a catastrophic Saturday-night emergency, and the retainer is the difference between the two.

The backups were repaired and, crucially, proven with a real restore drill, bringing a backup up on a separate machine and confirming the data was actually there. "We have backups" and "we can restore" are different sentences, and only the drill settles which one is true.

Alerting was set up on symptoms that matter rather than noise, so the disk trend, certificate expiries, and query latency would raise a flag with room to act, not a page after the fact. Patching and certificate renewals were brought onto a monthly cadence. The slow-query pattern was tuned before users ever felt it.

The outcome

This is the pattern a reliability retainer is for: every finding was a future emergency, caught on a Tuesday afternoon instead of a Saturday night. None of these four became an outage, because none of them got the chance.

The founder now gets a monthly reliability summary, a plain-language account of what was checked, what was found, and what was handled. The low-grade dread was replaced by something closer to confidence, and the on-call phone stopped feeling like a fuse.