The launch that melted the infrastructure

The situation

A SaaS company got the break they had chased for a year: a feature in a publication their entire market reads. Traffic climbed twelvefold inside an hour. This was supposed to be the best day in the company history.

Instead the site went down and stayed down. The autoscaler was configured and did try to help, spinning up new instances to meet demand. But every new instance died within minutes of starting, so the fleet never grew. From the outside it looked like the infrastructure had simply melted under load. From the inside, it was worse than that: the system was actively fighting its own recovery.

What we found

Two failures were compounding, and neither was raw capacity.

The first was a boot-time dependency. On startup, each new instance fetched its configuration from an internal service. Under normal load that service answered instantly. Under the spike it was itself overwhelmed, so new instances hung waiting for config, tripped their startup health check, and were OOM-killed before they ever served a request. The autoscaler dutifully replaced them, and the replacements hit the same wall. That is a crash loop: the recovery mechanism feeding the outage.

The second was the database connection pool. It had been sized for the old, smaller fleet. The few instances that did manage to stay alive competed for a pool far too small for the traffic, so they starved each other of connections and slowed to a crawl. Even a healthy instance could not do useful work.

Underneath both, two read endpoints were taking roughly eighty percent of all read traffic and hitting the database directly on every request, with nothing in front of them.

The fix

The priority was to break the crash loop, because until new instances could survive boot, nothing else could help.

The boot-time config fetch was removed from the critical path by baking configuration into the machine image. Instances now start with everything they need and depend on no other service to come up. That single change turned the crash loop into a fleet that actually grew when the autoscaler asked it to.

With instances staying alive, the database connection pool was right-sized for the new fleet so instances stopped starving one another. Then a cache was placed in front of the two endpoints carrying eighty percent of read traffic, which took enormous pressure off the database at a stroke.

Finally, rather than let the autoscaler slam the whole fleet up at once against a cold cache and a recovering database, capacity was brought back in controlled waves, warming the cache and letting the database settle between steps.

The outcome

The site was serving traffic again the same day and held steady at the new twelvefold load, on what turned out to be the biggest day in the company history after all.

The more durable result was that the near-miss became real capacity. Because the boot path no longer depended on anything at runtime and the hot endpoints were cached, the system was verified to hold at twenty times the old baseline. The next spike stopped being a threat and became a number they had already tested past.