The integration that silently stopped

The situation

A logistics company ran on a quiet integration: orders captured in one system flowed automatically into the fulfillment system that actually picked, packed, and shipped them. It had worked without attention for years, which is exactly why nobody was watching it.

Then it stopped. Orders were still being placed and still being taken, but none of them reached fulfillment. There were no errors in the logs, no failed jobs, no alerts. Every dashboard was green. The first sign anything was wrong came from a customer asking where their delivery was, and by the time the team went looking, three full days of orders were sitting unfulfilled.

What we found

An upstream API had deprecated an authentication flow the integration relied on. On schedule, the integration tried to refresh its access token, and that refresh started failing against the retired auth scheme.

A failing token refresh should be loud. This one was silent, because of how the code handled errors. A broad catch-all wrapped the sync, caught the authentication exception, logged nothing useful, and let the job report itself "complete." The sync finished successfully every time, having done nothing. This is the most dangerous shape of failure: not a system that breaks, but a system that lies politely, reporting success while quietly doing no work.

Because success was self-declared rather than measured, every layer of monitoring above it believed the good news. Nothing was watching whether orders actually arrived at the other end.

The fix

Recovery came before repair, and repair came before prevention.

First, the missing orders had to be safe. The event log, which recorded every order as it was placed, was confirmed intact and the affected three-day window frozen so nothing could overwrite it. That log was the reason full recovery was even possible.

Second, the token flow was rebuilt against the upstream provider new authentication scheme, and the sync was verified end to end: an order placed now genuinely arrived in fulfillment, confirmed by looking at the destination rather than trusting a "complete" message.

Third, the three days of missed orders were replayed from the event log into fulfillment, in order, with de-duplication so nothing shipped twice. Every order was recovered.

Finally, the failure mode itself was closed. The catch-all that swallowed errors was replaced with handling that surfaces and alerts on failure, and a heartbeat check was added that measures the integration by results, whether orders are actually landing in fulfillment, not by whether the job claims to have finished.

The outcome

The backlog cleared the same day, and because recovery moved faster than word of mouth, most customers were notified before they noticed anything was wrong.

The report left behind one principle worth more than the fix: never trust a system that reports success without evidence. A green checkmark that nobody can trace back to real work is not reassurance, it is a blind spot waiting for its moment.