Debugging a production bug: a business owner's guide

TL;DR

You don’t need to read code to run a good bug hunt, you need to run it like an incident. Establish business impact first, preserve evidence before it rots, ask “what changed?”, insist on reproduction before fixes, and measure progress by narrowing (what’s been ruled out), not by hours spent. Mitigate now, root-cause after. And know the tells that it’s time to bring in senior help.

Step 1, Size it in business terms first

Before anyone touches a keyboard, answer: who is affected, doing what, costing how much? “Checkout fails for ~20% of mobile users since Tuesday” is a workable incident. “The app is buggy” is not.

This isn’t bureaucracy, it sets urgency honestly (a typo can wait; silent payment failures cannot), and it defines done: the bug is fixed when that number returns to zero, not when a developer feels optimistic.

Step 2, Preserve the evidence

Production evidence rots fast: logs rotate, dashboards age out their high-resolution data, and the instinct to “just restart it” destroys the crime scene.

Have your team capture, now:

• Logs from the affected window, exported somewhere safe
• Timestamps of first occurrence and every occurrence since
• Exact error messages and screenshots from real users, verbatim, not paraphrased
• Affected examples: which users, which orders, which requests

If the system is down and must be restarted to restore service, snapshot what you can first. Ten minutes of preservation routinely saves days of “we can’t tell anymore.”

Step 3, Ask the highest-value question: “What changed?”

Most production bugs that start on a particular day were caused by something that happened that day. In rough order of likelihood:

• A deploy (yours)
• A dependency, API, or third-party change (someone else’s deploy)
• A config, DNS, or certificate change
• A data milestone, some table or usage pattern crossed a size where the old approach stopped coping
• An expiry, token, cert, trial of a service someone forgot was load-bearing

“Nothing changed” is almost never true; it usually means “nothing changed that we track.” If the answer is a recent deploy and the bug is severe: roll back first, investigate second. Restoring service beats winning the argument.

Step 4, Insist on reproduction before fixes

The single most expensive pattern in production debugging is fixing by guesswork: someone plausible-sounding ships a “likely fix,” everyone waits, the bug returns, repeat. Each cycle burns a day and pollutes the evidence.

The discipline to hold your team to:

1. Reproduce it, reliably trigger the bug, in staging or production
2. Explain it, the story of why it happens, consistent with all the evidence
3. Fix it, the change that breaks that story
4. Prove it, the reproduction now passes, and the business metric from Step 1 recovers

If it can’t be reproduced yet, the correct next step is making it observable, targeted logging or tracing around the suspect path so the next occurrence identifies the culprit. That’s real progress. “We shipped another guess” is not.

Step 5, Measure progress by narrowing

You can manage a debugging effort without understanding the code by asking one question in every check-in: “What have we ruled out since last time?”

Good answers sound like: “It’s not the payment provider, their logs show our requests never arrive. It’s in our outbound path.” That’s a search space cut in half. Bad answers sound like busy motion: “We’re still looking into it.” Hours spent is not a metric; hypotheses eliminated is.

Two useful management rules:

• Timebox theories. Any single theory gets a few hours, not a week. Sunk cost is the debugger’s disease.
• Separate mitigation from root cause. It’s fine, often right, to restore service with a workaround (roll back, disable the feature, restart nightly) while the investigation continues. Just don’t let the workaround quietly become the fix; the root cause will collect interest.

The traps

• The blame detour. Whose fault it is has zero diagnostic value. Teams afraid of blame hide evidence, exactly what you can’t afford.
• The heroic all-nighter. Debugging quality collapses with sleep. A tired engineer chasing a race condition at hour sixteen creates bugs faster than they fix them.
• “It fixed itself.” No, the trigger receded. Intermittent bugs that vanish unexplained return at scale, at the worst moment. Log it, instrument for it, keep the file open.
• Fixing symptoms serially. Three “unrelated” bugs in a week often share one root cause. If you’re playing whack-a-mole, stop and look for the mole machine.

When to bring in senior help

Call for experienced reinforcement when: the team is on its third “likely fix”; the bug involves the hard categories (race conditions, memory corruption, data integrity, anything that only reproduces under real load); the person who understood that code is gone; or the daily cost of the bug exceeds the cost of a specialist by lunchtime. Bringing in a senior engineer at day two is a good decision; at day ten, it’s the same decision with eight extra days of losses.