Fix: Segmentation fault (core dumped)

Segmentation fault (core dumped) means the process tried to read or write memory it has no right to touch, and the kernel killed it with SIGSEGV. The process is gone, along with whatever it was serving. The good news is in the second half of the message: a core dump was written, which is a full snapshot of the process at the instant of the crash. One backtrace from that dump usually names the exact function, and often the exact line, that crashed.

Segmentation fault (core dumped)

What this error means

The CPU’s memory management unit enforces which pages of memory a process may access. Dereferencing a null or dangling pointer, writing past the end of a buffer, overflowing the stack with runaway recursion, or calling into a corrupted structure all land on an invalid address, the MMU faults, and the kernel delivers SIGSEGV. The default action is to terminate the process and, if core dumps are enabled, write its memory image for post-mortem debugging. A segfault is always a bug, in your native code, in a library you load, or occasionally in mismatched binaries, and it is never something to restart-loop past silently, because memory corruption can damage data before it crashes.

Common causes

• Null or dangling pointer dereference in C or C++ code, including use-after-free where memory is touched after being released.
• Buffer overflows: writing past array bounds, corrupting adjacent memory that fails later somewhere unrelated.
• Stack overflow from unbounded recursion or oversized stack allocations.
• A crashing native extension inside a managed runtime: Python C extensions, Node native addons, or JNI libraries.
• ABI or version mismatches: a plugin compiled against different headers than the binary that loads it.
• Race conditions in threaded code corrupting shared structures, producing crashes that move around between runs.

How to fix it

STABILIZE first. Let the supervisor restart the service to restore availability, but keep the crashed state: do not deploy anything or clean up until the core dump is secured, because it is your only evidence.

systemctl status myservice
coredumpctl list

1. Confirm dumps are being captured. If core_pattern points at systemd-coredump, use coredumpctl; if dumps are disabled, enable them so the next crash is not wasted.

cat /proc/sys/kernel/core_pattern
ulimit -c unlimited

For a systemd service, allow cores in the unit:

[Service]
LimitCORE=infinity

2. DIAGNOSE by opening the dump and reading the backtrace. Install the debug symbols for your binary and its libraries if the trace shows only addresses.

coredumpctl gdb <pid-or-name>

Inside gdb:

bt full
info registers
thread apply all bt

3. The top frames of the backtrace name the crashing function. A frame in your code is the direct lead; a frame in a library usually means your code handed it a bad pointer or freed memory, so look one or two frames up the stack for the caller.

4. FIX the code. For memory bugs that resist reading, reproduce under the address sanitizer, which reports the illegal access with the allocation and free sites the moment it happens:

gcc -g -fsanitize=address -o myapp myapp.c
./myapp

For crashes inside a managed runtime, update or rebuild the native extension named in the trace against the exact runtime version, and check its issue tracker for the crash signature.

5. Verify the fix under the same load and input that produced the crash, not just a smoke test, since memory bugs are input dependent.

How to prevent it

• Run C and C++ test suites under AddressSanitizer and UndefinedBehaviorSanitizer continuously, so memory bugs fail in CI instead of production.
• Ship binaries with debug symbols available (or a symbol server), so production cores are readable immediately.
• Keep core dump capture enabled and monitored: alert on every SIGSEGV, because a crash rate of one is already a bug.
• Pin and rebuild native extensions together with runtime upgrades, never mix prebuilt binaries across versions.
• Prefer memory-safe languages for new components where performance allows.

When to call a senior engineer

Call for help when the backtrace points into memory corruption that happened long before the crash, when the fault moves between runs and smells like a race, or when a vendor library crashes and you need proof of where the bad pointer came from. Erzon engineers debug core dumps at the assembly level when needed, isolate corruption with sanitizers and watchpoints, and turn a recurring mystery crash into a specific fixed bug.