Fix: Got error 28 from storage engine

Error 28 means a disk MySQL relies on is full. The number 28 is the operating system’s errno for ENOSPC, No space left on device, which MySQL passes straight through when a write fails. This is one of the few database errors that is genuinely an infrastructure problem rather than a query or schema problem: some filesystem, either the data directory or the temporary directory, has no free space, so writes cannot complete. Because it can stop all writes, it often takes the database down.

ERROR: Got error 28 from storage engine

What this error means

MySQL writes to disk constantly: table data and indexes go to the data directory, and transient work like large sorts, GROUP BY aggregations, and implicit temporary tables spills to the temporary directory (tmpdir, often /tmp). When any of those filesystems runs out of space, the write returns ENOSPC and MySQL reports error 28. The two common cases look identical in the error but need different fixes: the data volume filling up (from data growth, binary logs, or undo growth) versus the tmpdir filling up during a single heavy query. You confirm the meaning with perror 28 and locate the full mount with df.

Common causes

• The data directory volume is full from steady data growth, large tables, or unrotated binary logs.
• Binary logs accumulating because binlog_expire_logs_seconds is unset or too long.
• A big query spilling to tmpdir (often /tmp on a small root filesystem) with a large sort or temp table.
• Undo tablespace or the InnoDB redo logs growing on a tight data volume.
• A separate process on the same host (application logs, backups) consuming the shared disk.
• A filesystem that is out of inodes rather than out of bytes, which also surfaces as ENOSPC.

How to fix it

STABILIZE first. The database cannot write, so the priority is finding the full mount and freeing space safely before anything else.

1. DIAGNOSE: confirm error 28 is ENOSPC and see every mount’s free space.

perror 28
df -h
df -i

perror 28 prints No space left on device. df -h shows byte usage and df -i shows inode usage; check both.

2. DIAGNOSE: identify which mounts MySQL uses so you know whether the datadir or tmpdir is the problem.

SHOW VARIABLES LIKE 'datadir';
SHOW VARIABLES LIKE 'tmpdir';

Cross reference those paths against the full filesystem from df.

3. STABILIZE: free space safely. Purge old binary logs through MySQL, never with rm, and clear OS logs and old backups outside the data directory.

PURGE BINARY LOGS BEFORE DATE_SUB(NOW(), INTERVAL 3 DAY);

Do not hand delete files inside the data directory. Removing .ibd, ibdata, or binary log files directly will corrupt the database.

4. FIX the tmpdir case: if tmpdir sits on a small filesystem, point it at a volume with room and restart MySQL.

[mysqld]
tmpdir = /var/lib/mysql-tmp

Ensure the directory exists and is owned by the mysql user, then restart:

sudo systemctl restart mysql

5. FIX the capacity case: if the data volume is simply too small, expand the disk (grow the cloud volume and the filesystem, for example with growpart and resize2fs) or move the data directory to a larger volume.

6. FIX recurrence: set binary log retention so logs cannot silently fill the disk again.

SET GLOBAL binlog_expire_logs_seconds = 259200;

7. Verify MySQL can write again and the mounts have headroom.

df -h

CREATE TABLE _erzon_write_check (id INT);
DROP TABLE _erzon_write_check;

How to prevent it

• Alert on disk usage for both the datadir and tmpdir mounts well before they reach full, for example at 80 percent.
• Set binlog_expire_logs_seconds so binary logs rotate automatically instead of growing unbounded.
• Put tmpdir on a volume sized for your largest sorts and temp tables, separate from a tiny root filesystem.
• Monitor inode usage as well as byte usage, since many small files can exhaust inodes first.
• Add index and query tuning so large sorts do not need to spill huge temp files in the first place.
• Keep backups and application logs off the database data volume.

When to call a senior engineer

Call for help when the disk fills repeatedly, when you are unsure which files are safe to remove, or when MySQL will not start after a disk full event and you suspect corruption. A senior engineer can safely reclaim space, resize storage without data loss, and find the query or retention gap that keeps filling the disk. Erzon can get a down database writing again and put the monitoring in place so a full disk never surprises you twice.