My Docker host was upgraded from Debian 12 to Debian 13 and worked normally until a reboot. Afterward, Docker would not start properly because the root filesystem was full. Most of the disk usage is under /var/lib/docker: overlay2 initially used about 22 GB, while containers used around 73 GB. However, docker system df reported only 269.7 MB used by containers, with nearly nothing reclaimable. Removing dangling images freed roughly 2 GB, and stopping a newer UniFi-related stack released another 17 GB, allowing more containers to start. Further investigation showed that container logs were consuming about 53 GB. What is the safest way to clear those logs and configure Docker to limit their size in the future?
3 Answers
Commands such as docker system prune can remove stopped containers and other unused resources, while docker volume prune can remove unused volumes. These only reclaim a small amount in this case and should be used carefully because unused volumes may contain data you still need. Moving Docker's data-root to a separate, larger filesystem is another option, but it does not solve runaway logs by itself; log rotation and fixing the noisy container are still important.
The large difference between the container directory size and docker system df is most likely JSON container logs. First identify the biggest files with a command such as du -ah /var/lib/docker/containers | sort -h | tail. If they are Docker JSON logs, you can empty them without deleting the containers by truncating the relevant files, for example: truncate -s 0 /var/lib/docker/containers/*/*-json.log. Be careful to target only the log files and verify the paths before running the command. Also investigate which container is producing so much output, since a restart loop or repeated error can refill the disk quickly.
Configure Docker's JSON logging driver in /etc/docker/daemon.json, creating the file if it does not exist: { "log-opts": { "max-size": "10m", "max-file": "3" } }. Then restart Docker. The settings generally apply to newly created containers, so recreate the containers or stacks afterward rather than assuming existing containers have picked up the limits. Choose retention values that fit your troubleshooting needs.
Stopping the newer UniFi stack recovered about 17 GB, so it appears to be generating a lot of output. I also found roughly 53 GB of logs and am checking which container is responsible.