My Docker host was upgraded from Debian 12 to Debian 13 and worked normally until a reboot. Afterward, Docker would not start properly because the root filesystem was completely full. Most of the space is under /var/lib/docker: overlay2 initially used about 22–34 GB, while containers appeared to use roughly 73 GB.
However, docker system df reported only 269.7 MB for containers, 19.13 GB for images, and very little reclaimable space. Removing dangling images freed only about 2 GB. Stopping one recently added service with docker compose down released another 17 GB, allowing more containers to start.
Further investigation showed that around 53 GB is being used by container log files. What is the safest way to remove or truncate those logs, determine which container is producing them, and configure Docker log rotation so this does not happen again?
3 Answers
Commands such as docker system prune mainly remove unused images, stopped containers, networks, and build data. They will not solve a disk filled by logs from running containers. Volume pruning is also risky because it removes unused volumes, so only use it after confirming that the data is not needed. For a longer-term setup, you can move Docker's data-root to a larger filesystem, but that only relocates the problem unless log rotation is configured too.
Configure Docker's json-file logging driver in /etc/docker/daemon.json, creating the file if it does not exist: {"log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}. Restart Docker after saving it. These defaults generally apply when containers are created, so recreate existing containers or explicitly configure logging in your Compose files; restarting alone may not change the settings on containers that already exist. Also investigate the noisy service, especially if it is repeatedly restarting or logging an error loop.
The large difference between the container size reported by docker system df and the space used under /var/lib/docker/containers is probably JSON log files. First identify the biggest ones, for example with: find /var/lib/docker/containers -name '*-json.log' -printf '%s %pn' | sort -n. You can safely empty an active JSON log with truncate -s 0 /path/to/container-json.log, which releases the space without removing the container. Be careful not to delete the container metadata directories themselves.
Stopping the newer service released about 17 GB, and checking the files showed that roughly 53 GB was in logs. I am now working through the largest files and setting up rotation.