How can I identify which filesystem is making the server slow?
Asked by
MellowCedar42
My server has become very sluggish: `df -h` takes several seconds, system load is high, but CPU usage remains low. I suspect a filesystem problem, especially because several directories are mounted over NFS. `mount -a` completes successfully and the mounts appear to be available, but even simple commands such as `ls` or changing into those directories can take a couple of seconds. What's the best way to determine which filesystem or mount is causing the delay, and how can I investigate the underlying problem?
Answers
`atop` is another useful option. If a block device is struggling, it can highlight the device in red, making it easier to see whether the delay is coming from local storage rather than an NFS mount.
Reply to this answer
Check disk and I/O wait as well. Tools such as `iostat` can show whether a local block device is spending most of its time waiting on operations. High I/O wait with low CPU usage is a common sign that the system is blocked on storage or a network filesystem.
Reply to this answer
NFS is the most likely culprit here. Since `df` checks every mounted filesystem, test them one at a time with commands such as `df /var` or another known local path, then compare that with each NFS mount. You can also run `strace df` and look for a `statvfs` call that takes a long time. In my case, this identified the problematic NFS mount. It wouldn’t unmount normally because something still had it open, so I used `umount -l` to detach it and then mounted it again. Once the NFS share was reconnected, the server responded normally.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowCedar42: That approach helped me find the slow NFS filesystem. A normal unmount reported that it was busy, but a lazy unmount with `umount -l` worked. After running `mount -a` to reconnect it, the machine became responsive again. I’m still trying to determine what caused that share to slow down in the first place.