Windows can sometimes feel laggy or stutter even when CPU, memory, and disk usage appear low. I'd like to learn a systematic approach for diagnosing performance problems on Linux as a system administrator or power user. What should I check first, how can I identify whether the bottleneck is CPU, memory, storage, networking, or an application, and which tools are most useful for deeper investigation?
4 Answers
Start by figuring out what the system is waiting on instead of immediately changing settings. A practical first pass is `uptime` for load averages, `top` or `htop` for processes, `free -h` for memory, `vmstat 1` for CPU and memory activity, `iostat -xz 1` for storage, `df -h` for filesystem space, and `journalctl` plus `dmesg` for errors. On newer kernels, also check `/proc/pressure/cpu`, `/proc/pressure/memory`, and `/proc/pressure/io`; PSI can expose contention that ordinary utilization percentages miss. If the system looks healthy, investigate the application, network latency, DNS, external services, or database performance.
Learn `top` or `htop` thoroughly rather than treating them as dashboards. Understand load average, CPU states, memory and swap usage, process states, and how to sort by different columns. Also compare the machine against its normal behavior: record disk activity, memory pressure, and other useful metrics over time so you can recognize sudden changes. High memory usage is not automatically a problem because Linux uses spare memory for caching; look for reclaim pressure, swapping, or actual application impact.
Once the broad system checks point toward a particular process, go deeper with tools such as `strace`, `perf`, `iotop`, `iftop`, or `atop`. `strace` can show whether a process is blocked on files, sockets, or other system calls, while `perf` and async-profiler help locate CPU hotspots. Tools such as `tuned-adm` can apply known performance profiles, but they are presets rather than a substitute for measuring the actual bottleneck. Brendan Gregg’s performance material is also a strong way to learn how these tools fit together.
For a more structured approach, use the USE method: for each major resource—CPU, memory, storage, and network—check utilization, saturation, and errors. For example, high load with low CPU utilization may indicate I/O wait or another form of contention, which `vmstat` and `iostat` can help locate. `sar` is especially useful because it keeps historical performance data, making it possible to investigate incidents that happened hours or days ago.
Historical data is incredibly helpful for incidents that are already over. A few days of `sar` records can show exactly what changed around the time a slowdown occurred.
That gives me a much better starting point. I’ll focus on identifying the resource being waited on instead of just collecting random metrics.