A 2024 Linux crisis-tools reference covers utilities ranging from procps to bpftrace, but one omission stood out to me: sos. In an outage scenario, a VM snapshot was restored before the team had preserved the system state. That brought the service back temporarily, but erased logs, command output, and other forensic evidence, so the underlying problem returned later.
I would normally run sos before making major changes. It collects logs, configuration, process information, network details, storage data, and other system state into one archive. That would preserve useful evidence without requiring the team to investigate a live production system under pressure. It is open source, available by default on many enterprise Linux distributions, and generally takes a single command to run.
Is sos something you include in your incident procedures? Are there other tools or practices you use to capture enough information before restoring, rebooting, or rebuilding a system?
4 Answers
sos is useful, but it serves a different purpose from the low-level observability tools in that reference. It is more of a broad troubleshooting and support-collection utility, and the list was intentionally minimal. Administrators still need to choose collection tools that fit their environment. It also is not completely universal: some distributions favor their own equivalents, such as supportconfig. In addition, probing a heavily loaded production system can sometimes add risk, so it is worth testing the collection process beforehand.
I learned about sos years ago and have used it regularly. It is excellent for troubleshooting, but I would not describe it as a recovery tool. Recovery is usually about restoring access or service; root-cause analysis comes afterward. Once an organization collects reports routinely, the next challenge becomes storing, classifying, and comparing them over time.
Important system data should ideally be sent to a remote collector continuously rather than relying only on a report created during an outage. If the guest may be compromised, preserve the affected snapshot or disk image for investigation. If the host is compromised, power it down when appropriate, acquire an image, and reprovision it instead of trusting the existing installation.
There are distribution-specific alternatives and related tools. supportconfig is commonly used on some enterprise distributions, while hotsos provides broad diagnostics on Ubuntu systems. The sos project itself has been around for years and is used for support cases on major Linux distributions, so checking its manual and testing it as part of normal operations is a good starting point.
Whatever tool you choose, make sure its output can be written somewhere safe and that the collection process is reliable on your actual node configurations. Reports are less useful if they fail during the exact kind of restart or outage you are trying to investigate.
That distinction makes sense. I tend to gather as much information as possible from the terminal before changing anything, because a rollback can remove the evidence needed to understand why the failure happened.