A Debian 13 virtual machine running PostgreSQL under a heavy write workload suddenly stopped. It runs on OpenStack with KVM and Cinder, likely backed by Ceph. Both the root volume and a separate data volume use ext4, and both now report corrupted superblocks. The VM had to be manually restarted through OpenStack, but audit and recovered journal logs do not show anyone intentionally changing or shutting down the system. The cloud support team says they have found no infrastructure problems. Much of the data still appears intact: listing and extracting files with 7-Zip works, but fsck and other recovery attempts have not made either filesystem mountable. What kinds of failures could corrupt two separate virtual disks during the same incident, and what should be investigated next?
3 Answers
Two volumes becoming unusable during one crash points more toward a shared layer than two unrelated ext4 failures. Investigate the guest and hypervisor I/O path, including libvirt cache settings and whether the disks use writeback or unsafe caching. A hard failure while metadata is still being cached could affect several virtual disks at once. Safer modes such as cache=none or directsync may reduce performance but avoid acknowledging writes that have not safely reached storage. The exact cache configuration and failure behavior need to be confirmed by the provider.
Do not keep experimenting on the original volumes. Take provider-level snapshots or cloned volumes first, work from copies, and use filesystem-specific recovery tools rather than treating 7-Zip extraction as proof that the filesystem can be repaired. Since both devices were affected, preserve logs and timestamps before rebuilding. Check PostgreSQL crash recovery, the guest kernel log, SMART or host storage diagnostics if available, and whether the VM experienced resets or lost I/O. Backups and replicas are essential, but the repeated-failure risk should still be investigated with the platform team.
There are several possible layers involved: a guest kernel or filesystem bug under heavy concurrent writes, a VM or hypervisor crash, a problem in the virtual disk caching path, or an OpenStack/Ceph issue. If the backend really is Ceph, ordinary storage corruption is less likely because writes are normally acknowledged only after the required replicas have persisted them, but that does not rule out problems in the guest, host, network, or virtualization layer. Ask the provider for hypervisor, Cinder, and Ceph event logs covering the exact timestamp, including resets, I/O errors, failed volumes, and host evacuation activity.
By “died,” I mean the VM was left down and OpenStack had to be instructed to start it manually; the automatic restart did not work.