A Debian 13 VM running PostgreSQL under heavy write load went down, and both its root and separate data volume now appear to have corrupted ext4 superblocks. The VM runs on OpenStack with KVM and Cinder, possibly backed by Ceph.
Guest logs haven't shown an obvious cause, and filesystem repair hasn't made the volumes mountable. Backups and a replica exist, but what should be investigated to understand whether this could happen again?
Start with the shared path, not just the two filesystems. Separate virtual volumes can still depend on the same hypervisor, storage service, network, or physical hardware. A failure in any of those could affect both at once.
Ask the cloud team for the host and storage events around the crash, including whether the hypervisor restarted or lost connectivity. A clean guest journal doesn't rule out a failure below the guest, especially if the VM stopped abruptly.
Check the virtual disk cache settings and the full flush path with the provider. If writes are acknowledged before they're durable, a host failure can leave both volumes missing recently acknowledged filesystem metadata.
That doesn't prove caching caused this. The cache mode alone isn't enough to establish what happened, and the actual storage backend and its configuration matter too. Ask how guest flushes are handled all the way through to storage.
p.406!teaclip:
For that scenario, the key event would be failure below the guest, such as a host crash. Get the provider to check the host timeline rather than assuming the VM itself crashed first.
Corruption on both volumes doesn't automatically mean Ceph or ext4 is at fault. If Ceph is involved, ask for its health and error history at the time, not just its current status. Also check for storage network problems and errors in the virtualization stack.
Some would consider a storage fault less likely if the backend correctly acknowledges durable writes, but that depends on the actual configuration. There isn't enough here to rule it out.
A filesystem or kernel bug under load is another possibility, as is faulty host memory. Neither is established by the symptoms, though. Preserve the available logs and get host diagnostics before rebuilding anything that might contain evidence.
For a replacement VM, sending logs to a separate server could make the next failure easier to investigate if local logs stop abruptly.
Since backups and a replica are available, use them for recovery rather than relying on further repair attempts against the only damaged copies. Keep copies or snapshots of the affected volumes for investigation, and verify the restored database before putting it back into service.
Answers are reviewed before publication.
m.53+zipperhook:
It would also help to pin down what went down. A guest crash and a hypervisor crash point the investigation in different directions.