For those running PostgreSQL on Kubernetes with an operator such as CloudNativePG, Zalando, or Crunchy, how do you confirm that your backups are genuinely restorable? A completed backup does not necessarily mean recovery will work.
I manage several CloudNativePG clusters and have tested recovery in a small k3s environment using CloudNativePG 1.30, the barman-cloud plugin, and S3. I create a temporary cluster from the newest backup, run a few SQL checks, measure how long it takes to become ready, and then remove it. So far, a few hundred megabytes restore successfully in about a minute.
How do you handle this in practice? Do you run restore checks manually, with a scheduled job, in CI, or through a dedicated tool? What do you validate afterward: cluster and pod health, expected tables and row counts, timestamps, application queries, or something more thorough? I'm also interested in how common restore testing is, including cases where teams do not test regularly.
4 Answers
We used to restore only during disaster-recovery exercises, roughly every six months. That proves the process works occasionally, but scheduled automated restores are a better safety net because they catch expired credentials, broken storage access, incompatible configuration, and changes in the recovery procedure much earlier.
We create a new CloudNativePG cluster pointing at the S3 backup location and run that process weekly from a scheduled job. After the cluster is ready, we run a few validation queries and then delete it.
One useful check is a table that receives frequent inserts. We verify that its newest timestamp is close to the expected recovery point—for us, within roughly five minutes of the latest WAL or point-in-time target. That gives us more confidence than checking only whether PostgreSQL started. Trying to compare every row in the database would be excessive for most environments.
We run a restore job immediately after a backup completes. It brings up a temporary PostgreSQL instance from the backup, performs basic health checks and representative queries, and tears the instance down afterward. That catches problems with both the backup contents and the recovery procedure without leaving test databases running.
For smaller databases, a temporary PostgreSQL container is enough. I restore the backup with Docker Compose and compare the schema and important data against production. It is not a substitute for a full disaster-recovery exercise, but it is a cheap way to verify that the backup can be opened and that the expected data is present.
That matches my concern. The restore itself is easy to automate, but validating the recovered data and deciding how much of the application to exercise seems like the more important design choice.