I currently run restic on each server and push backups to an R2 bucket every hour. The credentials are limited to that bucket and allow object reads and writes, but the server also has the repository password. If the server is compromised, an attacker could potentially run restic forget or otherwise delete the backups.
I'm considering using restic's append-only mode through rest-server, but I'm not sure whether that provides enough protection. I'd like to design the backup strategy from first principles, including retention, credential isolation, immutable storage, and reliable recovery testing. How do you manage backups for your servers, and how do you make sure they remain recoverable after a compromise?
4 Answers
Start with recovery rather than just backup creation. Regularly restore backups into a test environment and track whether the restore succeeds, how long it takes, and whether the resulting data is valid. Those recovery objectives should drive the retention, storage, and verification design.
Another pattern is to make the server capable only of writing new backup data, while all deletion, pruning, and retention maintenance runs from a separate trusted system. Historical snapshots can be copied into daily, weekly, or monthly retention tiers. This limits what an attacker on the source server can destroy, though immutable storage is still preferable because retention schemes alone depend on detecting the compromise before the old copies age out.
If the workload is genuinely stateless, the simplest backup strategy may be rebuilding the servers from infrastructure-as-code and restoring only persistent data. That can eliminate the need to back up disposable system state, but databases and other irreplaceable data still need their own tested backup plan.
Restic is a reasonable choice because it provides encryption and retention policies. For protection against a stolen server credential, use storage-side immutability as well. R2 Object Lock with compliance-mode retention can prevent objects from being deleted until the retention period expires, even if the token is compromised. Choose the retention window based on how quickly you expect to detect an attack.
That separation makes sense: the compromised server should be able to add backups but not remove the older recovery points, while cleanup happens elsewhere after verification.