I currently use restic to back up each server to an R2 bucket every hour. The credentials on each server are limited to reading and writing objects in that bucket, but the server also needs the restic repository password. If a server is compromised, an attacker could potentially run restic forget or otherwise destroy the repository, leaving me without backups.
I'm considering using restic's rest-server with append-only mode, but I'm not sure whether that fully addresses the problem. I'd like to understand how others design their backup systems, especially around preventing deletion, retaining multiple recovery points, and regularly verifying that backups can actually be restored.
4 Answers
Start with recoverability rather than just backup creation. You should regularly restore backups in an isolated environment and track whether the restore succeeds, how long it takes, and whether the resulting data passes validation. Once you know your recovery objectives, you can choose the retention, storage, permissions, and tooling that support them.
Use immutable retention on the object-storage side. R2 Object Lock with compliance-mode retention can prevent objects from being deleted or overwritten until the retention period expires, even if the server credential is compromised. Keep the retention period long enough to cover the time it might take to notice an incident, such as 30 to 90 days, and use a separate trusted process for cleanup after the lock expires.
That’s stronger than relying only on restic append-only mode. Repository maintenance can still require deletion privileges, so it’s better to isolate pruning and other destructive operations on a separate, tightly controlled host.
A common pattern is to make the source server able to write only to a limited ingest area, while the backup server handles snapshots and retention. For example, maintain a current copy, create daily and weekly snapshots from the backup side, and keep older weekly or monthly versions for a longer period. If the source is compromised, it should not have access to delete the historical snapshots.
Restic is a solid option because it provides encryption and retention, but the exact setup depends on what you’re backing up. Stateless services may only need infrastructure definitions and persistent data backups, while databases often have their own backup tooling that handles consistent snapshots better. Whatever tool you choose, use separate credentials for writing backups and for pruning, and keep at least one copy protected by immutability or otherwise isolated from the source server.
That makes sense. A backup that has never been restored is really only an assumption, so I’ll add scheduled recovery tests and measure the results.