I want to back up database dumps, shared files, and configs using SCP, rsync, or SFTP. How can I stop a compromised source machine from deleting or changing old backups, and reliably tell whether each run succeeded?
How can I keep Linux file-copy backups safe from a compromised source?
Answers
Let each source push to its own restricted directory, then have the backup server take ZFS or btrfs snapshots after successful runs. Keep snapshot creation, deletion, and retention under backup-server control, not accessible to the source credentials.
A compromised source can still corrupt or delete its live backup directory, and future snapshots may capture that damage. Older snapshots give you a way back as long as the attacker cannot administer them. Separate accounts and permissions per source also limit access between hosts.
Reply to this answer
Pull with rsync over SSH from the backup server instead. The source then needs no credentials for modifying the backup destination, and you can keep schedules and failure reporting in one place.
Restrict the SSH key on each source to the required rsync command and the data it needs to read. For point-in-time copies, rsync's link-dest option can make unchanged files share disk space between backup generations. Protect those generations from modification on the backup server.
_d58!portnotch.7:
Hard-linked files share the same contents, so changing a file in place in one generation can change it in the others too. Treat completed generations as read-only.
Reply to this answer
If the scripts are already getting complicated, look at Restic, Kopia, or rsnapshot. They offer backup features around repeated runs and recovery that you'd otherwise have to assemble yourself.
Still check the repository permissions and retention setup. A backup tool is not automatically protected from a source that can delete its stored history.
Reply to this answer
For monitoring, capture the transfer command's exit status and alert on failures. Rsync has distinct exit codes, including one for source files that vanished during transfer, so decide how your monitoring should handle that case.
Also test restores. A successful copy does not prove a database dump is usable or that ownership and permissions will come back as expected. A non-root receiver may need extra care to preserve or restore ownership.
Reply to this answer
One permissions detail: writing a file and deleting its directory entry are different operations on Linux. File permissions govern writing its contents, while directory permissions govern removing or renaming entries. That distinction can help with design, but a writable current copy is still not a substitute for protected backup history.
Reply to this answer
Add your answer
Answers are reviewed before publication.
p.962+canvasvalve:
Check that snapshot retention leaves enough history to catch damage that goes unnoticed for a while. A successful transfer alone does not mean the source data was healthy.