I'm looking for a reliable way to recover from accidental deletions or corruption caused by an update—not just protection against disk or node failure. One of my applications uses SQLite along with static files, so a complete backup would need an online SQLite backup plus a copy of the related files.
When running the app in Docker, I used a host cron job that accessed the bind-mounted data, created the SQLite backup, archived the files, and uploaded everything to remote storage with rclone. Now I'm moving to Kubernetes and want storage to be managed through PVCs, potentially using Longhorn as the cluster grows. Should the backup run as a Kubernetes CronJob, a sidecar container in the application pod, or through a backup tool designed for PVCs? I'd also like to ensure that the database and files represent a consistent point in time.
3 Answers
A Kubernetes-native backup tool may be a better fit than writing a custom scheduler for every application. Tools such as K8up can use restic and either create a Job that accesses a PVC or execute an application-provided backup command inside a pod and capture its output.
For databases such as PostgreSQL or MySQL, use their application-aware backup tools. For SQLite and custom file layouts, you still need an export or script that understands how the application’s data fits together. A generic volume snapshot or archive can protect files, but it may not produce a transactionally consistent database backup.
For an app-specific backup, a sidecar can be a practical choice. Mount the same PVC, run the SQLite backup and file-copy script there, and upload the completed archive to remote storage. This avoids depending on a host cron job or on whichever node currently runs the application.
A separate Kubernetes Job can also work, but the volume’s access mode matters. With ReadWriteOnce, another pod may need to run on the same node as the application. With ReadWriteOncePod, a second pod cannot mount the volume at the same time, so a sidecar is usually simpler.
Make sure the SQLite backup and file archive are consistent with each other. If the application can change both during the backup, temporarily pause writes or use an application-level export that coordinates the database and files. Prevent overlapping runs, treat the backup as successful only after the remote upload finishes, and monitor the age of the last successful backup from outside the application pod. Most importantly, regularly restore a backup to a fresh PVC and test the application rather than only checking that the archive can be extracted.
Keep the responsibilities separate: version Kubernetes manifests in source control, use database-native backups for database services, and back up object or file storage independently. For a custom SQLite application, the backup target is the application’s data rather than the Kubernetes cluster itself.
If possible, designing applications to keep durable files in object storage and state in a managed or centralized database reduces the number of custom PVC backup workflows. When that isn’t possible, an application-aware script running through a sidecar or backup Job is still a reasonable solution.
I’m focused on backing up application data, not the cluster configuration. The manifests should already be versioned separately; the difficult part is protecting the SQLite database and files owned by each application.
That’s close to what I was considering. I wanted to compare the different approaches because I used host-based scripts with Docker, but I’d rather avoid tying backups to a particular Kubernetes node. A centralized database and object storage would simplify things, although not every self-hosted application supports that.