Our on-premises file server holds video production material, including raw 4K image sequences. During busy production periods, the data can grow by more than 2 TB per day. The raw material is eventually converted into much smaller mezzanine or delivery files, but the team often leaves the large source sequences in place while working.
We currently maintain several online and offline backup chains. The storage is expandable, but a new full backup with compression and encryption takes at least 48 hours, and the production team can add another couple of terabytes while it is running. The primary server is ZFS with about 65 TB of usable capacity and is roughly 70% full. Online backup storage is about 60 TB, while offline backups use rotating 20 TB disks. Keeping multiple full chains consumes a lot of space and can leave only one usable retention set during a new backup.
I'm trying to find a practical way to protect the raw footage without relying on users to remember to compress or archive it. I'm considering automated compression, proxy generation, shorter retention on primary storage, or moving older projects to tape or another archival tier. I'm hesitant to create a clearly marked no-backup area because people will eventually store important data there and expect it to be restored. How would you design this workflow?
5 Answers
Don’t make the editors manually compress everything in the middle of a job. Add an automated archive or media-management workflow that watches agreed project folders and handles proxy generation, mezzanine conversion, or archival copying in the background. A separate processing host is preferable so an editor’s workstation or overnight render queue is not responsible for the job.
The editors can continue working with the original material, while the system creates lower-resolution proxies for day-to-day editing and moves the raw material to a protected archive when the project reaches an agreed stage.
For video, tape or another sequential archive tier is often a better fit than keeping many full disk-based backup chains. Keep recent projects on disk for fast restores, then copy finished or inactive projects to two separate offline or off-site sets. Use short disk retention for the raw workspace and longer retention in the archive.
Document the distinction clearly: the archive is the recovery copy, while the working share is only for active projects. Also verify tape or disk restores regularly; an archive that has never been tested is not much of a backup.
The numbers suggest the backup design needs capacity planning as well as optimization. If a full backup takes 48 hours while the source grows faster than the backup can complete, adding more retention alone will not solve it. Consider faster storage and network paths, parallel backup streams, changed-block tracking, and separate policies for active projects, finished projects, and temporary renders.
Also leave more headroom on the ZFS pool and the backup target. Running a pool close to full can hurt performance, and needing to delete a backup chain before starting the next one leaves an uncomfortable recovery gap.
Protect the primary server against operational mistakes separately from the backup system. ZFS snapshots, a recycle-bin style delete workflow, and sensible quotas can provide quick recovery from accidental deletion without forcing the backup system to retain every temporary render and raw sequence indefinitely.
I would avoid a no-backup share unless its behavior is impossible to misunderstand and management explicitly accepts the risk. A better approach is to make all normal project space backed up, then automatically archive or expire data according to project status and age.
Treat the file server as active workspace, not the long-term archive. Set a defined retention period for completed projects, then move them to cheaper secondary storage or tape as soon as they stop changing. That reduces the amount of data that every backup chain has to process and keeps the primary storage from remaining near capacity.
For active work, use incremental or changed-block backups if your backup software supports them, rather than repeatedly creating large full copies. Compression and deduplication can help, but they will not make several terabytes of new, already-compressed video disappear.
Offloading that processing from the edit suites is important. A failed overnight conversion on a shared workstation is frustrating, whereas a dedicated queue can retry jobs, report failures, and keep the production machines available for actual editing.