Our on-premises file server stores video production material, including raw 4K footage and image sequences. During active projects, the data can grow by more than 2 TB per day. The raw material is often compressed later into mezzanine formats that save roughly 30–60% of the space with an acceptable loss of quality, but the production team does not always do this promptly.
We currently maintain seven full backup chains across online on-premises storage and offline off-site disks. Storage capacity can be expanded, although the amount of duplicated data is becoming difficult to manage. A new encrypted and compressed full backup takes at least 48 hours, and production data continues growing while the backup is running.
The primary file server is a 65 TB ZFS system that is about 70% full. Online backup storage is 60 TB, while offline backups use rotating 20 TB disks. The online backup currently holds about two full chains and is around 95% full, so starting a new backup often requires deleting one chain first and temporarily reducing retention.
I am trying to encourage the team to compress or archive material earlier, but that is easy to ignore when they are focused on a project. I have also considered creating a clearly marked workspace that is excluded from backups, but I expect people would eventually store important files there and ask for them to be restored. What storage, archiving, media-management, or workflow approaches would you recommend for handling this kind of rapidly growing video data?
5 Answers
The current capacity leaves very little room for a safe backup cycle. If a full chain takes 48 hours and the source grows continuously, the backup target needs enough headroom to hold the next chain without deleting the previous one first. Reducing retention may be necessary, but for long-term video archives, rotating tape or larger removable storage is usually more practical than keeping every full chain on expensive online disks.
Keep multiple independent copies, including at least one offline or otherwise isolated copy, and periodically perform restoration tests. RAID or ZFS redundancy helps the primary server survive hardware failures, but it does not protect against deletion, corruption, or ransomware.
Do not make compression dependent on the editors remembering to run a desktop application. Put an automated archive or media-processing service beside the storage system. It could watch agreed project folders, create proxies or mezzanine files, verify them, and then move the original raw material to cheaper storage after a defined period.
A media asset management system would make this much safer than relying on manually organized folders. If a full MAM is too expensive, a well-defined folder structure plus scheduled jobs can still provide most of the benefit. Offloading the processing also keeps the editing workstations available for production.
Use proxies for editing and archive the camera originals or raw sequences separately. Editors can work with lower-resolution files while the high-resolution material is moved to tape or another durable archive tier. Keep the relationship between the proxy and original explicit, and test that the originals can actually be restored before deleting anything from primary storage.
A no-backup share is risky unless it is technically isolated, clearly labeled, and governed by an agreed lifecycle. A better option is to give every project a temporary workspace with a fixed retention period and automatic warnings before cleanup. Anything that must be retained should be moved into a protected project or archive area.
Also consider protecting the file server against accidental deletion with snapshots, recycle-bin behavior, or a server-side trash area. Those measures improve operational recovery, but they are not a substitute for independent offline backups.
Treat the live file server and the backup system as separate problems. Keep only the material that needs immediate editing on primary storage, then move completed or inactive projects to a secondary archive much sooner. Shorter retention on the active workspace, combined with a proper archive tier, will reduce the amount each backup job has to process.
Also verify that the backup software is using efficient block-level incrementals, deduplication, and compression. Compressing video is not always useful—many finished video formats are already compressed—but raw image sequences can benefit greatly from an automated conversion or proxy workflow.
The source material includes large raw TGA sequences. We have tested converting them to H.264 and it saves substantial space without materially affecting the final output, but the editors often leave 500 GB or more of raw frames on the server because they are busy.
That would also avoid the common problem where someone starts a conversion in a graphical editing application overnight and discovers the next morning that the queue failed halfway through.