We have 3 to 4 PB of legacy archives on older media that need to be retained for compliance. Our normal network connection would be a bottleneck, and we need a way to find individual files later without restoring everything.
How would you plan the ingest and avoid surprises from temporary storage, request fees, and retrieval costs?
At this size, bring the AWS account team in before committing to a transfer design. They can help check capacity, current service availability, transfer options, and the cost assumptions for the target region.
Compare a dedicated connection with AWS DataSync or Data Transfer Terminal where available, plus any suitable physical bulk-ingest providers. Physical transfer may get around the network bottleneck, but you still need a plan for reading the old media, validating what was copied, and getting it into the desired storage class.
Object count matters as much as total bytes here. If the archive contains millions of small files, PUT requests, lifecycle transitions, and per-object archival overhead can add up. Packaging related small files into larger archive objects can cut the number of requests. Compress where the data actually benefits from it.
The tradeoff is retrieval granularity. If one small file is needed later, you may have to restore its larger package. Choose package boundaries around likely retrievals and keep a manifest mapping each original path to its package. Test the packaging and restore process before scaling it up.
Don't assume everything has to land in S3 Standard first. S3 objects can be uploaded with the Glacier Deep Archive storage class specified. Whether a particular migration tool or physical-ingest service supports that destination directly is a separate question, so check its workflow.
If the chosen route does stage in Standard, model how long the data stays there and include lifecycle transition requests. Estimate costs from both total bytes and object count, then run a representative pilot and compare the actual bill with the estimate.
Deep Archive fits data that can wait for recovery, but compliance data still needs a retrieval plan. Budget for restore requests, retrieval, temporary restored copies, and any data transfer out. Check current pricing for your region, including the 180-day minimum storage duration charge.
Try both a small targeted restore and a larger recovery scenario. Deep Archive restores take hours rather than providing immediate access, and the time and cost of moving the restored data onward are separate considerations.
A hot index doesn't have to mean a particular database. Pick something the team can maintain and search, and record the original identifiers, archive object keys, package contents, checksums, and retention metadata.
Reconcile the index against the transferred objects and the source inventory. For compliance, also check that the chosen retention and access controls meet the actual requirements before retiring the old media.
Answers are reviewed before publication.
buf_2x9:
Keep the manifests somewhere readily accessible, with checksums and enough metadata to find the right package. The index should be backed up too.