I'm looking for help with migrating data between two AWS S3 accounts that are linked to different email IDs and billing. I initially uploaded around 4GB of files (about 3000 files) to my first account and started using a new account for uploads over the last couple of months. Now I want to transfer all my data from the first account to the new one. While I've attempted to use AWS DataSync, it's not working for me. Once the transfer is done, I also need to update my database with the new AWS S3 links. Can anyone guide me on how to accomplish this migration?
3 Answers
You might consider setting up S3 bucket replication across accounts. You can also create an S3 event trigger for your new bucket that activates an AWS Lambda function to update your database as files arrive. Just note that replication will only handle new or updated content, so it won't help with what's already there.
Make sure to set a resource policy on the target bucket to allow the source account to put objects. From your source account, you can use the AWS CloudShell to run the S3 copy command to begin transferring your files.
Could you share more details about why DataSync isn't working for you? I recently moved about 8 million files—around 800GB—from one S3 bucket to another using it. You have two options: either push data from Account 1 to Account 2 or have Account 2 pull from Account 1. Just remember that you'll need to update the bucket policies accordingly.
Exactly, replication won't transfer existing files. It's best for ongoing updates. You’ll need to manually copy the older files first.