Hey everyone! I've been stuck for days trying to migrate my data from one AWS account to another. Most of my services are transitioned, but I'm struggling with OpenSearch, which is critical for me because I need to retain historical data. In my old account, I have a serverless OpenSearch collection that's public, and I'm backing it up to an S3 bucket with ACL enabled. It's been frustrating because OpenSearch has some limitations: I can't ingest data from one OpenSearch instance in a different account due to the data access policies preventing cross-account access. Plus, the ingest pipeline won't allow S3 data from a different account, resulting in access denied errors. Is there anyone who's successfully migrated an OpenSearch public collection between accounts? I have around 1.5 billion documents amounting to 230GB of data, and I've been considering copying the documents to a new S3 bucket in the destination account without enabling ACLs, but I'm worried about the cost. Any suggestions?
2 Answers
Copying documents to a new bucket can seem expensive, though 230GB isn't massive. The real concern is the GET and PUT requests involved in handling 1.5 billion documents, which can rack up costs pretty quickly, especially with KMS encryption involved.
I think I get what you're aiming for. You might want to snap shop your OpenSearch domain in Account 1 to an S3 bucket there and then set it to allow cross-account access so that Account 2 can reach it. Once that's sorted, you could make the new OpenSearch domain in Account 2 and restore your data from the snapshot.
I believe that's for domains, but I'm using collections. Aren't they different, or am I misunderstanding?
Right? The volume will definitely add up due to the double requests you’d incur during transferring, not to mention the potential overhead from decryption and re-encryption.