How to Migrate Daily MySQL Loads to S3 on AWS?

0
4
Asked By TechyTurtle49 On

Hey everyone! We're in the process of moving our workloads to AWS from an on-prem Cloudera setup. Currently, we use Sqoop for our daily loads from RDBMS to HDFS. I'm looking for a comparable tool within the AWS ecosystem that can help us achieve this, ideally without using binlog CDC, as our use case is straightforward—the tables we want to load have a clear updated_date and records aren't deleted. Any suggestions?

2 Answers

Answered By DataWiz007 On

You might want to check out AWS Database Migration Service (DMS). It's designed for just this kind of scenario!

Answered By GlueGuru92 On

AWS Glue would be a good fit! It can connect to JDBC-compatible databases like MySQL. You can set up a Glue job and pull rows based on your updated_date. Plus, you can store the data in S3 in formats like Parquet or CSV, similar to what you do with HDFS. It also allows scheduling this process daily. Just a heads up, though, connecting Glue to your on-prem infrastructure might be a bit complex. You can use Direct Connect (DX) or a Site-to-Site (S2S) VPN for that.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.