I'm trying to figure out how to capture every INSERT, UPDATE, and DELETE operation from our Aurora PostgreSQL database to S3 in Parquet format. This is important for compliance and historical analysis, essentially implementing a Slowly Changing Dimension (SCD) Type 2 for all our tables. AWS Database Migration Service (DMS) seems like the right tool since it supports Change Data Capture (CDC) and can automatically handle all tables using wildcard patterns without needing to configure them one by one. However, I'm worried that DMS was primarily designed for one-time migrations, not for continuous long-term usage.
Is there a built-in solution in AWS for this type of requirement? I prefer not to write code for each table or deal with potential issues around atomicity from the services that interact with the database itself.
1 Answer
We actually use DMS for ongoing replication from our Aurora database to S3 in Parquet format, and we've had very few issues. It’s been pretty reliable for us!
That's exactly the kind of setup I'm looking for! Do you partition the Parquet files by date? If so, could you outline your setup? Is it integrated with Lambda like a lot of AWS services?