I'm dealing with some serious issues in our custom-built data ingestion system on AWS. We set it up the 'cloud-native' way with multiple lambda functions for each SaaS source, orchestrated by Step Functions and monitored by CloudWatch. However, it has turned into a maintenance nightmare with 18 different lambdas, each written in varying Python versions and with inconsistent error handling. When failures occur, tracing the root cause takes ages, and sometimes we even end up with duplicate data due to partial extractions. I'm thinking about replacing this whole custom setup with a managed ingestion tool that could load data directly into Redshift while keeping some raw data in S3. Has anyone made this transition, and did it reduce your operational stress enough to justify the cost?
1 Answer
I actually made that switch about eight months ago! We replaced our custom lambdas with a managed tool for SaaS ingestion while keeping everything downstream intact. The data still lands in S3 in the same structure, so our Glue jobs weren’t affected at all. The best part? We stopped getting woken up at 2am for lambda failures – a huge relief! The cost of the tool ended up being similar to what we were spending on lambda executions plus maintenance efforts.
That’s exactly what I’m hoping for! I was worried that changing the ingestion layer would force us to redo the entire pipeline, but it sounds like you just swapped out the data producer without any major headaches.