Is it worth replacing the SaaS ingestion layer in my AWS data pipeline?
Asked by
CuriousCoder92
I've set up a pretty solid data architecture on AWS, using Redshift for data warehousing and QuickSight for visualization. However, I'm hitting a wall with the SaaS ingestion layer that connects over 25 applications. Each ingestion point is a custom-built Lambda or ECS task, created by different people over the past few years, leading to inconsistencies in code quality. When these tasks fail, it disrupts the entire data chain—my data lake doesn't get updated, Glue jobs work with old data, and dashboards display outdated figures. I'm considering swapping this chaotic ingestion layer for a managed tool, while keeping my data transformations and visualization layers intact. Has anyone successfully tackled a similar challenge? What was the migration process like, and did it alleviate your operational struggles?
Answers
It really comes down to the specific issues you're facing. If the problems are mainly about code quality, a good approach could be to develop Lambda middleware that standardizes some functions. On the other hand, if memory or scaling is the issue, consider leveraging ECS or Fargate to better manage workloads. Just be wary that the root cause impacts how you should tackle the problem!
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions