I've set up a pretty solid data architecture on AWS, using Redshift for data warehousing and QuickSight for visualization. However, I'm hitting a wall with the SaaS ingestion layer that connects over 25 applications. Each ingestion point is a custom-built Lambda or ECS task, created by different people over the past few years, leading to inconsistencies in code quality. When these tasks fail, it disrupts the entire data chain—my data lake doesn't get updated, Glue jobs work with old data, and dashboards display outdated figures. I'm considering swapping this chaotic ingestion layer for a managed tool, while keeping my data transformations and visualization layers intact. Has anyone successfully tackled a similar challenge? What was the migration process like, and did it alleviate your operational struggles?
1 Answer
It really comes down to the specific issues you're facing. If the problems are mainly about code quality, a good approach could be to develop Lambda middleware that standardizes some functions. On the other hand, if memory or scaling is the issue, consider leveraging ECS or Fargate to better manage workloads. Just be wary that the root cause impacts how you should tackle the problem!
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures