Hey everyone! I'm currently managing an AWS infrastructure setup that consists of a single EC2 instance, and I'm on the lookout for advice on how to restructure it to optimize costs while also ensuring better scalability.
Here's a brief rundown of my current setup:
1. I have a single m7a.12xlarge EC2 instance (48 vCPUs and 192 GB RAM) running a Flask backend API through Gunicorn, which is managed via systemd and reverse proxied by Apache.
2. There's a MySQL database that runs locally on the same instance.
3. I've got over 10 dynamic client portals (built with HTML/PHP) hosted on this server and actively using the backend API for various actions.
4. Additionally, I have several cron jobs set up for automation tasks like backups and notifications.
The main issue I'm facing is frequent server overloads due to Gunicorn's memory consumption. I've tried reducing the number of Gunicorn workers, but the API becomes slow. On the flip side, increasing them for better performance results in a massive memory spike. While we upgraded to a large m7a.12xlarge instance recently, we are still encountering overload issues. The entire system is tightly coupled, leading to a single point of failure where any spike in memory or CPU can affect all components (API, portals, cron jobs).
I'm seeking the most beginner-friendly, scalable, and cost-effective strategies for redesigning this setup. Here are some options I'm considering:
1. Migrating the MySQL database to RDS.
2. Splitting the portals and API onto separate EC2 instances.
3. Implementing API Gateway, Lambda, and Layers.
4. Exploring Amazon Fargate.
I'd really appreciate any suggestions on the best approach for a beginner, as well as any pitfalls I should keep in mind during the migration process. Thanks a lot in advance!
1 Answer
Definitely go for migrating MySQL to RDS first. You could also run your Python app in Docker and put it on ECS Fargate. For things that are scheduled, think about using Lambdas with EventBridge, but if they’re CPU-intensive, look into ECS scheduled tasks instead. For the frontend, consider moving static HTML to S3 and using CloudFront to manage delivery. Just keep in mind that PHP might pose some issues, but you can work it into Docker on ECS too. Most importantly, take the time to figure out how data flows through your application and aim to decouple as much as possible to prevent overlaps and interruptions.
Totally agree with you! Understanding your system will significantly ease the decoupling process. It's best to tackle this in stages—like migrating the front end to ECS first if feasible, or even running it on another set of EC2 instances behind an ALB. Don't forget, running the database in RDS is a solid option too.