Hey everyone,
I currently run a PostgreSQL database on a dedicated server where I have 6 scripts constantly connected. These scripts scrape data from a video game API and insert around 30 rows, spread across 3 tables, every second!
I'm considering moving to AWS for a cloud-based solution, ideally something with a similar monthly cost to my current setup of about €35/month. Can you recommend an AWS database setup like RDS or Aurora that would be suitable for my needs?
Additionally, I deal with large CTEs that run every minute and can take about a minute to execute—all of this needs to be handled in a cloud environment. Any thoughts on whether AWS can provide a setup that meets my requirements at a reasonable price?
5 Answers
Aurora could work, but I’m cautious about the expense. Traditional relational databases aren’t always the best for heavy data insertion workloads. Consider a Lambda + EventBridge architecture for data ingestion, and based on your usage patterns, you could opt for DynamoDB, S3, or EFS depending on what you do with the data afterward.
Running a managed DB 24/7 on AWS usually costs more than a dedicated server. Your current challenge seems to be managing insertions and compute costs—consider how often you need quick access to the data you've inserted versus just storing it for analysis.
AWS has various database options tailored to different needs. Since you're using PostgreSQL, RDS for PostgreSQL would be a solid choice. However, given the high amount of data insertion, you might also want to look into NoSQL options that could handle your workload better.
RDS can support multiple database engines, and it sounds like Aurora/PostgreSQL could fit the bill for you. If you're willing to explore NoSQL, DynamoDB could end up being cheaper for your specific case. You can estimate your storage and read/write costs using the AWS pricing calculator linked here: [AWS Pricing Calculator](https://aws.amazon.com/aws-cost-management/aws-pricing-calculator/)
If you’re focusing on data analysis and don’t need strict consistency, you could save all your data in S3 and analyze it later using Athena or DuckDB, which might be cost-effective.
Related Questions
Cloudflare Origin SSL Certificate Setup Guide
How To Effectively Monetize A Site With Ads