Hey everyone,
I currently run a PostgreSQL database on a dedicated server where I have six scripts constantly scraping data from a video game API and inserting it into my database around the clock. On average, we're looking at about 30 rows being inserted across three tables each second.
I'm considering moving things to the cloud and wanted to know if AWS has database solutions that would suit my needs. Right now, I'm paying about €30/month for my dedicated server, and I'd like to know what options AWS offers that might fit within a similar budget. Would Amazon RDS or Aurora be viable choices? Additionally, I'm running large Common Table Expressions (CTEs) every minute that take about a minute to process, and I'm concerned about whether a cloud setup would be significantly more expensive than my current €35/month VPS.
Thanks for your input!
6 Answers
If you're not concerned about atomic consistency and mainly do data analysis, consider saving your data to S3 and using tools like Athena or DuckDB for querying. It could save you costs significantly.
Aurora might be beneficial, but it could end up costing more. Relational databases aren't typically the best choice for heavy data insertion cases like yours. Instead, I recommend using AWS Lambda with EventBridge to handle your scripts. For storing the data, knowing how you use the data afterward would help in deciding between DynamoDB, S3, or EFS.
RDS supports multiple database engines, and Aurora/PostgreSQL sounds like a great fit for you. But if you're considering a NoSQL approach, DynamoDB could be a cheaper alternative. You should also check the AWS Pricing Calculator to estimate costs based on your storage and read/write needs.
AWS has various database options to fit different needs, so you'll want to find the best one based on cost. Since you're using PostgreSQL, RDS for PostgreSQL could work well. However, considering your high insertion rate, you might also want to check out NoSQL solutions.
Running a managed DB 24/7 will generally be pricier on AWS compared to your dedicated setup, but your actual needs matter. If the data is just for historical tracking and trends, there are more cost-effective solutions than what you're currently using. Identify what you need and it’ll guide your choice.
Just a heads-up, no 24/7 managed service on AWS is cheaper than your dedicated server. The cost isn’t just from database inserts; it's largely about compute expenses. Keep that in mind when making your decision!
Related Questions
Cloudflare Origin SSL Certificate Setup Guide
How To Effectively Monetize A Site With Ads