Hey everyone! I'm currently working on a project where I have an Excel file containing financial data, and it includes a separate sheet for each month. The data spans from June 2020 to now, and I update it daily by appending new data to the month's sheet. I want to conduct some analytics on this data, like finding the max and min transaction volumes and values for both monthly and yearly periods. I'm planning to use Python for this task. I'm considering two approaches: either storing all the data in Pandas DataFrames or putting it into a database. Given my requirements, I'm trying to decide whether to use EC2 or Lambda for this. I think Lambda might be better since I want to run the analytics weekly or monthly, and the computations will only take a few minutes. However, if I go with the database route, EC2 seems like a better choice. I'm new to cloud computing, so sorry if this is a basic question! I'd appreciate any advice or suggestions. Thanks!
6 Answers
For the best results, keep the compute and storage separate! They're different tools for different purposes.
Using Lambda sounds like a great idea! Just keep in mind whether your analytics will only cover the current month or the entire dataset. Also, remember that Lambda has a max execution time of 15 minutes, so make sure to profile your processing time. I suggest considering S3 for storage instead of a traditional database. It often simplifies things, especially if request times aren’t a critical concern. You could even set up an S3 trigger to invoke the Lambda function automatically when new data is added.
That's a solid point! S3 is great for managing data without the hassle of database transactions. Plus, you can use tools like S3 Select or Athena to query your data easily.
Lambda combined with S3 will fit your needs perfectly. Just make sure to strategize for future scaling!
Exactly, Lambda and storing data on S3 is a practical approach. It's beginner-friendly too! Just make sure you consider the learning curve for a noob like yourself.
If you're using Python to process large Excel files, be careful! It can get slow and memory-intensive. If your project scales with more data, consider AWS Batch instead. It avoids Lambda's limitations and allows you to handle larger jobs, plus you can still store results in S3.
If your processing is quick and wraps up in under 15 minutes, Lambda is indeed the way to go, especially for infrequent usages. If you're just focusing on temporary data, maybe saving to a CSV in S3 will work fine! But if you need something more permanent, a serverless database might be the right call.
Exactly! Just keep in mind that if you're not storing the data, once the Lambda function runs, anything in memory goes away.
Thanks for suggesting S3! I was focused on using Lambda for the whole process, but I see how separating storage helps.