I'm looking to implement a cost-effective logging architecture in AWS for our business. I'm thinking of using CloudWatch for log management and then sending those logs to S3 via Kinesis Firehose for long-term storage. I'm also considering alternatives like Grafana or other direct options to S3. What are some budget-friendly options to achieve this?
5 Answers
It really depends on how you plan to access your logs. We typically use FluentBit to push logs to Kinesis, then either route them to S3 for low-volume access or OpenSearch for operational queries. S3 works fine for ad-hoc queries, but OpenSearch is better for frequent operational access.
CloudWatch can get pricey at $0.50 per GB ingested. If you can log directly to Firehose, it'll save you a lot! It's definitely worth considering that route.
There's definitely a lot to explore in this area. We’re checking out OpenTelemetry for better centralized logging in our apps and think it could be helpful. Using something like OpenObserve has been good for us so far, especially since it's low on resources. You can send logs directly to S3, eliminating the need for additional archiving.
Nice to know that OpenObserve can simplify S3 storage! That's a great tip.
If keeping costs low is key, S3 is hard to beat with its pricing options and lifecycle controls. If querying logs isn't crucial, using Athena could save you a ton. Just consider how often you'll need to access the logs and what level of redundancy you really need. Sometimes the simplest solution is the most cost-effective in the long run.
If frugality is your top priority, setting up your own instances or a Kubernetes cluster can be the cheapest route, though it does require more management. First, assess your requirements, like log access frequency and necessary redundancy. Some solutions may seem more expensive upfront but could save you in the long run. It’s all about finding that balance between cost and ease of management.
Totally agree, that could really cut down on expenses!