Our AWS bill is about $2,400 per month, with roughly $901 coming from CloudWatch. The biggest CloudWatch charge is Logs Insights data scanning: about $597 per month. That was surprising because we only store around 27.8 GB of logs, which costs roughly $0.92 monthly. The important distinction is that Logs Insights charges for data scanned by queries, not simply for data stored. Grafana dashboards were automatically refreshing panels that queried large log groups, resulting in approximately 119 TB scanned each month—repeatedly querying the same 28 GB of logs. We are considering moving dashboards to a self-hosted log store, but in the meantime, what approaches work well for always-on operational dashboards without paying for every repeated scan?
4 Answers
For anything that belongs on an always-on dashboard, turn the result into a metric instead of repeatedly querying raw logs. CloudWatch metric filters, embedded metrics, or application/container metrics can capture counts, rates, and error levels once, and the dashboard can display those cheaply. Logs Insights is better suited to investigation than continuous monitoring.
This is a common weakness of using CloudWatch Logs as the primary high-volume logging backend. A separate log system such as Loki, or logs kept in object storage with an appropriate query layer, can be more predictable for repeated searches. Just make sure the replacement is partitioned and indexed sensibly; an unpartitioned Athena dashboard can create the same scan-cost problem.
The same pattern shows up with other metered query engines: a dashboard makes a scan-based service look like a free visualization layer, then refreshes turn it into a recurring workload. Before migrating everything, calculate which panels really need raw-log data and replace the rest with precomputed metrics or scheduled aggregates.
The immediate cost-saving step is to disable automatic refresh on the Grafana panels and make refresh manual or much less frequent. Also audit each query's time range and log groups—broad searches across every group multiply the scanned volume quickly. That should reduce the bill while you work on a longer-term migration.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures