We run a siloed DBaaS platform on plain EC2 instances and include network egress in each customer's SKU. I need a reliable way to attribute egress costs to a customer's database cluster, using resource tags to group instances. High-availability clusters may contain multiple EC2 instances, and NetworkIn/NetworkOut metrics make it difficult to separate customer traffic from replication and other internal traffic. We also run ClickHouse workloads that write directly to S3; that traffic appears as NetworkOut, but we do not want to charge customers for the transfer to S3. There are also several AWS egress pricing categories, including cross-AZ, cross-Region, and internet data transfer. What is the most accurate programmatic approach you have used to measure and allocate these costs while keeping monitoring expenses low? Hourly billing would be ideal, but daily billing is acceptable if hourly attribution is not practical.
1 Answer
Use the AWS Cost and Usage Report as the billing source rather than relying only on EC2 NetworkOut metrics. The report can expose separate usage and cost records for categories such as inter-AZ transfer, cross-Region transfer, and internet data transfer. Enable resource IDs where supported, then join those records with your instance inventory and tags to allocate costs to each cluster. Process the report programmatically with Athena or another batch pipeline. This is generally more accurate for billing, although it usually is not truly hourly; daily or several-times-per-day updates are more realistic.
That makes sense for the billing records. I was hoping to calculate it directly from monitoring data, but I may use the cost report for final attribution if the usage metrics cannot distinguish the traffic types.