We operate a siloed DBaaS platform on plain EC2 instances and pass egress charges through to customers. Each cluster is identified by tags, and a highly available cluster may contain multiple instances. I need a programmatic way to determine the egress generated by each instance group, ideally with hourly usage and costs, while keeping monitoring expenses as close to zero as possible.
The main complications are distinguishing inter-Availability Zone traffic, cross-Region traffic, internet egress, and the different AWS egress SKUs. Our ClickHouse deployments also write directly to S3; that traffic appears as network-out, but we do not want to charge the customer for those S3 transfers. We do not use NAT gateways or load balancers. Accuracy is more important than hourly resolution, so daily allocation would be acceptable if hourly reporting is impractical. What AWS data sources and calculation method would you recommend?
3 Answers
Treat S3 separately instead of trying to subtract it from a single NetworkOut number. Identify the S3 destination and transfer path, then apply the applicable S3 and EC2 transfer rules. Traffic to S3 in the same Region is commonly free from the transfer-charge perspective, but cross-Region traffic and certain access paths can incur charges. The CUR/CUD line items are the source of truth for the final billed amount; network metrics can only estimate or attribute usage.
For customer billing, a practical approach is to calculate each cluster's eligible transfer charges from the daily CUR, exclude the S3-related usage types you have agreed not to pass through, and reconcile the total against the AWS invoice. Document any shared or unattributable charges rather than forcing them onto an instance.
Use the AWS Cost and Usage Report, or the newer Cost and Usage Data Export, and query it programmatically with Athena. The report includes separate usage types for inter-Availability Zone transfer, Region-to-Region transfer, and data transferred out to the internet. Depending on the charge, EC2 resource IDs may appear in the line items, so you can map those IDs to your cluster tags and aggregate the costs across all instances in a highly available group.
For low cost and reliable billing, enable hourly granularity if you truly need it, but daily granularity is usually much cheaper and easier to reconcile. Keep the raw report in S3 and run scheduled Athena queries rather than collecting detailed network telemetry continuously.
The report is still the programmatic API in practice. Export it to S3 and query it with Athena, or consume the generated files with your own job. CloudWatch network metrics generally show bytes moved, not the exact billable SKU or final charge.
Be careful about treating EC2 NetworkOut as billable egress. It is an instance-level byte counter and does not cleanly identify whether traffic went to another AZ, another Region, the public internet, or S3. It also may not match the bytes used by AWS billing because different transfer paths and directions have different pricing rules.
VPC Flow Logs can help attribute traffic by destination, interface, and byte count, but collecting them for every ENI and retaining them can cost more than the monitoring benefit. They are better for validation and exception investigation than as the primary billing system. If you use them, filter aggressively and sample or aggregate the records before storing them.
That helps for the billing data, but I was hoping to calculate the usage directly from an API rather than manually inspecting the report.