I've noticed that about 30% of my GKE bill is eaten up by traffic costs under the "Network Inter Zone Data Transfer" SKU. My project relies heavily on internal traffic, which can add up to hundreds of terabytes each month between services. My cluster was initially set up with nodes spread across all available zones in the region, which I believe is the default configuration.
Currently, I've forced all nodes into a single zone to save costs, but I know this isn't ideal for availability. I'm wondering if there's a way to balance both having a multi-AZ cluster for improved availability while keeping intra-AZ traffic costs minimal.
While I can manually deploy separate application stacks for each AZ and load balance traffic, that feels overly complicated. Is there a more efficient method to encourage local communication between services in Kubernetes?
3 Answers
We recently switched to using a single AZ for processing, while keeping multi-AZ storage solutions like S3. It's been a massive cost saver. If you look at the outage history for your AZ, you'll find that the downtime is pretty minimal—typically less than an hour per year! It makes you question whether it’s worth spending 30% of your bill for such rare issues.
I spent seven years on AWS with several clusters in a single AZ and never faced any major issues that a simple instance restart couldn’t fix. The cost doesn’t seem justified to me, especially if no one’s breathing down your neck during downtime.
Have you checked into topology-aware routing? That could really help with your traffic costs by optimizing how requests are routed based on where your pods are located.
Not yet, but I’m definitely considering it after your suggestion!
There's no quick fix here, but you might want to look into using the `preferredDuringSchedulingIgnoredDuringExecution` node affinity rule. This would allow you to prioritize scheduling pods in one AZ without completely shutting down the others. If something does go wrong, your pods can still flip to the other AZ.
Just a heads up: If your workload is stateful, you’ll still have to deal with data transfer across AZs, which could keep those costs up. When running a database, think about partitioning your data in ways that limit network traffic, like doing joins locally and replicating smaller tables across both AZs.
In any case, be prepared that those traffic costs are a reality you might need to live with, especially as your usage grows.
Exactly! That was on my mind when I went for the single AZ approach.