I recently took on the task of managing a startup's Kubernetes environment, but I'm feeling a bit overwhelmed. Our platform is multi-tenant and multi-region, which makes it unique. Currently, each customer has one web app pod and three database pods (two replicas and one primary), and this setup is on bare metal, which limits our options. The problem is that the application isn't using the replicas efficiently; they serve only as failovers, causing a lot of wasted resources. We also aren't properly distinguishing between primary and replica databases, making it hard to monitor and manage them, especially since we only have three nodes in most regions, and our control planes also act as workers.
We've been tasked with resolving what's being termed as the 'Postgres problem.' I'm proposing a few changes: first, we should adjust the application to write to the primary and read from the replicas; second, we need to designate certain nodes solely for Postgres through taints and tolerations; and finally, we should redesign our database schema for multi-tenancy. However, these suggestions have been dismissed as unreasonable and potentially cost-prohibitive, due to the need for new nodes in each region. I'm looking for a sanity check on my approach and whether my suggestions are feasible given our constraints.
4 Answers
Can you elaborate on what the 'Postgres problem' is? It feels critical to identify that before deciding on the best path forward. Can't fix what’s not defined, right?
You're on the right track, but don't feel you need to implement all those changes to make an impact! You can make a big difference just by routing reads to replicas with a simple configuration change — CNPG handles creating `-rw` and `-ro` services for you. You might get some pushback on that, but definitely question why they think it’s unreasonable! Another quick win would be to apply `podAntiAffinityType: required` to your clusters to better manage where your pods are running and who can access them.
Also, instead of bringing in new nodes, consider re-labeling existing ones for Postgres. Taints and tolerations should work for your current setup; that's actually recommended by CNPG. Plus, don't forget you can use cluster hibernation for lighter tenants to save on computing power. Additionally, your idea about multi-tenant schema is solid, but it is indeed a bigger lift. Just remember, if it feels too costly, that’s more of a business challenge than a technical one!
Exactly! And adding PgBouncer can help tremendously with connection concerns, and it’s easy to set up. You can do these adjustments without heavy lifting to your app.
Good points! Just focusing on those quick adjustments could have a notable impact sooner without needing all those other changes right away.
Your plan sounds solid, but execution might be limited by budget constraints. For now, I'd recommend focusing on monitoring — maybe set alerts when the app's response times slow down as this can indicate if the database is getting overwhelmed by I/O or CPU usage. Understanding the bottleneck is key to knowing what to fix first.
We’ve noticed some redundancy in customer database tables, which bloats storage. Plus, it seems like scheduling is based purely on current load without accounting for unpredictable spikes from users.
Totally agree! Implementing pod limits might help balance the load, but using idle nodes should be the goal.
Why not check the official documentation? They have a lot of useful guidelines on how to manage CNPG better, which could clarify some of your issues. Here's a link to start your research: https://www.postgresql.org/docs/
Thanks! We’re already using CNPG, and it feels like a lot of info is hidden. But it can help clarify the 'postgres problem' you mentioned.
Good point! The issue seems to be a lot of duplicate data storage and inefficient scheduling, leading to performance hits in various regions.