We currently run several similar three-node k3s clusters on RHEL 8 VMs. Each product includes a three-replica Crunchy PostgreSQL cluster with PVCs backed by dedicated LVM volumes, a three-replica ClickHouse deployment, KeyDB, Redpanda, Longhorn for highly available S3 storage backed by a VersityGW pod, and several stateless applications exposed through nginx-ingress. MetalLB provides virtual IPs from our network infrastructure.
We are planning a larger k3s installation on new ESXi hardware, with three dedicated control-plane VMs and multiple worker VMs. The goal is to consolidate current products and future deployments into one cluster while keeping each product and environment—development, testing, staging, and production—as isolated as possible in terms of resources, networking, and operations. We also use operators, which install cluster-scoped CRDs, so separating those between teams may be important.
We are considering vcluster to provide isolated control planes, although that would add complexity. PostgreSQL and ClickHouse pods currently use node affinity to attach replicas to dedicated workers. Could multiple independent PostgreSQL or ClickHouse clusters safely share the same worker nodes if they use separate storage classes and appropriate scheduling constraints? We are not yet considering a single shared PostgreSQL, ClickHouse, or KeyDB service for every product.
What architecture would you recommend? Is consolidating these products into one large cluster worthwhile, given the effort required for resource, network, storage, and operator isolation? We also need several development teams to maintain their own environments while sharing the underlying infrastructure.
3 Answers
Multiple ClickHouse and PostgreSQL instances can technically run on the same worker node, and separate storage classes do not prevent that. The important controls are pod anti-affinity or topology spread, resource requests and limits, dedicated node pools, taints and tolerations, and storage with predictable performance and failure behavior.
For production databases, placing several replicas or several tenant clusters on the same worker reduces resilience: one node failure can affect multiple products, and noisy-neighbor I/O or CPU contention can impact all of them. I would allow co-location only when capacity is intentionally overprovisioned and the failure impact is acceptable. For critical environments, use separate node pools or at least enforce topology rules so replicas and important tenants are spread across independent nodes and zones.
CRDs are cluster-scoped, so they cannot be isolated by namespace. Installing or upgrading an operator may affect every tenant that uses the same CRD, and two teams cannot independently install incompatible versions of that CRD in one cluster. If separate operator lifecycles, versions, or administrative boundaries are requirements, use separate physical clusters or virtual clusters that provide their own control planes and API resources.
Namespaces are still useful for ordinary workload isolation, but they are not equivalent to separate clusters. Decide which boundaries actually need independent control planes, cluster-wide operators, upgrades, and failure domains before choosing the consolidation model.
A shared cluster can work, but the isolation has to be implemented inside Kubernetes with namespaces, quotas, limit ranges, network policies, RBAC, dedicated node pools, taints, and carefully designed storage policies. I would avoid making one shared PostgreSQL or similar database service the default for every product; it creates a large blast radius and couples independent teams and release cycles.
I would strongly consider keeping production separate from non-production, at least from an availability and capacity-planning perspective. Development and test workloads can often share a cluster, while production gets its own failure domain and upgrade schedule. k3s is capable of handling this scale if it fits your operational needs, though another Kubernetes distribution may provide more granular control-plane management.
k3s itself is reasonable here, especially if the team already knows it. I would not switch distributions solely because the cluster is getting larger. A virtual-cluster solution can help when teams need independent control planes, but it is most useful when you specifically want cluster-like boundaries without managing additional VMs.
The operators can deploy separate PostgreSQL and ClickHouse clusters in different namespaces, and network policies can block tenant-to-tenant traffic. That provides logical isolation, but it does not remove shared-node, shared-storage, shared-control-plane, or shared-upgrade failure modes.