We currently run several similar three-node k3s clusters on RHEL 8 virtual machines. Each product includes a three-replica Crunchy PostgreSQL cluster with PVCs backed by dedicated LVM volumes, a three-replica ClickHouse deployment, KeyDB, Redpanda, Longhorn for highly available storage, an S3-compatible gateway, stateless applications, NGINX Ingress, and MetalLB for virtual IPs.
We are considering moving to one larger k3s installation on new ESXi hardware, with three dedicated control-plane VMs and multiple worker VMs. The goal is to host current and future products in one cluster while isolating each product and its dev, test, staging, and production environments as much as practical. We would need separation through namespaces, resource controls, networking, storage, and possibly dedicated workers. Since our deployments use operators, they also install cluster-scoped CRDs, which may make tenant separation difficult.
Virtual clusters could provide separate control planes and operator installations, but they would add complexity. PostgreSQL and ClickHouse currently use node affinity to place replicas on dedicated workers. Is it reasonable to run multiple independent PostgreSQL or ClickHouse tenants on the same worker, provided they use separate storage classes and volumes? Would separate clusters or virtual clusters be a better design, and is consolidating these products into one large cluster worth the isolation and operational effort?
3 Answers
Running the products in one Kubernetes cluster is technically feasible, but the isolation work moves into Kubernetes: namespaces, quotas, limit ranges, taints and tolerations, node pools, network policies, RBAC, storage policies, and careful monitoring. I would avoid making one shared PostgreSQL or ClickHouse service the dependency for every product unless you deliberately want that operational model; separate database clusters are usually safer for failure and lifecycle isolation.
I would strongly consider keeping production in a separate cluster. A single large cluster can be efficient for development and nonproduction environments, but it creates a larger blast radius and makes resource planning, upgrades, and maintenance more complicated. k3s is suitable if it meets your operational requirements; changing distributions is not automatically necessary.
Multiple ClickHouse or PostgreSQL clusters can run on the same worker, and separate PVCs or storage classes can keep their data paths independent. The real concern is not whether Kubernetes allows it, but whether the workloads have enough CPU, memory, disk I/O, and failure isolation.
For production database replicas, sharing a worker means one VM, kernel, disk subsystem, or maintenance event can affect several tenants at once. Use anti-affinity or topology spread constraints so replicas of each database are separated across failure domains. Dedicated workers are preferable for critical tenants, while co-location may be acceptable for development or less important environments if you enforce resource requests, limits, and I/O capacity.
A different storage class by itself does not prevent noisy-neighbor problems or protect against a worker failure. It separates provisioning and placement of volumes, but CPU, memory, network, and node availability still need explicit controls.
CRDs are cluster-scoped, so namespaces cannot isolate them. Multiple teams can share a CRD, but they cannot independently install incompatible versions of the same CRD or fully control its cluster-wide behavior. If independent operators or CRD versions are a hard requirement, use separate physical clusters or virtual clusters with their own control planes.
For ordinary workloads, use namespaces plus strict RBAC, default-deny network policies, per-tenant quotas, dedicated service accounts, and separate node pools where appropriate. Treat production and lower environments differently rather than assuming one policy fits all of them.
Virtual clusters can help when each team needs its own control-plane view and operator installation. They are most useful when you want cluster-like isolation without managing many separate VM-based clusters, but they still add another layer to troubleshoot.