I'm currently using an on-prem Kubernetes cluster (rke2) primarily for stateless workloads, but I'm considering adding stateful workloads like Kafka and PostgreSQL. My main concern is storage management. I don't have much confidence in using NFS for databases, and tying storage to my hypervisor (currently VMware, planning to switch to Proxmox) feels risky in the long run. I'm wondering if it's feasible to use local storage as a CSI driver since Kafka and PostgreSQL handle data replication at the application level. Should I dedicate nodes for running these workloads despite potentially losing some scheduling flexibility? I'd love to hear any advice or real-world experiences from others who have tackled this.
5 Answers
We're currently using the vSphere CSI driver, and it has been working well for us. It's worth considering, especially if your environment is stable.
You should definitely consider starting with Rook. It manages storage well and would allow you to pass disks through to the VMs effectively.
If you're just reserving nodes for PostgreSQL or Kafka, it might be simpler to stick with VMs instead of adding Kubernetes complexity. But if you decide to go with Kubernetes, both PostgreSQL and Kafka have operators that can really simplify management. The use of local disks can work as long as your setup is relatively static and you don’t plan to move nodes around too much.
I'm using the CNPG operator, and it's been great for integrating backups to my preferred storage solutions. It could be a big help in managing PostgreSQL.
If you want redundancy, I suggest running 3 Kafka pods for that extra reliability. You can also start with internal storage as persistent volumes and then shift to external solutions as your needs grow. Just keep in mind that static storage options, like local disks, won’t move if the node fails, which is something to think about.
Related Questions
How To Get Your Domain Unblocked From Facebook
How To Find A String In a Directory of Files Using Linux