Hey everyone! I'm in the midst of designing an on-premises Kubernetes cluster and I'm curious about how to best handle the storage system. I've drafted three different configurations, but I can't shake the feeling that they might be overkill. I'd love to hear your feedback!
Here's my situation: I need a storage system that provides Persistent Volumes (PVs). I also want to ensure that I can store logs from all components, including the storage system itself. However, I realize that logging directly onto the storage could create a circular dependency, which I want to avoid.
Moreover, since storage is crucial for the whole system, any failure there could impact everything. To mitigate this, I think it could be wise to set up the storage in a dedicated cluster, separate from other workloads.
I've considered three clusters: the first for non-storage workloads (let's call it the app cluster), the second for robust storage services (the storage cluster, possibly with Rook/Ceph), and a third, smaller, reliable cluster for logging (the core cluster). By routing logs from both the core and storage clusters back to the storage cluster, I can reduce the risk of circular dependencies while still centralizing log management.
This design aims to manage node pools efficiently and keep everything on-premises, as I'm not looking to employ an external log storage service like Datadog. Thanks for reading!
2 Answers
Why not go with RKE2 and Longhorn? It’s super easy to set up and has great resilience. You might find that it meets all your needs without all the complexity!
Honestly, your setup sounds a bit complex for what you might need. We've been just using local disks for a while now without any major issues. Sometimes simpler is better!
Definitely, Longhorn is a solid choice! Plus, I've heard great things about Rook/Ceph for shared storage—could work for you too!