I'm a beginner with Kubernetes and I've set up a three-node k3s cluster at home, aiming to run services like Authelia and Gitea, with plans for Home Assistant later on. My setup includes mini PCs with a single gigabit NIC, which are in different rooms and connected through three switches, so I'm concerned about latency between them. The cluster is IPv6 only on a ULA, which gives me good address space for MetalLB (I've allocated it a /112 pool). I'm using BIND for internal DNS to manage records. Persistent storage will be handled by Ceph/Rook on the nodes' internal storage, so inter-node traffic is a concern since the hardware isn't very powerful (i7 8550U, 32GB RAM, each with a 1TB NVMe SSD). I've configured MetalLB in L2 mode, assigning each service a unique IP. This setup keeps the traffic local to the node running the service, which I believe helps minimize unnecessary inter-node traffic. Considering all this, I'm wondering if I really need an ingress controller. If I go that route, I'd end up with one service managed by MetalLB with a single virtual IP, simplifying some things but also introducing challenges like managing DNS records for CNAMEs or AAAA records. I'm also worried about security; using an ingress controller could expose unencrypted traffic between it and the pods. Based on these factors, I think it might be best to just expose my services directly with MetalLB. Am I missing anything?
2 Answers
Keep in mind that using Rook Ceph might consume a lot of RAM for OSDs. It can be great for larger clusters with many clients, but for your setup, it might be overkill. Consider simpler alternatives like NFS with a local provisioner, which could be less resource-intensive.
Interesting points! What happens when a node goes offline with the setup you're using? Is there any redundancy?
Honestly, you probably don’t need to stress about the latency between your nodes. Most real-world scenarios deal with traffic between data centers, and that’s typically way more complex. As for the ingress controller, its main role is to route traffic and handle encryption. If you skip it, how do you plan to manage routing? While you could manage DNS records individually for each service, a wildcard record could make it easier by routing all requests from a subdomain to a single IP, reducing hassle.
Absolutely, managing traffic can get tricky without an ingress. Maybe try looking at a wildcard DNS to simplify things.
I've used Rook Ceph, and it ended up taking about 12GB with three OSDs, which is solid, but running it on a shared 1Gbit NIC was slow for me, and I had to upgrade to a 2.5Gbit setup.