I'm diving deep into Kubernetes as it's a new requirement for my job, and I currently run a homelab on a minimal server (Alienware Alpha R1). To grasp Kubernetes better, I want to containerize some of my existing virtual machines as part of a bigger transformation. My plan is to scrap everything and start fresh with Proxmox. Here's a list of applications I currently run: Home Assistant, Plex, Radarr/Sonarr/Overseerr, Pi-hole, Windows Server 2019 (for Windows-related tasks), a general-purpose Linux VM for experimentation, ephemeral containers for coding, and a few others like Fortimanager and Fortianalyzer. I'm looking for advice on how to determine which apps should stay as VMs and which ones are better suited for Kubernetes. I plan to run full Kubernetes instead of K3s, with control-plane and worker nodes as virtual machines on Proxmox. Any tips would be appreciated!
4 Answers
Running on K3S has been great for me, but I have also considered moving to Talos for its lightweight nature. Having a core system helps me focus on learning. You might want to experiment with what's lightweight enough for your needs and see how it scales!
I've been experimenting with a similar setup, and my go-to rule is that if an app needs specific hardware access or is a fundamental service like a DNS server, I'd suggest keeping it on bare metal. For example, I run Pi-hole on Raspberry Pis as appliances because they make for easy backups and quick reprovisioning, which is crucial if Docker or my cluster has issues. On the flip side, I run Home Assistant in a VM, but might switch to metal if I need hardware access like Bluetooth dongles. For everything else that can run in a container, just throw it in Kubernetes and let it manage resources!
Run everything if you've got proper networking in place! I even manage my home router on Kubernetes. With a good VLAN setup, it's easy. I'm a network engineer turned cloud guy, so I've got my Fortinet stack running smoothly in my lab—just plan out the networking well!
Wow, that's ambitious! I was thinking about doing something similar with my networking gear. How's the performance been?
I strive to run almost everything on my Kubernetes cluster except for essential things required for networking and the cluster's operation. For instance, I keep my DNS server and Git server outside the cluster. If it's suitable for a container, I go ahead and deploy it on K8S!
Right there with you! I keep lightweight services running outside my cluster too, but the rest goes in as it's much simpler to manage.
I totally get you! I have a similar setup—most things run within the cluster except for critical network services. Keeping those external makes recovery easier!