I'm trying to streamline my homelab infrastructure and reduce the number of deployments across tools like Docker, Podman, k3s, and more, all while looking to integrate GitOps with ArgoCD or Flux and keep my secrets secured with SOPS. I have multiple nodes planned, and overall, there will be six: three in my main cluster and three are more like satellites.
Currently, I have a persistent container handling my Bitcoin/Lightning stack. It runs smoothly on Incus, which has provided a solid setup with its own IP on the network. I stumbled upon KubeVirt for running VMs inside Kubernetes, but I'm particularly interested in a solution for managing persistent containers.
I'm envisioning a setup where I could specify a template, like Ubuntu 24.04, and then manage it like a typical node while keeping the extensive manual configurations outside of Kubernetes. There are also some IP-PBX systems that don't have ready-made containers, so I need a way to run those in persistent containers too. Does anyone know of any Kubernetes-native solutions that fit this description?
2 Answers
You might want to explore using a StatefulSet. It keeps a static IP and hostname, which can be combined with node affinity to ensure it's scheduled on the same node each time. While you’d lose some of the benefits, like high availability, there are also options for distributed storage.
It sounds like you're better off using a virtual machine for what you’re attempting. You could theoretically set up a Debian pod with a PVC mount point, install Debian inside that mount, and run things from there. Still, it might be unnecessarily complex.
True, but since my nodes are mid-range ARMs, I wanted to avoid the overhead of VMs. Looks like Kubernetes might not be the best fit for this scenario, but I'm still considering trying KubeVirt just to see how it goes.
Absolutely! Some use cases simply work better on VMs. It may be worth looking into consolidating your hosts into something like Proxmox and running Kubernetes within VMs for more flexibility.
Oh! The StatefulSet aspect totally slipped my mind. Pairing it with a chroot might just be the path forward for me. Thanks for the tip!