I'm putting together a home lab on one physical machine and want to run a 3–4 node Kubernetes cluster as virtual machines, such as one control plane and three workers. I'd also like to use the system for local LLM inference and possibly leave enough headroom for smaller fine-tuning or training workloads.
What hardware should I target for this setup? I'm especially interested in recommended CPU core counts, RAM, storage, and GPU or NPU options. Specific mini PCs or compact systems with reliable GPU passthrough would be helpful too. I'm trying to stay budget-conscious, but I'm willing to spend more if the upgrade makes a meaningful difference for AI performance. Real-world experience with similar configurations would be appreciated.
5 Answers
If you plan to split one large language model across multiple nodes, ordinary Ethernet will likely become a major bottleneck. For a smoother setup, keep the model and GPU on one machine and let Kubernetes run applications that call it over the network. Distributed inference or training generally requires much faster interconnects and substantially more VRAM.
Most true mini PCs don’t include a useful discrete GPU. For local AI, a small-form-factor desktop with a dedicated graphics card is usually more practical than a mini PC. An RTX 3060 or newer is a reasonable starting point for running 7B-class models, although VRAM becomes the limiting factor quickly for larger models and training. GPU passthrough can also be temperamental on compact systems, so running inference on the host and exposing an inference service to the cluster may be simpler.
Running several virtual Kubernetes nodes on one host is useful if your goal is to learn cluster administration, scheduling, and failure behavior. It won’t provide real hardware redundancy, though, because the host is still a single point of failure. If you mainly want to run workloads, a single-node k3s installation directly on the machine will leave considerably more CPU, RAM, and storage available.
The main goal is to simulate a cluster and practice orchestration. I understand that separate virtual nodes don’t protect against the physical host failing, but that limitation is acceptable for this lab.
One practical example is a Minisforum MS-02 Ultra with a 235HX processor and 32 GB of RAM. It can run several worker VMs with workloads such as Cilium, Longhorn, Prometheus, application services, and llama.cpp using a passed-through GPU. The system supports up to 256 GB of DDR5 memory, which is valuable because RAM is often the first limitation when running four VMs. A low-power N100 mini PC with 16 GB can host a lightly loaded control plane separately, while the stronger system handles the workers and AI workload. CPU usage tends to remain moderate except during compilation or other intensive tasks.
For practicing autoscaling and cluster operations, a hypervisor such as Proxmox can work well. You can create VMs for k3s or Talos and experiment with a Proxmox-compatible autoscaling provider. Just remember that virtual nodes on one machine are primarily a learning environment rather than a highly available production cluster.
A compact system such as an NR200P-class build can handle inference and image-generation tools with a suitable GPU, but even a high-end card may not provide enough VRAM for serious training. An external GPU connected over OCuLink is another option, though it adds cost, space, cooling, and setup complexity.