I'm building a K3s homelab with the control plane in HA mode, using embedded etcd. The server nodes are virtual machines whose raw disks live on a NAS over 1Gbps iSCSI with a 1500 MTU, and the NAS uses spinning disks. I also tried NFS and experimented with its caching, but the storage still seems too slow.
The K3s servers repeatedly fail and restart with etcd Raft errors, including messages that applying an operation took longer than the 100 ms threshold. Is reliable K3s HA with etcd realistic on this kind of setup, or is flash storage effectively required? I'd prefer not to spend around $1,000 just to experiment in a homelab. Has anyone made a similar configuration stable?
3 Answers
For a lab, the simplest fix is usually putting a modest SATA SSD directly in the hosts running the K3s control planes. You do not necessarily need an expensive all-flash NAS; even inexpensive 120 GB SSDs can be enough for the operating system and etcd. If this is only for testing, running the control-plane VMs on one host with local SSD storage may be more reliable than trying to make networked spinning disks satisfy etcd’s latency requirements.
I had a similar experience running control-plane nodes on small single-board computers that booted over iSCSI from a four-disk RAID10 NAS. The workloads were mostly fine, but etcd was by far the least reliable part and produced failures almost every day. Moving the control planes to small dedicated mini PCs made the cluster dramatically more stable, even though those machines also run some workloads.
etcd is very sensitive to random write latency, so spinning disks over iSCSI can easily cause these Raft timeouts. A flash write-back cache in front of the iSCSI volume might help smooth the writes, and a local SSD or even a carefully configured RAM-backed cache could make a noticeable difference. NFS is generally an even worse fit for etcd because its latency and consistency behavior tend to be less predictable.
A cache attached to the VM would only be local to that node, though. That could improve performance, but it would not preserve the same storage behavior if the VM crashed or migrated to another host.
By mini PCs, do you mean small dedicated x86 machines with their own local storage? I’m curious what kind of hardware made the biggest difference.