I'm building a homelab K3s cluster with the control plane configured for HA and embedded etcd. The server VMs use raw iSCSI disks over 1 Gbps Ethernet with a 1500-byte MTU, backed by a NAS containing spinning disks. I also tested NFS and adjusted its caching, but etcd still cannot keep up with its read and write workload. The K3s servers repeatedly fail and restart with etcd Raft errors, including messages that applying an operation took longer than the 100 ms threshold. Is reliable K3s HA possible with this kind of storage, or do the control-plane nodes effectively need SSD-backed storage? I'd like to keep experimenting without having to spend a large amount on a dedicated SSD NAS.
3 Answers
I had a similar experience with control-plane nodes booting from iSCSI against a RAID10 NAS. The worker nodes were fine, but etcd was by far the least tolerant part of the setup and failed regularly under that storage configuration. Moving the control planes onto small dedicated machines made the cluster dramatically more stable, even though those machines also ran workloads. They don’t need to be expensive servers—small x86 boxes with local SSDs can be enough for a lab.
etcd is very sensitive to write latency and random I/O, so spinning disks behind iSCSI can easily cause Raft timeouts. A flash write-back cache in front of the iSCSI volume may improve things by absorbing synchronous writes and flushing them more efficiently. NFS is generally an even worse fit for etcd. A fast local SSD or properly backed RAM-based cache could help, but make sure the cache has power-loss protection and reliable persistence.
That kind of cache would be local to each VM or host, though. It might improve performance, but it wouldn’t preserve the same behavior if the VM crashed or had to migrate to another host.
For a lab, the simplest solution may be to put the control-plane nodes on local SSD storage rather than trying to make the NAS carry etcd. Even modest SATA SSDs attached to the virtualization hosts can reduce latency enough to avoid the Raft timeouts. You can keep bulk workloads and persistent application data on the NAS, while keeping the control-plane database on fast local storage. Three control planes on one host won’t provide real host-level availability, but it can still be useful for learning HA behavior.
By small dedicated machines, do you mean mini PCs or similar low-power systems? That sounds more practical than replacing the entire NAS.