Hey everyone! I'm gearing up for a new project and could really use some advice. Has anyone tried running a k3s cluster on Raspberry Pi 5s? I've managed to create a working demo of an MQTT stack (including Mosquitto, Telegraf, InfluxDB, and Grafana) but my next step is to enhance it to be Highly Available (HA). I have three Raspberry Pi 5s ready for this setup. A few things I'm curious about: 1) Is the current k3s release stable on the Pi 5? 2) Are there any notable hardware or ARM-related issues I should know about? 3) Any suggestions for networking or storage to make this work effectively? I'd appreciate any tips or insights from your experiences! Thanks in advance!
5 Answers
I’m running a setup very similar to yours, but with Raspberry Pi 4s. Setting up k3s was a great learning experience, especially seeing how it handles high availability. I learned that using external SSDs is a game changer for storage speed and capacity. If you're going the microSD route, opt for Class A1 cards—they last longer than the cheaper options.
For your datastore, I suggest using PostgreSQL on a separate node instead of the embedded etcd. It can significantly reduce disk I/O issues. I also found Longhorn to be excellent for persistent storage; it’s seamless and saves you from the hassles of NFS or Samba setups. Just a heads up though, I had minor ARM-related hiccups with some pre-built Docker images, but that’s improved recently. Good luck with your project!
I echo some of the previous experiences here—Pi 5s offer their own set of challenges. When I started with a few Pi 4s, I noticed severe limitations when trying to run common workloads. Switching to mini PCs has been a significant improvement. For example, you can get the NucBox G3 for a good price compared to the Pi 5, which only provides basic specs without storage. When your cluster depends on performance, it’s better to invest a bit more upfront for reliable hardware.
I previously set up an HA cluster with 15 Raspberry Pi 5s but encountered several challenges. There were various observability issues with kube-prometheus-stack, and the single binary design of k3s made immediate metric monitoring tricky. Eventually, I transitioned to vanilla Kubernetes, which worked smoother for me. Also, the unique SOC on the Pi 5 can create compatibility issues with ARM workloads. The additional downside is that it’s pricey when compared to mini-PC setups that offer better performance and cost efficiency.
For high availability, make sure you have at least three control planes. For your three nodes, they can all function as part of this control plane alongside acting as workers. This can help ensure your setup withstands node failures effectively. Just be ready for some troubleshooting along the way!
I can't comment extensively on k3s since I went with plain Kubernetes on Ubuntu, but I’ve noticed some images lack ARM support. When that happens, building your own might be necessary, though it’s becoming rarer. I also ran into some complications with Istio due to the Raspberry Pi OS kernel options—consider switching to Ubuntu Server if you run into kernel-related issues!
Yeah, I noticed that too. They just don’t have the memory to handle heavy workloads! I switched to GMKtec mini PCs, with each having 32GB of RAM. Cost me around $1k, but worth it for the performance.