We have 1,280 identical RK3588 ARM64 bare-metal nodes already deployed, powered, networked, and accessible through remote KVM in a U.S. commercial data center. We may eventually expand to similar racks in other U.S. locations, but we do not currently have a dedicated infrastructure engineering team.
The goal is to turn this fleet into reliable containerized compute without building an entire cloud platform from scratch. We strongly prefer a turnkey or managed solution, including a provider that could operate the hardware and Kubernetes environment for us.
I'd like practical advice from people who have operated large fleets of physical machines or edge Kubernetes environments. In particular, how would you evaluate K3s versus standard Kubernetes, Talos and Rancher, Omni or MAAS-based provisioning, one very large cluster versus multiple smaller clusters, remote reimaging, monitoring, tenant isolation, ARM64 image compatibility, hardware failures, storage, and networking?
I'm also wondering whether running 1,280 relatively small nodes is inefficient compared with consolidating workloads onto fewer larger servers. On the other hand, smaller nodes may reduce the blast radius of individual failures. Most importantly, are there managed-service providers or infrastructure companies that will operate Kubernetes on customer-owned bare metal at this scale? I'm more interested in technologies that have been run successfully in production than in a theoretical list of tools.
6 Answers
Start with a bare-metal provisioning system such as Canonical MAAS or an equivalent PXE-based workflow. It gives you a reliable way to inventory, reinstall, and recover nodes remotely. Use NetBox or a similar source of truth for hardware and network information, then apply post-install configuration with Ansible, Puppet, or Chef. This separation makes replacing a failed node much less painful.
Do not choose the cluster layout before understanding the workloads. A single enormous cluster may increase control-plane and operational blast radius, while several regional or workload-specific clusters are easier to upgrade and isolate. Conversely, many small clusters create more management overhead. The number of nodes is not automatically a reason to use K3s; benchmark K3s and full Kubernetes against your actual scheduling, networking, storage, and observability requirements.
A paid platform may be more sensible than assembling and maintaining a large collection of open-source tools. Platforms in the Spectro Cloud or similar category can manage the metal lifecycle and Kubernetes layer together, with centralized policy, upgrades, and support. The licensing cost may be small compared with the engineering time required to integrate, secure, monitor, and operate MAAS, configuration management, Kubernetes, and observability yourself. If a commercial product is not possible, MAAS plus Rancher is a reasonable starting point.
The appealing part of that approach is having one system for both the physical fleet and the clusters instead of treating provisioning and Kubernetes as unrelated problems.
Talos with Omni is worth evaluating if you want an immutable, declarative Kubernetes operating model. A bare-metal provider can install Talos through PXE and DHCP, while the management plane handles cluster configuration and coordinated updates. It supports ARM64, but I would not assume that a setup tested on a handful of nodes will behave the same way across thousands. Validate management-plane scale, upgrade concurrency, failure recovery, and the specific RK3588 hardware before committing.
The main ARM64 risk is less about Kubernetes itself and more about the RK3588 software and hardware ecosystem. Most mainstream container images now support ARM64, but check every dependency, especially databases, service-mesh components, monitoring agents, and images that assume x86 instructions. Verify kernel support and required CPU features as well. Some ARM system-on-chip vendors provide a heavily patched kernel and then stop maintaining it, so long-term kernel updates, drivers, firmware, and replacement hardware need to be part of the evaluation.
The nodes are only one part of the design. Plan separately for network capacity and failure domains, centralized or replicated storage, hardware telemetry, out-of-band recovery, image registries, logging, metrics, and security boundaries. These small systems may not be suitable for hyperconverged storage, so you may need dedicated storage servers or an external storage layer. Before selecting a provider, ask for references operating comparable ARM64 bare-metal fleets and require a pilot that exercises mass reprovisioning, rolling upgrades, node failures, and multi-tenant isolation.
The case for many small nodes is not only capacity efficiency. Losing one node takes out a smaller amount of work, so the failure blast radius can be much lower than with a fleet of a few large servers.