We're exploring the deployment of on-prem Kubernetes clusters primarily due to customer requirements for keeping data off the cloud. We've got a couple of engineers on board with past experience in on-prem environments who are suggesting we go for bare metal with vanilla Kubernetes and Ubuntu for the nodes. However, since bare metal isn't an option for us right now, we'll be utilizing our existing VMware infrastructure.
Management has requested we use Bottlerocket as the operating system for consistency with our EKS clusters in the cloud. But we have some reservations, as Bottlerocket seems primarily designed for AWS, and we haven't seen much discussion about its use in on-prem setups. Has anyone successfully used Bottlerocket for on-prem deployments? What are your recommendations or challenges?
7 Answers
I use Bottlerocket in EKS, but I wouldn't recommend it for bare metal. For instance, using certain tools can be tricky with it. Talos has generally worked better for me in different environments.
If you're considering alternatives, Metal3 or MAAS combined with Talos might be your best bet. Avoid rushing into Bottlerocket—it could lead to a lot of unnecessary complications.
OpenShift integrates well with VMware and offers benefits like autoscaling, which might be more beneficial for your use case.
I’d steer clear of Bottlerocket for on-prem setups. I used to work with EKS Anywhere and found Bottlerocket to be a major headache for anything outside AWS. It's limited, requiring AWS components for local access, and you'll struggle with configurations due to its cloud-init restrictions. Trust me, it’s a pain compared to options like Talos. If you're interested, I’ve even compared them in detail in a blog post.
I completely agree with your points. Also, for security needs, you might want to consider RHEL or CentOS instead of Ubuntu.
I’m a big fan of Talos; I've been using it for my production and QA clusters on VMware with zero issues since I set up my first cluster two years ago!
Last time I checked, Bottlerocket had some issues with UEFI vs legacy BIOS boot. You might want to consider Kairos as a hybrid OS alternative if you need to support various hardware configurations, which makes Talos a tougher sell in your case.
What’s the reasoning behind not using Talos? I’m genuinely curious about your team's concerns.
Some team members are worried about the troubleshooting capabilities of Talos, but it’s a solid choice overall.
Thanks for your insights! We're still in the design phase and weighing opinions. Some team members are interested in Talos for its security features, but they're worried about troubleshooting complexity. Bottlerocket is under evaluation, but we have significant concerns about its on-prem adoption.