Hey everyone! I've been working at a company heavily using AWS for a few months. There's a mess of old infrastructure with various versions of Terraform and ArgoCD because many engineers have come and gone. I'm trying to clean things up and tackle our cost management issues with EKS, which feels complicated and overly expensive for us. I've had experience with Rancher, RKE2, and K3s, but I'm looking for a Kubernetes distribution that's straightforward and "just works."
One thing I struggle with is how EKS upgrades work. While it's advertised as easy, the reality is that after upgrading the control plane and managed nodes, I still have to upgrade all the addons and helm charts manually—something my predecessors overlooked. I've worked with RKE2 in the past and built a complicated upgrade flow that turned out to be fragile. Right now, I'm considering using Talos but I've had limited success in my home lab.
Any recommendations for a Kubernetes setup on AWS that effectively uses ASGs for on-demand instances and Karpenter for spot instances while being easy to upgrade?
4 Answers
If you find EKS complicated, be prepared for a bumpy ride elsewhere. From your post, it seems like cost isn't just an EKS problem. Focus on your node setup. Remember, Kubernetes upgrades are complex no matter where you host it. Maybe gaining more experience with K8s and AWS could help before making drastic changes. Also, have you considered ECS or Fargate as alternatives?
Switching from EKS can help save costs, but you should assess if the control plane is really your biggest expense. Depending on your team size and expertise, managing self-hosted K8s could add complexity that may not be worth it. Plus, AWS support is a lifesaver if something goes wrong with EKS, which is a big plus!
When it comes to ASGs or Karpenter, I’d lean towards Karpenter. It can help reduce costs by choosing cheaper nodes and optimizing placements, but I'd keep ASGs for stable core components. Having a blend can work well for normal workloads!
I have always used both ASGs AND Karpenter. I need some stable nodes, managed by ASGs, to run core components, while I can use Karpenter provided nodes for the 'normal' workloads.
It really depends on your team’s size and the scale of your node setup. For smaller teams, self-managing could be a resource drain, but if you’re running larger nodes, it may work well. I’d suggest looking into Flatcar or Talos Linux for a smoother experience!
That’s not an issue, I can manage a fleet of k8s by myself, with the right tools. I will take a look at Flatcar, thanks!
TL;DR it won't.