I'm trying to bootstrap Ubuntu optimized nodes on EKS with Karpenter, but I'm running into issues. Based on prior experience, I can bootstrap these nodes on different clusters using launch templates and userData scripts. However, now that Karpenter is involved, the situation has changed. I've confirmed that all security groups, subnets, and tags are in place, and I can successfully curl to the cluster endpoint.
I've checked everything related to permissions, as well as the Karpenter role and service account configurations, but the kubelet on the nodes won't start. While the instances are being built in AWS and show up in the node pool, they aren't joining the cluster and just remain in an unknown state. When I inspect the instances, I find that kubelet isn't running, and I can't find any relevant errors in the instance or in the controller pod logs.
I've explored various userData scripts, including some pseudo-code examples, but to no avail. I'm starting to think that using the canonical optimized AMIs might not be feasible, and instead, I might have to use a stock AMI and set it up from scratch. Is there something I am overlooking that could resolve this issue? Please, no generic chatbot scripts. I've tried those and they generally lead to outdated or irrelevant information.
2 Answers
Are you sure you need to use Ubuntu AMIs? I ask because Karpenter generally works better with Amazon Linux. It might be simpler to go with that unless you have a specific reason for Ubuntu. Have you considered switching to a standard AMI just for troubleshooting?
I noticed you mentioned not finding errors related to kubelet. Did you check the system logs thoroughly? Often, kubelet startup issues stem from misconfigured userData scripts, especially under Karpenter. Also, make sure your networking setup allows kubelet to communicate with the EC2 endpoint.
That's not really what they're asking! The goal is to find out if it's even feasible to use Ubuntu. We can discuss the AMI choice after addressing the core issue.