For the last couple of hours, our Karpenter installation has been unable to create new nodes in us-east-1. Other regions are working normally, but EC2 API calls are returning 503 RequestLimitExceeded errors, including DescribeLaunchTemplates and launch template operations. Karpenter is also reporting that its retry quota has been exhausted. Is anyone else seeing this across multiple accounts, or did AWS change its API limits? AWS has since posted an incident about increased error rates for EC2 Launch Template APIs in us-east-1, which may also affect services that depend on launch templates.
3 Answers
The issue seems to be recovering gradually. One quick check is to open the EC2 Launch Templates page in the console. If the templates load normally, Karpenter may be able to create nodes again. If the console still returns a rate-limit error, waiting for AWS to finish resolving the incident is probably the only practical option.
Yes, this appears to be an AWS-side incident rather than a Karpenter configuration problem. People are seeing failures across multiple accounts, including accounts that do not use Kubernetes, and several EC2 APIs are affected. The regional service status indicates elevated error rates for Launch Template operations in us-east-1.
The failures are not limited to Karpenter. Other customers are reporting intermittent problems with different EC2 APIs in the same region, so workloads that rely on launch templates may also be affected. Monitoring the AWS service health status and retrying after recovery is the safest approach.
That matches what we observed: once the Launch Templates page started loading again, Karpenter began successfully launching new nodes.