Karpenter can't create new nodes in us-east-1. DescribeLaunchTemplates is returning 503 RequestLimitExceeded errors, followed by a retry quota exceeded message. Other regions are working. How can I tell whether this is our API usage or a wider AWS issue?
Karpenter is hitting EC2 API rate limits in us-east-1. What should I check?
Answers
Check AWS Health for the affected region and EC2 service first. Similar failures were reported across multiple accounts and on other EC2 APIs, including accounts without EKS, so this may not be specific to Karpenter.
That doesn't establish the cause of your errors, though. If there is an event, check its current status and details rather than assuming a closed event means everything has recovered.
Reply to this answer
The failed call is DescribeLaunchTemplates, so start there rather than assuming EC2 instance capacity is the problem. Look at when the errors began and whether other workloads or tools in the same account and region are making the same API calls.
The retry quota message is a separate clue: the client has run out of retry capacity after failures. It doesn't by itself tell you whether your account's request rate or a broader service issue caused the initial errors.
Reply to this answer
If the errors keep going and Health doesn't explain them, open an AWS support case with the affected region, timestamps, API operation, status code and request IDs. That gives support something concrete to investigate without guessing at the cause.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions