I'm building my first Amazon EKS cluster from scratch after previously managing clusters created by other teams. I'm migrating from Beanstalk and currently plan to use Karpenter for x86 and ARM capacity, Kyverno, KEDA, External Secrets Operator with Parameter Store, Kong as the ingress/API gateway, and Argo CD for GitOps. The cluster will use the default AWS VPC CNI for now. What components would you consider essential for a production-ready setup, and would you recommend switching to Cilium or Calico? I'd also appreciate advice on networking, observability, certificates, DNS, identity, and anything else that is easy to overlook when creating an EKS platform from scratch.
5 Answers
The default AWS VPC CNI is a reasonable starting point, especially if keeping the first cluster simple is a priority. It supports network policies now, and you can move to Cilium later if you actually need eBPF-based features, richer observability, or more advanced Layer 7 policies. For larger clusters, pay close attention to pod IP capacity, prefix delegation, subnet sizing, and whether ENI or route-table operations become a bottleneck.
Make observability part of the initial design. A Grafana stack with Prometheus, Loki, and Tempo is a solid option, and the OpenTelemetry Operator can help collect telemetry and provide automatic instrumentation when applications support it. Be clear about how application metrics will be exposed and collected, rather than only monitoring node and pod health.
A few commonly useful additions are ExternalDNS for automatically managing Route 53 records, the AWS Load Balancer Controller for ALB and NLB resources, and cert-manager for certificates needed inside the cluster. Decide early whether workloads will use IAM Roles for Service Accounts or EKS Pod Identity. Also treat the entire EKS bootstrap process, networking, and add-ons as infrastructure managed through IaC. If the VPC is shared, reserve appropriate pod CIDR space or use dedicated pod subnets to reduce the risk of IP exhaustion.
Your list is already more sophisticated than many production clusters, so I’d avoid installing every component on day one. Karpenter makes Cluster Autoscaler unnecessary, while Kyverno and KEDA should be driven by concrete requirements rather than treated as mandatory platform components. KEDA makes sense for an application that scales from SQS message depth, but Kyverno can wait until you have policies you actually need to enforce.
If your workloads will also provision or manage AWS services such as databases, Crossplane may be worth evaluating. It can be useful for giving application teams a consistent Kubernetes-based interface to external AWS resources, but it is another substantial layer, so I’d add it only if that workflow is genuinely needed.
We’ve seen very large clusters experience route churn and ENI attachment delays, so the default CNI can require careful planning at high pod counts. For a smaller initial environment, though, it may be the least disruptive choice.