I'm building my first Amazon EKS cluster and would appreciate feedback on the architecture. I currently separate workloads into three groups: System on On-Demand nodes for essential platform components and internal services; Application on On-Demand nodes for regular services that always maintain at least two replicas; and Application on Spot nodes for background workers, burst capacity, and extra replicas created during scaling.
The System group currently includes Grafana, Tempo, Loki, Prometheus, Alertmanager, Karpenter, Velero, LiteLLM, cert-manager, Kong, KEDA, and n8n. The application groups contain our application services and workers.
I'm also using Kong for API gateway functionality, including rate limiting, protection for the login endpoint, security headers, and blocking endpoints that should not be publicly accessible. System workloads run on Graviton ARM instances, while application workloads use x86 because some legacy services are still architecture-specific.
Does this three-tier workload separation make sense? Are there important components, operational practices, or failure scenarios I should consider before expanding the cluster?
5 Answers
The separation is sensible, especially keeping critical services on On-Demand capacity and using Spot for interruptible workers and burst replicas. I would focus less on adding more controllers and more on verifying the boundaries during failures. Test Spot interruptions, node replacement, Karpenter outages, and observability or ingress failures to make sure one problem doesn't take down unrelated workloads.
For a first cluster, keep the baseline fairly simple: private nodes, managed add-ons such as the VPC CNI, CoreDNS, and kube-proxy, and IAM roles for service accounts from the beginning. Pin Kubernetes and add-on versions, plan the VPC and subnet layout carefully, and avoid introducing a service mesh until the Deployment, ingress, autoscaling, and IAM paths are reliable. Most early EKS problems come from networking and permissions rather than a lack of Kubernetes components.
I would add a GitOps tool such as Argo CD or Flux, along with External Secrets Operator. Git should be the source of truth for manifests, while secrets can come from AWS Secrets Manager or Parameter Store. For mostly stateless workloads, rebuilding the cluster from Git may be more reliable than depending heavily on Velero. Stateful operators with native backups can handle databases and similar services more directly.
Make sure Karpenter itself is not running only on the nodes it provisions. Keep Karpenter, CoreDNS, the ingress controller, and other capacity-critical components on a small, stable managed node group or another capacity source that cannot be consolidated away. Otherwise, a bad consolidation event or major Spot interruption could remove the components needed to bring new capacity online.
A small always-available node group with two or three nodes is a practical safety net for those core components. Set appropriate requests and disruption policies so they remain schedulable during node events.
Kong is a reasonable choice when you need API gateway features such as rate limiting and policy controls. An ALB controller is simpler for basic HTTP or HTTPS ingress, but it doesn't replace a full API gateway. Since Velero and your system workloads run on ARM while applications run on x86, test backup and restore workflows across architectures, especially for any images or plugins that are architecture-specific.
That approach also makes disaster recovery easier to test. A fresh-cluster rebuild is often more useful than assuming a backup restore will work without regularly exercising it.