We've got a team of about 100 using EKS, primarily on EC2 nodes with some workloads running on Fargate. We're integrated with AWS services like RDS and S3, but after multiple security audits pointing out vulnerabilities, our leadership wants us to develop a solid hardening plan before expanding into more namespaces. Although we've tackled some basic AWS guides and played around with OPA policies, we've encountered issues with wide IAM mappings in aws-auth and potential pod escapes in our testing.
I remember the ChangeHealthcare breach from last year where attackers accessed their EKS cluster through a misconfigured IAM role and moved laterally through pods, putting patient data at risk. That's exactly the kind of incident we want to avoid.
I'm looking for insights on how to prioritize security. What best practices have you implemented in production, specifically around:
- Effective IAM and RBAC configurations (maybe IRSA examples?)
- Network policies and security groups for workload segmentation
- Image scanning and runtime checks that don't impact performance too much
- Monitoring systems that can detect anomalies or drifts early
- Hardening nodes and pod security standards
Have you come across any checklists or mind maps that have been successful?
4 Answers
It's crucial to know what specific gaps have been flagged during your audits. If there's anything you can share, it would really help. We recently discovered that our pods were running with overly permissive IAM roles due to a misconfiguration, and we need to address that quickly. I'm curious about the scanners you're using for these issues as well!
Start with the basics! Use IRSA and RBAC to enforce least privilege, and segment workloads using Network Policies and security groups. Integrate CI/CD with image scanning using tools like Trivy or Grype and implement runtime checks with tools like Falco or OPA policies. For node hardening, consider AWS AMIs or Bottlerocket. Audit logging and anomaly detection should also be a key part of your strategy. The goal is to have layered defenses rather than chasing every audit finding because most breaches stem from basic gaps, not exotic vulnerabilities. CIS Benchmarks for EKS can be really useful as a checklist!
It would be helpful if your team could proactively build upon the practices instead of just pointing out gaps after the fact. This ensures that security concerns are addressed in the development phase.
Honestly, switching to per-pod identity with IRSA can eliminate many lateral movement risks. When you stop using node-level IAM roles, a significant portion of potential security issues just disappears!
Totally agree! That seems like a game changer for making clusters more secure.