I'm part of a 40-person SaaS team primarily consisting of engineers, and we're running everything on AWS EKS along with GitHub Actions and ArgoCD. As we've transitioned from a startup to something more akin to an enterprise, the application security (AppSec) challenges have escalated. We've got around 130 microservices spread across three EKS clusters, and while software composition analysis (SCA) works decently during pull requests (PRs), our dynamic application security testing (DAST) and interactive application security testing (IAST) processes are in disarray. Scans are irregular, and scaling has been a nightmare. We're dealing with NodeJS and Go apps that frequently trigger OWASP Top 10 vulnerabilities. It feels impossible to shift left with only myself and one part-time developer advocate to manage notifications. Additionally, our use of a monorepo complicates context management. With SOC2 and PCI compliance on our shoulders, we can't afford to ignore runtime or Infrastructure as Code (IaC) vulnerabilities any longer. How are other mid-sized teams effectively handling the shift-left approach for AppSec? Are there tools or policies, like custom slack bots for triage, that can help? Any tips on EKS strategies to prevent risky deployments without hindering our speed? I've checked out demos, guides, and blogs, but nothing seems fitting for our current setup.
4 Answers
At a similar scale, we've recently subscribed to Wiz for managing our security policies. We treat our policies and any exceptions as code in our Terraform repository, which keeps everything transparent and manageable. We scan base images daily and allow for temporary ignores to be added by the on-call person if a scan fails, which helps us push new images out smoothly. Developers can also suggest application-specific ignores that come with an expiration date, ensuring we stay compliant while allowing some flexibility.
With so many microservices and just two people focusing on security, it's crucial to prioritize your efforts. Don’t attempt to scan everything equally; instead, categorize your services based on their exposure and the sensitivity of the data they handle. Focus DAST efforts on the services that are public-facing or involve PCI data. For EKS, consider using tools like Falco for runtime threat detection and the Trivy Operator for continuous image scanning—all within your cluster. It helps to generate VulnerabilityReport CRDs that you can easily manage without needing additional infrastructure. Since you're already using SCA effectively in PRs, you could integrate a tool like Semgrep for static analysis, which is quick enough for regular scans without hindering dev productivity. Plus, maintaining scan results as Kubernetes CRDs can streamline SOC2/PCI record-keeping.
Scaling DAST and IAST can be tough without proper orchestration. Some teams have had success by scheduling scans for off-peak hours or during nightly builds, funneling only the actionable alerts to Slack or Teams for better visibility. Otherwise, there's a risk that alerts get lost in the shuffle.
The key to success at your scale is implementing automated deployment gating with context-aware policies. Not every PR needs a comprehensive DAST or IaC scan; automatically blocking risky deployments—like those with severe OWASP findings or unapproved dependencies—using GitHub Actions and ArgoCD can really lighten the cognitive load. Start by ensuring critical microservices are prioritized, and gradually expand coverage across the board.
For shift-left initiatives, we utilize IntelliJ and VSCode plugins alongside pre-commit hooks, which help catch security issues before they even reach the PR stage.