We received a notification that one of our AWS resources may have been involved in activity resembling scans against remote hosts. The implicated resource appears to be behind a NAT Gateway. I enabled VPC Flow Logs, turned on GuardDuty, and started malware scans on recently created resources. What other areas should I investigate while the logs accumulate?
Update: The cause turned out to be a supply-chain compromise in a quickly assembled proof-of-concept application. Fortunately, it was isolated in a separate environment, and all affected credentials have been rotated.
3 Answers
Treat this as a potential compromise rather than assuming it is just a misbehaving application. Use the NAT Gateway information to identify the account, region, VPC, and workloads using it, including resources in other regions. Review CloudTrail closely for unexpected API calls, logins, credential use, new users or roles, security-group changes, and newly created instances. VPC Flow Logs can help identify which private host initiated the scanning traffic, but you may need a way to filter or visualize them because they become noisy quickly.
Also investigate the application and its dependencies, not just the AWS infrastructure. A compromised package, build tool, container, or deployment artifact can explain outbound scanning even when the account itself was not taken over. Review dependency and build history, compare deployed files with known-good versions, check newly introduced packages, and rotate any credentials that the affected environment could access.
If you identify a suspicious instance, isolate it immediately by removing its network access or stopping it. Preserve evidence before changing too much: take snapshots, create read-only copies on a temporary investigation host, and inspect them for malware, altered binaries, persistence mechanisms, cron jobs, systemd services, login history, command history, and unusual processes. Avoid assuming that cleaning the instance in place makes it trustworthy.
For anything production-like, I would treat the instance as burned. Capture the evidence you need, then replace it from a known-good image and migrate only verified application data. Rebuilding is generally safer than trying to clean a compromised host.