I'm facing challenges with cleaning up outdated images in AWS Elastic Container Registry (ECR). The 'describe-images' API shows all images in the registry but doesn't indicate which images are actively in use by ECS tasks, EKS pods, or running containers. This situation complicates the cleanup process because while lifecycle policies can automatically remove older images, they lack the capability to identify what is currently in active use. I want to avoid accidentally deleting images that are still needed for live workloads.
Here's what I've explored so far:
- Setting lifecycle policies to retain a specific number of the most recent images.
- Implementing untagged image cleanup scripts.
- Manually cross-checking ECS task definitions and EKS pods.
Has anyone developed a solid strategy for this? Do you maintain a list of in-use image digests, or is there a best practice out there that I might be missing?
5 Answers
A good practice is to ensure your services are using the latest images. Set up an ECR lifecycle to delete the oldest 10 images or something similar. Just be cautious with using the last pulled date, because it's unclear if an image was pulled by a production application or just for testing purposes. It might help to implement a tagging process where you tag images as in use when they’re pushed, and untag them when you update.
If your needs are specific, consider using Inspector for ECR scanning. It provides that information, but it can be pricey. For smaller setups, the other suggestions might suit you better. AWS Config can be another option; it shows which images are used for ECS task definitions, but it might miss EKS info. Depending on how automated you want everything to be, you can also query the Kube API for more details.
They recently added a field that shows the last pulled date, which can help since many things pull images regularly. However, it's not a perfect solution if you have long-running pods or instances. It's worth trying!
I keep the 10 most recent images and use separate repositories for different environments like dev, stage, and production. It also helps to have separate repos for base images that don’t change often and don’t take up much storage. Cross-checking ECS task definitions is wise, but it’s not foolproof without a solid process in place.
I just rebuild images from the latest commit every time and only keep the most recent ones per stage. My CI/CD pipeline updates the task definitions upon deployment, so there’s never an ambiguous in-use image. If I ever need an older image, I can recreate it directly from the Git commit. This keeps my ECR clean and allows me to maintain a minimal rollback window for production.
Thanks! I'll give that a shot.