I have an AWS Lambda function deployed from an Amazon ECR container image and triggered by SQS. If processing fails, messages eventually go to a dead-letter queue, so I created alarms for messages appearing there. This is meant to monitor the overall message-delivery path, not just the function code.
Recently, an incorrect ECR lifecycle policy deleted the image and the Lambda could no longer run. I have fixed the policy, but I want to test the monitoring setup by reproducing that failure.
The function cannot be created without a valid image, and deleting the image afterward does not immediately cause invocations to fail. I have also tried changing the description, memory, environment variables, and version, but the function continues to work. How can I force the next invocation to perform a true cold start, or otherwise simulate the situation where Lambda can no longer retrieve its container image from ECR?
2 Answers
A cold start does not necessarily download the container image from ECR. Lambda can cache the image within its service, and that cached copy may continue being used even after the ECR image is deleted. The cache duration is not something you can control or reliably observe.
Changing an environment variable or another function setting usually recreates execution environments and forces cold starts, but that will not help test a missing-image scenario if Lambda still has the image cached. To reproduce that failure, delete the image or remove Lambda's permission to access it, then wait until the cached copy is no longer available. There is no supported way to force immediate eviction of that image cache.
It helps to separate two meanings of cold start. Normally, a cold start means Lambda has to create a new execution environment because no suitable idle environment is available. The function's code or container image may already be cached by the Lambda service, so creating a new environment does not necessarily require downloading the image from ECR.
You do not have direct control over, or visibility into, when Lambda evicts that cached image. Changing the runtime architecture or runtime version can force new execution environments, but it still does not guarantee an ECR image re-download.
That explains why deleting the image did not fail immediately. Is the cache lifetime documented, or is it effectively unpredictable?