We run a Lambda function from a container image stored in ECR, with messages arriving through SQS. If processing fails, messages eventually go to a dead-letter queue, and we use that queue to monitor message-delivery problems.
Recently, an incorrect ECR lifecycle policy deleted the function image, causing the Lambda to stop processing messages. The policy has been corrected, but we want to test our alarms by simulating the same failure.
The function cannot be created without a valid image, and deleting the image after creation does not cause existing invocations to fail—the function continues running. Changing the description, memory, environment variables, or version also has not reproduced the problem.
How can we force the next invocation to perform a true cold start or otherwise make the service fetch the container image again from ECR? Is the image cached by the Lambda service, and if so, how can this missing-image scenario be tested reliably?
3 Answers
For the monitoring goal, testing the dead-letter-queue alarm separately may be more dependable than trying to reproduce an internal image-cache expiry. You can use a controlled test message or a deliberately failing test path in a non-production setup, then verify that the message retries and eventually reaches the DLQ. The actual missing-image scenario depends on service-side caching and cannot be forced on demand.
A Lambda cold start usually does not mean downloading the container image from ECR every time. The service can cache the image independently of the execution environments, so deleting the ECR image may not have an immediate effect. There is no documented way to force that image cache to be purged or to control how long it remains cached.
For a realistic failure test, remove the image or revoke the function's permission to retrieve it, then wait for the cached copy to expire and for Lambda to need the image again. The exact cache lifetime is not something you can rely on or configure. Changing an environment variable or similar function setting does recreate execution environments, but it does not necessarily force a fresh image download.
A normal cold start is simply the creation of a new execution environment because no suitable idle environment is available. It does not necessarily involve fetching the function image from ECR. Lambda may already have the image cached, and the cache is managed internally with no useful visibility or manual purge control.
Changing the architecture, such as switching between ARM and x86, or changing the runtime configuration, can cause new environments to be created. Those are useful ways to test initialization behavior, but they will not reliably reproduce a missing-image error.
That explains why deleting the image did not make the function fail immediately. I was assuming every cold start downloaded the image again. Is there any known or guaranteed cache duration?