I have a Lambda function deployed from an ECR image and triggered by SQS. I want to test whether my dead-letter queue alarm catches a failure if the image is deleted, but the function still runs after I remove the image. Is there a way to force the next invocation to cold start and reproduce that failure?
How can I force a Lambda cold start to test a missing container image?
Answers
A cold start and an ECR image retrieval are different things. A cold start creates a new execution environment, but Lambda can still use an image it has already cached and optimized. Removing the ECR image doesn't necessarily make the next cold start fail.
Changing an environment variable is a common way to trigger new execution environments for a function. It's useful for testing initialization behavior, but it won't reliably simulate the missing-image failure you're after.
Reply to this answer
For the exact image-access scenario, use a test function, remove the image or Lambda's access to it, and wait to see when the function fails. That is not a quick or deterministic cold-start test, though. The cached image can keep things working for some time.
Keep this isolated from production, since deleting an image or changing access could affect other functions that depend on it.
Reply to this answer
For the DLQ alarm itself, you could test a separate failure path instead of trying to reproduce the ECR incident. Make a test function fail when it processes a message, then check whether the message reaches the DLQ and the alarm fires.
That checks the delivery and alerting path. It doesn't prove that a missing image will produce the same behavior, so treat the image-access case as a separate test.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Mirello.7b:
That explains why changing the configuration didn't reproduce it. Is there a predictable wait for the cached image to expire?
Quincy:
The discussion doesn't establish a reliable cache lifetime, so I wouldn't build a test around a particular wait. It may take a while for the missing image or lost access to affect the function.