I'm running into issues with Lambda A invoking Lambda B asynchronously. While Lambda A is functioning well, Lambda B is showing a significant delay in processing async events. It's only getting 2 or 3 concurrent invocations, although we expect around 200. There don't seem to be any throttling issues with Lambda A. I'm starting to think that an ongoing outage might still be impacting the internal queue for async Lambdas. Has anyone else experienced similar issues or found a fix?
5 Answers
I had a few async invocations that came through around 11 AM, then they vanished for a couple of hours. Thankfully, they executed around 1:30 PM! Looks like things might be getting back to normal.
Yes, I just saw an update from AWS about 15 minutes ago. They mentioned they slowed down SQS polling for Lambda Event Source Mappings to recover from invocation errors, but now they're ramping it back up as the invocations become successful again. I had a hunch that was the case based on our metrics, but now it’s confirmed.
I've recently noticed some activity again with Lambda and SQS in my setup—some Lambdas finally invoked after being stuck for a while.
Just a heads up, us-east-1 is still listed as unhealthy. AWS Lambda is marked as one of the services still impacted. I wouldn’t rely on anything running there at the moment, especially for non-IaaS workloads.
It might help to adjust the timeout settings or increase the memory allocated to your Lambdas. After doing that, I was able to create a new execution environment, which resolved some issues for me.
That did help me get more than just 2-3 concurrent invocations, but I’m still getting a backlog of async events—it's about a 5:1 processing rate. Looks like I've got to sit tight for a bit.
I get that, but the problem is I can tweak the timeout and memory but our Lambda’s container images can't be changed. They take a while to spin up and then eventually fail. It’s a bummer since an update in the container would help with the async events we're dealing with.
EDIT: Just figured out I can actually modify the container images now!