I'm managing over 50 AWS Lambda functions, and I want to set up a process to notify our team on Slack whenever any of the Lambdas experience a panic. My idea is to have all Lambdas send their panic messages to a specific CloudWatch Log Group. From there, I'd route notifications through an SNS Topic to Slack. I'm aware that I might need an additional Lambda function to format the messages correctly for Slack, but I'm wondering if there's anything major that I might be overlooking in this setup?
1 Answer
You’re mostly on the right path! Just a couple of tips:
- Make sure to consistently log all panic or error messages in your Lambdas so that you can effectively filter them in CloudWatch.
- Consider using a subscription filter to send only the relevant panic logs to the SNS.
- You'll definitely need a Lambda function to convert the CloudWatch logs into a clean format for Slack (using Blocks is a good choice!).
- Also, keep an eye on the rate limits for SNS and adjust your settings to prevent any timeouts if you're sending a lot of messages at once.
- As a bonus, you might want to store those logs in S3 or OpenSearch for better visibility over time; while Slack is great for immediate alerts, it won't help with long-term monitoring.
What do you mean by Blocks? Also, good call on using S3 for archival!