How to Set Up a Lambda in a Private Subnet to Receive SQS Events?
Asked by
TechExplorer42
I'm working with AWS CDK, setting up a VPC that has both public and private subnets. In the private subnet, I've got a security group configured to allow HTTP traffic from the VPC's CIDR block and also to permit communication within the same security group. I'm also running a Postgres RDS Aurora within this VPC.
I've created a Lambda function that's supposed to read messages from an SQS queue and write data to my database, but I'm not seeing the SQS messages hitting my Lambda. The SQS setup is very basic:
```
const sourceQueue = new sqs.Queue(this, "sourceQueue");
```
And my Lambda is set up like this:
```
const myLambda = new NodejsFunction(
this,
"myLambda",
{
entry: "path/to/index.js",
handler: "handler",
runtime: lambda.Runtime.NODEJS_22_X,
vpc,
securityGroups: [privateSG],
},
);
myLambda.addEventSource(
new SqsEventSource(sourceQueue),
);
// policies to allow access to all SQS actions
```
There are conflicting sources of information on whether I need a VPC endpoint for SQS. I've heard that creating a VPC endpoint could allow messages to reach my Lambda, but I'm unsure if this approach is secure enough given that I can't just create IaaS resources directly. Additionally, I'm concerned about how to restrict access to the queue while still allowing my Lambda to fetch messages and write to the DB securely. Am I on the right track or is there a better way to handle this, especially since I'm worried that using SQS might unintentionally expose my service? I also have two Lambdas that process JSON data that need to work in a specific order, and I want to manage message failures using a dead letter queue. Suggestions are greatly appreciated!
Answers
You’re on the right path, but don’t overthink it too much. SQS can be secured with IAM permissions, so it’s not strictly necessary to use a VPC endpoint if you manage those permissions well. Just make sure your IAM policies don’t allow unauthorized users to send messages to the queue.
Reply to this answer
So, AWS is a public cloud, meaning services like Lambda and SQS are accessible from the internet. When you move a Lambda to a private subnet, it loses access to those public services unless you set up additional infrastructure.
If your Lambda isn't receiving messages from SQS, you likely need a NAT Gateway or a VPC endpoint. The VPC endpoint is ideal here because it allows your Lambda to communicate with SQS without exposing it to the internet, keeping your traffic within AWS's network. I'd recommend setting up the queue with a resource policy that only allows access from your specific VPC endpoint and your Lambda's IAM role, plus using security groups to restrict traffic to your Lambda's security group only.
Reply to this answer
You really don’t need to have a VPC endpoint for this setup! The Lambda service will poll the SQS queue, not your code directly. Focus on setting up appropriate IAM permissions for your Lambda function.
Reply to this answer
If you need your Lambda to interact with both your private DB and SQS, keeping it in a private subnet makes sense. Just ensure everything is configured correctly on the IAM side. If the messages are arriving out of order, consider using a dead-letter queue to handle retries effectively. Just remember, if your colleagues have broad access, they could still interact with SQS unless you restrict IAM permissions appropriately.
Reply to this answer
To clarify, as long as your Lambda has the right IAM permissions, it doesn’t need a VPC endpoint to use SQS as an event source. When SQS triggers your Lambda, it handles the internal polling. Just ensure your Lambda is set to the private subnet and the necessary security group is in place for accessing your DB - that’s really the primary concern if you’re avoiding public exposure.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions