I'm dealing with a tricky problem related to a Network Load Balancer (NLB) in my staging environment where it reports healthy targets but doesn't seem to be forwarding traffic to my EC2 instances. Here's the architecture breakdown:
- **VPC A:** Where the NLB is located, it's a shared VPC.
- **VPC B:** Hosts my two Windows Server EC2 instances.
- **VPC Peering:** I've got it set up between A and B, with bidirectional routes configured.
### NLB Setup Details:
- Listeners for UDP (2020) and TCP (2021)
- Targets in the target groups for both protocols
- Health checks: the UDP group uses a TCP health check on port 2021
### Security Groups:
- NLB Security Group allows TCP 2021 and UDP 2020 from `0.0.0.0/0`
- EC2 Security Group permits the same protocols from `10.0.0.0/8`
### Observations:
I can connect to the EC2 instances directly using their private IPs, and both ports work fine. However, when I try to connect to the NLB's DNS name from my external IP, the connection times out. AWS shows both instances as healthy in their target groups.
### When I Ruled Out:
- The application itself is responding since direct IP tests work.
- Health checks are passing without issues.
- Hairpinning or loopback isn't a problem; I've tested from outside.
- VPC peering looks active, and the routes are configured correctly.
### The Question:
Why would an NLB report that its targets are healthy but not forward any traffic? Has anyone else faced this, particularly with UDP/TCP traffic across VPC peering?
5 Answers
One thing you might want to check is the DNS resolution for the NLB. Make sure it's correctly routing traffic from your external source to the NLB. Even if your targets are healthy, there could be a networking issue preventing your machine from reaching the NLB directly.
It's worth checking if there's any access control in place that's blocking traffic between the VPCs. Sometimes, simple Network ACL settings can prevent connections, especially since the instances are in distinct subnets.
It sounds like the core issue is that NLBs don't work well with VPC peering like ALBs do. For cross-VPC traffic to function properly with an NLB, you typically need to set up a PrivateLink endpoint. That's why your health checks might pass—they're coming from within AWS’s infrastructure—but actual client traffic can't get through.
To fix this, you have two options: either use AWS PrivateLink for cross-VPC access, or move your NLB into the same VPC as your EC2 instances.
You should verify if the applications on your EC2 instances support running behind a proxy. We ran into a similar issue, and the vendor informed us that the version we were using didn't support it, so we had to get a different license. If they do support it, consider using a VPC reachability analyzer to diagnose the issue.
Enabling VPC flow logs can provide insight into what's happening with the packets. Check if the network interface is receiving traffic and what the status of that traffic is—whether it's being accepted or rejected. Also, note that if your EC2 instances are multi-AZ, ensure the NLB is set to enable cross-zone load balancing; it's not enabled by default for NLBs, unlike ALBs.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures