I'm dealing with a perplexing networking issue on my EC2 instance running Amazon Linux 2023. When I try to curl port 8080 from my laptop, I receive an "Empty reply from server" message, yet when I test it from the instance itself (using localhost), I get a "Connection refused" error. It appears that nothing is listening on port 8080 at all. Here's a summary of what I'm seeing:
- From my laptop (external): `curl --connect-timeout 2 http://52.201.X.X:8080/` returns: `curl: (52) Empty reply from server`
- From the EC2 instance (localhost): `curl --connect-timeout 2 http://localhost:8080` returns: `curl: (7) Failed to connect to localhost port 8080 after 0 ms: Could not connect to server`
- A check using `ss -tlnp` shows that there is no service listening on port 8080.
In contrast, when I try a different port, such as 8088, it times out as expected due to security group settings blocking it.
Here's what I've looked at so far:
- Docker is installed but hasn't been running any containers. I stopped Docker, and the same issue persists.
- iptables NAT table shows no rules redirecting port 8080.
- There's an Elastic IP attached to the instance.
The "empty reply" implies that something is making the TCP handshake before it closes the connection, but since nothing is listening on the instance, I'm puzzled about what could be accepting the connection. Any insights would be appreciated!
5 Answers
Have you checked the security groups and the Network ACLs on the subnet? You mentioned you see nothing wrong there, but it’s easy to overlook something.
Could it be that you’re on a corporate network or using a firewall that performs man-in-the-middle (MITM) actions on your HTTP requests? That might explain why you're getting unexpected responses.
I figured it out! I was still connected to NordVPN earlier.
Although you’ve figured it out, just a thought: if the inbound SSH connection to your instance came from a VPN, that could cause silent drops if the security group wasn’t configured to allow it from outside the VPN. Just something to consider for future troubleshooting!
If nothing is listening, something external could be intercepting it. This could be local proxies, antivirus software, or even your ISP affecting how your connection behaves.
Yeah, I realized that I was still connected to my VPN. I think there was a firewall rule on that exit node messing with things.
I suggest using tcpdump on both your laptop and the EC2 instance to see which packets are flowing. It might help clarify where the blockage is.
Also, consider running a TCP traceroute to diagnose the connection path.
Ah, I did check that! The security group only allows ports 22 and 80, and the NACL is wide open. Interestingly, curl from inside the EC2 instance to the public IP times out, but it doesn’t time out from my laptop.