How to Troubleshoot WebSocket Upgrade Issues with NGINX Ingress?

I'm working on setting up WebSocket connections using an NGINX ingress in a bare-metal Kubernetes environment, but I'm having trouble with upgrade requests not going through properly. Here's the setup I'm working with:

- A bare-metal Kubernetes cluster
- An external NGINX reverse proxy forwarding to a MetalLB IP
- The MetalLB directs traffic to the NGINX Ingress Controller, which then forwards it to a Node.js `socket.io` server inside the cluster on port 8080.

The traffic flow looks like this: **Client → NGINX reverse proxy → MetalLB IP → NGINX Ingress Controller → Pod**.

The problem arises when I try to establish a WebSocket connection. I can successfully do a direct curl to the pod via `kubectl port-forward`, and I get the right response:

HTTP/1.1 101 Switching Protocols

However, when going through the ingress path, the response is:

HTTP/1.1 200 OK
Connection: keep-alive

This indicates that the connection is falling back to plain HTTP, leading to an immediate closure, and the upgrade does not occur.

I've configured the Ingress with the necessary annotations to force WebSocket upgrades and set appropriate timeouts. Here's part of my Ingress YAML:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: websocket-server
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
spec:
ingressClassName: nginx
rules:
- host: ws.test.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: websocket-server
port:
number: 80
```

Additionally, here's the relevant part of my external NGINX proxy configuration:
```
server {
server_name 192.168.1.3;
listen 443 ssl;
client_max_body_size 50000M;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
...
}
```

Despite following the NGINX documentation claims that WebSockets should work out of the box, I'm still facing issues. If anyone has managed to troubleshoot this or knows how to confirm if the NGINX ingress configuration is being applied properly, I'd really appreciate the advice!