Our production site becomes unreachable several times a day for roughly 1–2 minutes, then recovers without intervention. The stack is Windows Server with IIS, ASP.NET Core MVC and Web APIs, an Angular frontend, SQL Server on a separate 8 GB server, a three-node Elasticsearch cluster, and a separate monitoring server. Traffic reaches IIS directly without a reverse proxy, CDN, WAF, or load balancer.
External monitors in different geographic locations report socket connection timeouts rather than HTTP errors. The incidents occur both during busy periods and when traffic is light. DNS is fast, the hosting provider reports no infrastructure problems, Windows remains online, and neither IIS nor ASP.NET Core logs show failed requests. Previous SQL connection-pool exhaustion was addressed with caching, and current SQL metrics look healthy.
I am considering OpenTelemetry, distributed tracing, runtime metrics, HTTPERR logging, HTTP.sys and IIS queue monitoring, Windows Performance Counters, and dashboards correlating Windows, IIS, SQL Server, Elasticsearch, and application data. Before changing application code, I want to determine which layer is actually failing.
How would you investigate this systematically? What would you monitor first, and how would you distinguish a network, TCP, Windows, HTTP.sys, IIS, ASP.NET Core, SQL Server, or Elasticsearch problem? Are there specific tools or metrics that are especially useful for short socket timeouts that never appear in application logs?
5 Answers
Start by splitting the failure into separate paths. Run a small, timestamped probe every few seconds from the monitoring host against the public endpoint, the server's LAN address, a loopback endpoint, a static IIS file, SQL Server, and Elasticsearch. Record DNS time, TCP connection time, TLS time, time to first byte, and total response time.
If the public endpoint fails but the local endpoint works, investigate routing, firewalls, or the provider. If even a local static file fails while Windows is healthy, focus on IIS, HTTP.sys, the application pool, TCP resources, or host-level filtering. If IIS responds but dependency probes become slow, then investigate SQL or Elasticsearch. A small truth table from these probes is more useful than one large dashboard.
A reverse proxy on the same machine may improve HTTP-level visibility, but it will not record a connection that never reaches or is never accepted by the host. I would not add one as a substitute for packet captures and TCP evidence. First establish exactly where the handshake stops, then move upward through HTTP.sys, IIS, ASP.NET Core, and the dependencies.
I would check IIS and Windows events before instrumenting the whole application. Look for application-pool recycles, idle-timeout or rapid-fail protection, scheduled tasks, antivirus or filtering-driver activity, Windows updates, and resource exhaustion. Thread-pool starvation, a long garbage-collection pause, a full HTTP.sys queue, port exhaustion, or an overlapped recycle can all produce a short outage with little useful request logging.
Compare normal and incident-time values for queue length, current connections, rejected requests, process memory, CPU, GC pauses, thread-pool availability, TCP resets, and port usage. Also verify that the application pool is not being suspended after idle periods and that the host is not relying on DHCP for a critical service address.
Use separate health checks for basic reachability and dependency reachability rather than one endpoint that only returns healthy or unhealthy. A local probe can log whether the web process itself was reachable even when external monitoring failed. Similar checks for SQL and each Elasticsearch node can reveal whether the application is waiting on a dependency.
OpenTelemetry and distributed tracing are still worthwhile, but they only describe requests that IIS accepts. They will not explain a TCP handshake that never completes. If Elasticsearch is involved, verify that the client is configured with multiple nodes and can fail over instead of depending on one node.
A message saying “unable to connect” usually means the request may never have reached IIS, so application tracing cannot explain it. Capture evidence at the TCP layer during the next event. A circular packet capture using tools such as pktmon or netsh trace, synchronized with an external probe, can show whether the SYN arrived, whether the server sent a SYN-ACK, whether the connection was reset, or whether the handshake completed and TLS or HTTP stalled.
Also collect HTTPERR logs and five-second performance data for TCP connection failures and resets, active connections, HTTP.sys queue depth and rejected requests, IIS connections, application-pool recycles, process private bytes and handles, available memory, thread-pool state, garbage collection, and ephemeral-port usage. One-minute incidents disappear quickly if you only retain daily averages.

Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures