We run a public-facing .NET e-commerce platform using ASP.NET Core MVC, Angular, SQL Server, Elasticsearch, RabbitMQ, and IIS across several domains and subdomains. The site also receives substantial search-engine crawling and traffic from unidentified bots.
We have seen incidents where CPU and RAM looked normal while the application was slow or effectively unavailable. Other problems have included TCP exhaustion, Elasticsearch performance issues, excessive bot traffic, and slow requests that were not visible in basic infrastructure dashboards.
For a system like this, which dashboards and alerts would you consider essential? I am especially interested in application metrics such as request rate, latency percentiles, status-code rates, exceptions, thread-pool starvation, garbage collection, and allocations; IIS metrics such as queues, connections, worker-process health, recycling, and failed requests; network indicators such as connection failures, TIME_WAIT, bandwidth, client patterns, and suspicious user agents; and dependency metrics for Elasticsearch, SQL Server, and RabbitMQ.
What production metrics have proven especially valuable only after an incident? I would prefer practical operating experience over a generic monitoring checklist.
3 Answers
Start with user-visible service health rather than only host metrics. Monitor the application externally from more than one location, including a real or representative transaction, and track request rate, p50/p95/p99 latency, timeout rate, 4xx/5xx rates, and availability. Break those numbers down by endpoint, domain, status code, and dependency where possible.
For IIS and .NET, useful saturation signals include request-queue depth, active connections, worker-process restarts, application-pool recycles, thread-pool starvation, GC pauses, allocation rate, and outgoing connection-pool exhaustion. Add structured logs and distributed traces so a slow request can be followed through SQL Server, Elasticsearch, and RabbitMQ. Alerts should focus on SLO symptoms and sustained saturation, not every individual warning.
For the dependencies, monitor both health and pressure. SQL Server needs query duration, blocking, deadlocks, wait statistics, connection-pool usage, and storage latency. Elasticsearch needs cluster health, JVM and heap pressure, search latency, rejected or failed queries, slow logs, unassigned shards, and disk watermarks. For RabbitMQ, watch queue growth over time, consumer count, publish and delivery rates, processing latency, memory alarms, and dead-letter volume.
Queue length by itself can be misleading, so alert on whether a queue is draining and how old the oldest message is. The same principle applies to request queues and connection counts: trends and saturation are usually more useful than a single threshold.
Do not overlook traffic shape and failure handling. Track connections by state, connection failures, ephemeral-port or equivalent connection exhaustion, bandwidth, top clients, user-agent patterns, and requests to clearly suspicious paths. Rate limits, caching, bot controls, and load-shedding can prevent abnormal crawlers from consuming capacity.
The exact dashboard depends on whether the system is on-premises or cloud-hosted and whether a layer-7 load balancer is in front. I would also alert on load-balancer 4xx/5xx rates and latency. After each incident, perform a root-cause review, turn the missing signal into a dashboard or alert, and document the troubleshooting steps. Monitoring without a repeatable incident and post-incident process tends to become noisy rather than useful.
A practical setup usually evolves this way: begin with external availability, latency, errors, and dependency health, then add the saturation metric that explains each recurring incident. Alert on symptoms first and use logs, traces, and detailed counters for diagnosis.
A synthetic check is particularly valuable here. A server can respond to a basic health URL while search, checkout, or another critical workflow is broken.