I'm trying to understand what teams consider normal for production observability. Once an application is live, do you actively analyze incoming traffic patterns, or do you mainly rely on infrastructure and application metrics?
For example, if a service suddenly reaches 80% CPU, would you investigate top client IPs, user-agents, requested endpoints, bots versus real users, and unusual traffic spikes? Or is that level of analysis mostly limited to larger organizations?
If you do this kind of monitoring, what does your stack look like—access logs from Nginx, Traefik, or IIS; Loki, ELK, or OpenSearch; Splunk, Datadog, or something else? I'm trying to figure out what production teams generally keep visible all the time versus what they investigate only after an alert fires.
5 Answers
This isn't really about organization size. It's more about whether the team has previously been burned by missing data. You don't need to analyze top IPs all day, but having searchable access logs when an alert fires can save hours of guessing. Bot-versus-human analysis may require user-agent, ASN, or real-user telemetry rather than ordinary metrics.
Metrics tell you that something is wrong, logs help explain why, and APM traces show what happened inside the application. Prometheus and Grafana are common for infrastructure and application metrics, while Nginx or Traefik logs sent to Loki, ELK, or OpenSearch make traffic investigations practical. An APM tool can then connect the problematic endpoint to slow database calls or other code-level causes.
Don't only monitor traffic that arrives. Scheduled jobs and expected outputs need monitoring too, because a job can run successfully while producing nothing, leaving normal dashboards green. Expected-volume checks and critical-user-journey SLOs catch failures that request metrics alone won't reveal.
For an 80% CPU spike, I'd first check whether users are affected: latency, errors, queue depth, throughput, recent deployments, and dependency health. Then I'd compare the request rate and traffic pattern with the CPU timeline. A threefold increase in legitimate traffic, a bot loop, and an inefficient query can look identical on a CPU graph but require completely different fixes.
Most teams don't constantly watch every traffic dimension. Keep the basics—request rate, error rate, latency, saturation, and perhaps endpoint and status-code summaries—available through your metrics. When one of those signals goes bad, pivot into access logs to inspect IPs, user-agents, bots, and individual requests. High-cardinality data is expensive and usually isn't useful as an always-on metric.
The hard part isn't collecting every possible field; it's deciding which signals people will actually use. Alerts should reflect real user or business impact rather than exist just because the data is available.