When a production service starts using a lot of CPU, do you routinely look at things like endpoints, client IPs, user agents, and bots? Or do you mostly rely on infrastructure and application metrics until something looks wrong?
I'm curious what a practical setup looks like and how much detail is worth keeping all the time.
I'd start with request rate, error rate, and latency for each service, then use those signals to decide when to dig deeper. CPU at 80 percent isn't necessarily a problem if traffic is up and response times are still fine.
Keep status codes and normalized endpoint routes available in metrics, but be careful about label cardinality. Raw URLs, client IPs, and user agents can create a huge number of time series. Access logs are a better place to query those details when an investigation calls for them.
For a CPU spike, check whether request volume or the mix of endpoints changed. A surge in legitimate requests, a noisy crawler, and an expensive database query can all show up as high CPU, but they need different responses.
Access logs from a proxy such as Nginx or Traefik can help with top endpoints, IPs, user agents, and status codes. Something like Loki or OpenSearch makes those logs easier to query. Treat user agents and IPs as clues rather than a perfect way to separate bots from people.
Traffic analysis won't necessarily tell you why the application is slow. If the request rate looks normal but latency or errors climb, look at application performance monitoring, traces, dependencies, and recent deployments too. The useful starting point is whether users are seeing a problem, not just whether one infrastructure graph crossed a threshold.
Don't forget to monitor traffic that should have arrived but didn't. A quiet service can look healthy on error and latency graphs because there are no requests to fail.
The same goes for scheduled jobs. A job can finish successfully and still produce no useful output, so an expected run or output volume check may catch things that exit status and duration miss.
There isn't one depth that makes sense for every service. Pick signals the team will actually use, based on what the service does and how critical it is. You can collect a mountain of fields and still miss the one thing you needed to investigate.
Review those choices periodically. A small set of useful metrics plus searchable logs can be a reasonable starting point without building a full-time traffic analytics project.
Answers are reviewed before publication.
c.762+buttonledger:
Normalized routes matter here. A route template is much more manageable than a separate metric label for every URL containing an ID.