I'm looking to gather some fundamental metrics about the services I'm running, specifically to find out how often they're being used and who's connecting to them. I realize that ideally, I'd instrument the code and use APM, but that's just not feasible for some of my projects. I'm after basic layer 4 statistics like the number of TCP connections per second and packets per second. Ideally, I'd like to know details like which pod or deployment is accessing a service. Some of my research suggests that a service mesh might be what I need, but that seems excessive for my requirements. Any suggestions on tools or methods for achieving this visibility without going overboard?
5 Answers
If you're using a Cloud service, consider an eBPF-based observability platform. It can surface metrics you're looking for quite effectively. Additionally, if your service is behind an Ingress, check out the `/metrics` endpoint from the Ingress Controller for traffic stats and TCP connection information, as long as it’s set up correctly.
Using OpenTelemetry (Otel) collectors on your nodes combined with eBPF filters could be a good path if you want to avoid heavier instrumentation. You can set up a free Grafana Cloud account and forward telemetry data there. Over time, you could begin augmenting your code if it makes sense. Some off-the-shelf applications might already emit Prometheus metrics that OpenTelemetry can grab! Also, if you use Nginx, compile it with metrics enabled and scrape that data with OpenTelemetry collectors. Just make sure your deployment strategy keeps metrics from being double-registered. Segmenting your service route topology can help with this too!
Have you looked into the kube-prometheus-stack helm chart? It could simplify deploying monitoring tools right in your Kubernetes cluster.
You should remember that Services in Kubernetes work like DNS and iptables, so there really isn’t a lot to instrument directly. However, iptables provides counters and logging, and it’s worth checking if Kubernetes exposes any of that. It might not be readily visible though!
That’s true! I was hoping for some hidden Kubernetes magic to expose those stats more easily.
A classic setup would be to use Prometheus and Grafana for monitoring your services. You could even add Loki for log aggregation to get a comprehensive view of what’s happening.
Absolutely, eBPF is a great way to go. You might also want to look into using Cilium CNI for added functionality!