I'm working on a complex observability setup where we have a single node running multiple applications alongside a k3s cluster. We're using a Vector agent to handle our metrics and logs, which scrapes data from a couple of APIs we've set up: node-exporter and kube-state-metrics. However, I've run into some issues.
The node-exporter collects node-level metrics, but because we're running a lot of other applications on the same node, it doesn't give me isolated information just for the k3s cluster. Meanwhile, kube-state-metrics doesn't provide current CPU and memory usage at the pod level, leaving me in a lurch on how to obtain pod-level metrics.
I've explored the kubelet /metrics endpoint to see if I could use the Vector agent to pull these metrics, but it hasn't worked out. I've also attempted to use metrics-server without success.
Can anyone tell me if we can scrape metrics from the metrics server, and if so, how to connect? Additionally, are there other exporters that can expose pod-level CPU and memory usage?
2 Answers
Regarding your queries, first, I don't think the metrics server can be scraped directly as it’s not built for that—it's meant for internal Kubernetes use and doesn’t provide Prometheus-compatible metrics. As for your second question, using Prometheus could help with gathering pod-level stats, pulling metrics from various sources like kubelet and node-exporter, but I get that you can’t use it due to the pull model. There are alternatives like cAdvisor which can expose metrics at the pod level, and you can also try checking if the Vector agent can be tweaked to pull from it.
You might find it tough to scrape from the metrics server since it’s designed more for just providing metrics to Kubernetes itself rather than being a source for external metrics tools. I’ve used cAdvisor before for pod-level metrics—it can be exposed via the kubelet’s metrics endpoint, but you’ll need to set up permissions properly. Just make sure to get the correct URL for cAdvisor, using something like `curl -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://:10250/metrics/cadvisor` might help.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures