Hi everyone! I'm in the process of setting up a log aggregation system using Grafana, Loki, and Promtail. I've managed to configure Promtail to pull logs from both the VMs and Kubernetes pods, but I'm struggling to find a straightforward way to capture Kubernetes event logs as well. Does anyone have any lightweight solutions they could recommend?
5 Answers
OpenTelemetry is definitely a great choice!
I used to rely on the kubernetes-event-exporter GitHub repository, which worked well but hasn’t had much recent maintenance. However, there are plenty of forks out there, so those might be worthwhile checking out. Nowadays, I use Metricbeat to capture and send not just events, but other metrics to Logstash.
Consider setting up OpenTelemetry and using Loki as your target. It's pretty flexible, and if you want to switch solutions later, it's super easy. I recently transitioned to SigNoz with just a few lines of changes!
When you say "k8s logs," are you referring to logs from the pods? Because that's what comes to mind for me! 😅
Promtail is actually deprecated and doesn’t support event logs. If you're building a new setup, you might want to look into alternatives like Alloy or Fluent Bit. Personally, I've found Fluent Bit really solid since it can connect to the Kubernetes API to gather events. You can’t just read them from a log file since they’re fetched via the API.
Oh wow, I didn't know Promtail was deprecated! I've been following some old guides. I'll check out Alloy for now, sticking to the Grafana stack.
Oops, I meant Kubernetes events! Thanks for the clarification!