Hey everyone! I recently set up a Kubernetes cluster on three VMs in my vCenter environment, targeting a backend API to start with, and eventually plan to add a UI application too. Currently, I have three replicas of the API running with a NodePort for access, and I've implemented a secret for MongoDB credentials and a ConfigMap for some environment variables. I'm also using a Persistent Volume that all the nodes can access via NFS.
However, I've noticed that the logs from my Python Flask API are being written with some delay, which prompted me to look into a log collection solution for my Kubernetes cluster that could work for both my applications. I'm attempting to use Grafana with Loki and Promtail, along with MinIO hosted on an external VM in the same network. Unfortunately, I've faced multiple issues that caused Loki to crash when connecting to MinIO, even though it's configured correctly.
Given that I've seen reports about MinIO discontinuing feature development, I'm not sure if it's worth continuing with it. Can anyone suggest alternative tools for log collection that would be effective for my setup? Any insights would be greatly appreciated!
5 Answers
It sounds like the log delay could be related to your Python app. You might want to explore using the PYTHONUNBUFFERED environment variable to see if that helps. Keep in mind that the NFS sync could also contribute to this issue. I'm eager to build a comprehensive logging stack since I want to deploy more applications down the line.
Rather than using MinIO, consider sending logs to S3 directly with Loki. Promtail is working but is deprecated, with Alloy as its replacement. If Loki continues to crash, check if you need to allocate more resources for your Loki and scraper.
Consider swapping out MinIO for Ceph or Garage for better stability. Also, look into replacing Promtail, which is deprecated, with Alloy. It should give you a more solid logging stack. Good luck!
I’ll look into that. Thanks!
You might find some helpful resources at kube-logging.dev for setting up your logging framework!
You might want to set up an OpenTelemetry collector and route logs to Loki, then visualize them in Grafana. You can have your application send logs directly to the OpenTelemetry collector using cluster DNS. This collector can also handle metrics and traces, forwarding everything to Grafana Tempo. It could simplify your logging setup!
I’ll check on it. Thanks!
It can come from that, or NFS sync and others. I want to implement a stack for log collecting as I will deploy other apps in the future.