Struggling with Setting Up Logging for My Kubernetes Cluster
Asked by
TechieTurtle52
I've recently set up a small on-prem Kubernetes cluster for my work and we're about to launch our first application. Up until now, we haven't implemented any logging or alerting systems, but it's crucial that we do so before going live to avoid flying blind. After some research, I found that the ELK stack and LGTM stack are popular options. LGTM seems to be favored due to its ability to eliminate some of ELK's challenges. I have experience with both Elastic/Kibana and Grafana from other projects but haven't set them up from scratch myself. My decision was to go with Grafana and I'm currently trying to set up Loki with the official Helm chart in single binary mode with three replicas and a separate MinIO for storage. However, I'm finding the process pretty frustrating. The documentation for the Helm chart feels incomplete—the official page doesn't clearly explain the requirements for local persistent volumes, and the values reference is lacking key information, like the necessary http_config for skipping cert checks. I've had to figure things out mainly through Googling error messages which has been very annoying. Is this a common issue, or am I just struggling with this Helm chart and its documentation? I'm open to alternative solutions if anyone knows of any lightweight options aside from ELK and LGTM, as I see a ton of different projects on the CNCF Landscape regarding observability that are worth considering. Thanks for letting me vent a little!
Answers
You're absolutely not alone in feeling this way! I've been working with Kubernetes for several years now, and I can say that the observability landscape can be really tricky unless you're using a paid SaaS product. It's mostly trial and error, unfortunately. I’ve written a series of articles on some of these challenges, but haven’t published them yet. Just know you're not the only one going through this.
Reply to this answer
Add your answer
Answers are reviewed before publication.
CuriousBear21: Knowing it’s been tough for others definitely helps! It makes me feel like I’m not just messing things up. I’ve spent days navigating through docs and community forums for solutions, so it’s relieving to hear I’m not the only one! When you publish your articles, I’d love to check them out.