I'm running the ELK stack (Elasticsearch, Logstash, Kibana) on a Kubernetes cluster that I host on a Raspberry Pi 5 (8GB). Everything works perfectly right after installation: Elasticsearch initializes, Logstash connects with SSL using a CA cert from Elastic, and Kibana is up and running. However, I encounter a significant issue after a server reboot. The Elasticsearch pod gets stuck at 0/1 Running, and as a result, both Logstash and Kibana fail to connect. Even if I delete the Elasticsearch pod manually, the issue persists.
Looking at the logs, I see several connection error messages indicating that Logstash can't connect to Elasticsearch. Elasticsearch logs warn me that it's treated as a single-node cluster while it's configured to discover other nodes, suggesting I need to adjust the discovery settings. The only way I can get everything working again is to delete the PVC and redeploy the entire ELK stack. I've already verified that CPU and memory resources are adequate and that the CA cert is set correctly, but the Elasticsearch pod still doesn't become ready after a reboot. What could be going wrong?
1 Answer
It sounds like your Elasticsearch pod might be having issues with its storage. Have you checked the events for the pod and the PVC to see why it might be stuck? Running a "describe" command on the pod should give you more insights on what's blocking it from running. Also, ensure that nothing is changing on the volume between reboots.
Good call! It's crucial to check the pod events. Just updated my post with the relevant logs and PVC events for better clarity.