Our team is currently using Google Kubernetes Engine (GKE) for our Kubernetes deployment, and I'm seeking advice on how to effectively manage application logs. Right now, we're outputting logs to stdout/stderr, but I've been asked to change that so the logs can be written to files and persisted to a Persistent Volume Claim (PVC). This requirement seems to add a lot of complexity to the application, like having to manage file storage and log rotation. I still want to ensure that if a pod crashes, I can retrieve its logs for troubleshooting. Are there more efficient approaches to managing logs in GKE? Any recommended resources or best practices would be greatly appreciated!
3 Answers
Setting up an ELK (Elasticsearch, Logstash, Kibana) stack could be beneficial. You can send logs directly or use a filebeat daemon set to gather logs from your pods. Just ensure that all your applications log in a consistent format, preferably JSON, and include custom fields for environment and application identification.
You might want to look into using a dedicated logging system like Stackdriver, Loki, or Elasticsearch. They can manage and analyze logs more efficiently than just writing them to files.
You should consider that logs are typically stored in files by default, even when they’re also sent to stdout/stderr. Other logging tools retrieve container logs this way. The real question is why you need to use a PVC for logs. Is it just for access reasons? GKE has built-in logging options, and it’s worth asking what problem you’re trying to solve here. If the challenge is accessing or filtering logs, adding another file may complicate things unnecessarily. Think about whether the storage and I/O costs are justified.
Exactly! Also, understanding your budget and the volume of logs you generate is crucial for determining the right solution.