I'm currently running a Caddy pod in my Kubernetes cluster that uses a Persistent Volume Claim (PVC) to store TLS certificates. The pod has node affinity set, meaning during a rolling update, the new pod can be deployed on the same node and thus use the same PVC. However, I've been running into issues: if the original node runs low on resources, the new Caddy pod can't start up properly. Since Caddy mainly stores TLS certificates, I'm wondering what the best approach is to avoid these problems. One potential solution I considered is configuring Caddy to save its TLS certificates on AWS S3, which would allow me to remove the node affinity. But would that slow down my application? If S3 isn't suitable, is relying on a PVC with ReadWriteMany (RWX) the only option?
3 Answers
You might want to look into using AWS EFS with the EFS CSI driver for your PVC. This eliminates the need for node affinity, allowing any pod to be scheduled on any node as long as your application is structured to support that. And don't forget, Kubernetes Secrets are also a solid option for storing those certificates.
I recommend using Kubernetes Secrets to store your TLS certificates instead of a PV. If you're using Let's Encrypt, definitely go with cert-manager for simplicity and better management.
Instead of using Caddy directly for provisioning TLS certificates, consider using cert-manager. It can handle certificate management for you, and Caddy can then just mount the secrets that cert-manager generates. This way, you avoid all the hassle with PVCs and node affinities.
I’m worried about relying on a third-party like AWS. I’ve set up MinIO within my cluster. Can I potentially use that to store my secrets instead? If not, I'm okay falling back to Kubernetes Secrets.