I'm setting up TLS certificate management for a production service running in Kubernetes, where certificates are mounted via Secrets or ConfigMaps. I want my Go application to automatically detect and reload these certificates when they change, especially during cert-manager rotation. I've explored two main strategies:
1. Utilizing fsnotify to monitor the parent directory of the mounted certificates (like /etc/tls) to trigger an in-app reload when changes occur. This is effective since Kubernetes replaces the entire symlinked directory during updates.
2. Implementing a sidecar container (like reloader or cert-manager's webhook approach) to detect certificate changes and either send a signal (like SIGHUP or HTTP) to the main container or restart the pod.
I'm curious to know what has worked best for others in production. Are there any potential issues with inotify-based approaches, especially across different distros or container runtimes? And do you prefer the sidecar pattern for reliability and separation of concerns?
7 Answers
Check out Reloader on Docker Hub. It's the most pulled image and really popular for this sort of task!
You didn't mention how TLS is terminated. Many ingress controllers will reload automatically when certs change, so keep that in mind!
The TLS termination is done by our Go proxy app specifically because we prefer not to use ingress controllers for TCP traffic due to additional functionalities.
I use Reloader to simplify things. It handles the restarts and reloads for me without any hassle. Just a couple of annotations and it works like a charm!
Thanks for sharing! Does Reloader actually restart your app when the secret changes, or does it trigger a reload of the certs without needing a restart?
I recommend using fsnotify on the parent directory (e.g., /etc/tls). It's quick, prevents restarts, and consistently works well with Go. Just make sure to watch the directory, not individual files, due to how symlink swaps work.
For a more robust solution focusing on separation of concerns, consider using Linkerd with mutual TLS (mTLS).
If your Go proxy can handle certs in memory, you could create a custom controller to notify it when certs change. Honestly, it could be a simple bash script in a container that does the job!
That’s what I’m leaning towards too! Sounds like you're discussing using SIGHUP. I'm curious if anyone's had experience using fsnotify instead of signals, and what perks or drawbacks they noticed.
How exactly does the certificate get into the container?
We mount the Kubernetes Secrets as a volume, so the certs are available right there.
He’s likely terminating TLS inside the container; otherwise, he wouldn't need to worry about reloading the cert. But it's a valid question of why they are doing it that way. Is it about trust in your cluster or needing mutual TLS?