Hey everyone! I'm a first-time poster and pretty new to Kubernetes. I'm looking for advice on the best methods for managing and storing secrets in Kubernetes. Currently, I'm using Git for source control and Flux to handle my deployment manifests. My Kubernetes cluster is running on Google Cloud Platform (GCP), and I'm encrypting secrets in Git using SOPS with a GCP KMS key. Right now, my secrets are stored in the same repository as my application, which allows for a refresh of the secret to update the pods that use it. However, I'm seeing potential issues with shared secrets that are used by multiple applications. If a secret is stored in a separate repository, refreshing it won't necessarily trigger all the consuming pods to update since there's no change in the manifest. Does anyone have a clever way of using Flux and GCP services to handle secrets in a GitOps manner that also refreshes the relevant pods? I'm also open to using GCP's Secrets Manager, but I'm unsure if there's a suitable driver to trigger those updates. Thanks in advance for any tips!
6 Answers
I’d recommend checking out Reloader! It helps manage dependencies a lot better and can automatically refresh pods when secrets change. You can find it here: https://github.com/stakater/Reloader.
Would this work with secrets stored externally, like in GCP Secrets Manager? I’m not a fan of base64 encoded secrets.
You might want to explore a secret operator or a CSI secret provider; both are part of the CNCF sandbox. They help fetch secrets from providers like GCP and make them available to your pods. You can trigger refreshes on the Kubernetes side either by altering annotations or through a scheduled job.
Hashicorp Vault is a great option for secret management. You can authenticate and retrieve secrets during your CI/CD pipeline, store them as environment variables temporarily, and then use them in YAML manifests. This way, your secrets don't end up in your repos at all.
You might want to give SealedSecrets a go. It’s quite effective for managing secrets in Kubernetes. Check it out and see if it fits your needs!
How does SealedSecrets compare to SOPS? They seem to follow a similar approach, but I'm really looking for a solution that can signal pod environment changes on secret updates.
If you're sticking with GCP and using GKE, consider the Secrets Manager GKE add-on. It’s good for secret rotation and can automatically push updates to running pods. It should meet your needs pretty well!
That’s what I’m leaning towards! The auto rotation is exactly what I need. Have you used this in your projects? Does it play nice with a GitOps setup?
I personally use External Secrets combined with Reloader. External Secrets pulls secrets from a provider into your cluster, and Reloader makes sure that pods reload when secrets or configmaps change—definitely a solid approach!
This looks promising! Have you scaled it up in a production environment yet? I'm trying to do things the right way from the start.