I'm bootstrapping clusters with Terraform and utilizing ArgoCD for managing add-ons and deployments. For those of you familiar with Argo, I'm curious about how you manage application secrets. I've noticed there are several SaaS solutions that integrate well with external-secrets, making this process easier, but are there any open-source options that can achieve similar results? In the past, I've set up encrypted config files in a repository combined with Terraform, which worked but had a very clunky user experience. Any advice or suggestions would be appreciated!
10 Answers
Spectro Cloud has been a game-changer for us!
When I interview for cloud roles, I often ask about this topic. There are pros and cons to each method, especially with ArgoCD. I've gone with SOPS combined with KSOPS/Kustomize for version-controlling secrets effectively. I could share some examples if anyone would find that helpful.
I've worked with SOPS too, mostly with Terraform. I'd love to check out those KSOPS and Argo examples if you’re able to share.
I think external secrets paired with one of their many providers is currently the best practice for on-prem or unmanaged clusters. The ArgoCD team has shifted towards using declarative tools like external secrets instead of plugins and SOPS. If you're on a managed cluster like AKS or GKE, using workload identities is super effective, as it lets pods access resources without needing to handle sensitive secrets directly.
I’m trying to sidestep cloud-provider-specific solutions since we need something that works across multiple clouds and on-prem as well. I was hoping for a more user-friendly solution, but it sounds like there’s no one-size-fits-all answer here.
ESO, Vault, and various secrets managers work well, but I wouldn't recommend the standard opaque secrets.
Yeah, ESO with Vault can be quite a lot of overhead, but it might be the best route at this point.
It honestly depends on what you need. If centralized management is crucial, OpenBao is a great Vault alternative. As for in-repo solutions, both Sealed Secrets and SOPS work well. Personally, I prefer using SOPS and giving each user and cluster their own age key, and integrate it with KSOPS for Argo, which handles decryption smoothly.
I've definitely used SOPS before but didn't realize it could work with Argo. That’s super helpful, thanks!
You might want to check out External Secrets Operator. It's definitely open source, so I'm not sure why it was mentioned otherwise. It integrates pretty well with your existing setup.
Yeah, I get that it's open source, but I've mainly used it alongside SaaS for the management part. I'm curious, what do you connect external-secrets to for actually handling the secrets?
Most of the time, I rely on the CSI secrets driver. If not, I use external secrets as a fallback.
I automate secret rotations wherever I can, setting it up to write new credentials as K8s secrets directly. This way, no one needs to hold onto passwords since they're kept solely within the environment. Of course, this method has its limits, particularly if you're not operationally all-in on Kubernetes, but it's efficient in many contexts.
Using the CSI secrets driver is likely how Kubernetes maintainers recommend managing secrets, but support can be hit-or-miss in different helm charts. Ideally, avoiding secrets altogether using something like IRSA is best, but if you must choose, it really depends on your needs between ESO, Sealed Secrets, or Vault. All are valid depending on your scenario.
Trying to avoid Vault because of the footprint, but can use it if needed. I was hoping for something more user-friendly, but it seems Vault is still the default for many.
I’ve been leaning towards OpenBao and ESO for managing secrets effectively.
I’d really appreciate that example, please!