In a production Kubernetes environment, which approach do you generally recommend for storing, accessing, and rotating secrets? The main options I'm considering are native Kubernetes Secrets, the Secrets Store CSI Driver, or having applications use workload identity such as IRSA or pod identity to call a cloud secrets manager directly. Native and mounted secrets can still be exposed if a workload is compromised, while direct API access requires SDK integration and careful handling of refresh and rotation. What setup has worked well for your organization, and how do you handle permissions, auditing, and short-lived credentials?
5 Answers
There isn’t a magic storage mechanism that protects a secret from a compromised workload that is authorized to use it. Whether the value comes from an environment variable, a mounted file, or a secrets API, the application eventually has access to it. The important controls are workload isolation, least-privilege identity and RBAC, encrypted storage, restricted control-plane access, audit logs, and short-lived credentials with frequent rotation. For many teams, native Secrets are perfectly reasonable when etcd is encrypted, access is tightly controlled, and the cluster is well managed.
Try to eliminate long-lived secrets where possible. Workload identity can replace many cloud access keys, and mTLS or short-lived certificates can replace some application API keys. When static credentials are unavoidable, use a managed or centralized secrets service, restrict access based on the service account, rotate frequently, and make sure the application can reload or restart safely. Sealed configuration files are useful for some static, single-consumer cases, but they don’t remove the need to protect the decryption keys and deployment pipeline.
A common practical choice is an external secrets manager plus an operator that syncs only the required values into Kubernetes. It works with cloud secret stores or Vault-like systems and avoids building provider-specific SDK logic into every application. Keep the stores and service accounts narrowly scoped—especially avoid giving every workload permission to read an entire shared secret store—and make sure rotation behavior is tested. The tradeoff is that synchronized values exist in both systems, so both locations need to be secured.
For higher-security environments, applications can use workload identity to call Vault or a cloud secrets service directly. This avoids persisting the value in Kubernetes and gives the provider a clear audit trail tied to the workload identity. It also adds operational complexity: applications need provider integration, retry and refresh logic, and a plan for what happens when credentials rotate. An injection agent or init-container pattern can be a reasonable middle ground when changing every application to use an SDK is impractical.
The right answer depends on the environment and the way the secret is consumed. A small deployment with a few static values may be best served by native Kubernetes Secrets with encrypted etcd and strong RBAC. Larger or multi-cluster setups often benefit from an external manager and automated synchronization, while especially sensitive systems may prefer direct retrieval with workload identity. Don’t add Vault, CSI, or custom SDK integration purely for appearances—start with the simplest design that meets the threat model, then invest in rotation, observability, and access controls.
That permission model matters a lot. A broadly scoped shared store can be more dangerous than a simple native Secret if anyone who can create an ExternalSecret can request arbitrary data. Use namespace- or workload-specific stores and identities, and audit every read.