What’s the best way to manage and rotate secrets in production Kubernetes?
Asked by
MellowPine47
For a production Kubernetes environment, what secret-management pattern are you using? The main options I'm considering are native Kubernetes Secrets, a Secrets Store CSI Driver, or having applications authenticate with workload identity such as IRSA or pod identity and retrieve secrets directly from a cloud or external secrets manager. Native and CSI-mounted secrets may be exposed if a workload is compromised, while direct API access avoids Kubernetes-stored or mounted copies but requires SDK integration and careful handling of refresh and rotation. How do you balance security, operational simplicity, permissions, auditing, and short-lived credentials?
Answers
For most teams, an external secrets manager combined with an operator is a practical default. The operator can sync selected values into Kubernetes Secrets, while access is controlled through service-account or workload identity permissions. Keep those permissions narrowly scoped—avoid a broad cluster-wide store that lets anyone creating an ExternalSecret read everything in the backend. Rotation is also much easier when the external system supports short-lived or dynamic credentials.
Reply to this answer
Use workload identity and eliminate secrets where you can. Cloud identity systems can replace static cloud credentials, and mTLS can replace some application API keys. For the remaining values, a system such as Vault, OpenBao, or a cloud secrets manager works well with an operator, agent, or CSI integration. The best delivery method depends on the application—some can read files and reload them, while others only read environment variables at startup.
Reply to this answer
Native Kubernetes Secrets can be perfectly reasonable when the cluster is operated properly. Encrypt etcd at rest, prevent external access to the control plane, enable tamper-resistant audit logging, and make sure service accounts and users cannot read more Secrets than necessary. Be especially careful with LIST permissions: listing Secrets can expose the contents of every Secret in a namespace, not just the names. For higher isolation requirements, a provider that delivers only the specific secret to a workload may be preferable.
Reply to this answer
The important point is that once a workload can use a secret, a compromise of that workload may expose the secret regardless of whether it came from etcd, a CSI mount, an environment variable, or an API call. The real improvements come from reducing blast radius: use workload identity instead of long-lived cloud keys, prefer short-lived credentials, rotate frequently, restrict RBAC, and isolate workloads. Don’t assume that moving the value out of Kubernetes makes a compromised application safe.
Reply to this answer
There isn’t one universally correct solution. Static, single-namespace values with modest requirements may be fine as native or sealed Kubernetes Secrets. Shared, frequently rotated, or dynamic credentials usually justify an external manager. Direct application integration offers more control but increases development and dependency costs, so it’s worth reserving for cases where the stronger isolation or refresh behavior is actually needed.
Reply to this answer
Add your answer
Answers are reviewed before publication.
SilverKite6: That’s why I prefer dynamic credentials where possible. A compromised pod may still obtain the current value, but its useful lifetime can be limited and revocation is more manageable.