I'm learning Kubernetes while building a three-tier application with a React frontend, Node.js backend, and Amazon RDS. I understand that ConfigMaps are for non-sensitive configuration and Kubernetes Secrets are for sensitive values such as database usernames and passwords, but I'm unclear about the usual production setup on AWS EKS.
I know credentials should not be committed directly to source control, for example:
stringData:
DB_USERNAME: admin
DB_PASSWORD: mypassword
I keep hearing that production systems use AWS Secrets Manager, but I'm not sure how the complete flow works. Does the backend Pod retrieve a credential from Secrets Manager and then connect to RDS? How is this organized when an organization has hundreds or thousands of applications and databases? Would secrets typically use names such as `prod/orders-db`, `prod/payments-db`, and `prod/users-db`?
I'm also trying to understand the differences between EKS Pod Identity, IRSA, the AWS Secrets and Configuration Provider, External Secrets Operator, and Kubernetes Secrets.
Could someone give a simple production-style example covering these points?
1. Where is the credential initially stored?
2. How does the backend Pod obtain it?
3. How does IAM restrict a workload to only the secrets it needs?
4. Is a Kubernetes Secret created, or can the value be injected directly at runtime?
5. How does rotation work?
6. What configuration is safe to commit to source control?
4 Answers
A typical flow looks like this: an administrator or deployment process creates the credential in Secrets Manager; an ExternalSecret resource references that secret; the operator retrieves it using its IAM role and creates or updates a Kubernetes Secret; and the backend Pod reads the Kubernetes Secret through an environment variable or volume mount. When the value changes in Secrets Manager, the operator eventually updates the Kubernetes Secret. Applications need to reload the value or restart the Pod, depending on how the credential is consumed.
Another option is the AWS Secrets and Configuration Provider, which mounts the value directly into the Pod through a CSI volume, avoiding a Kubernetes Secret object. That can reduce persistence inside the cluster, although the application still needs to read the mounted file. In either case, only the reference and access configuration belong in source control.
The access chain is usually: a service account is associated with an IAM role using IRSA or EKS Pod Identity; that role is allowed to read only a specific secret or small set of secrets; the workload uses an AWS integration such as External Secrets Operator or the AWS Secrets and Configuration Provider; and the application receives the resulting value.
For example, an orders API service account could be associated with a role that has `secretsmanager:GetSecretValue` only for the production orders secret and permission to use the relevant KMS key. It should not have permission to read payment or user database credentials. Kubernetes RBAC separately controls who can create or modify the manifests and Pods that request those secrets. IAM does not replace Kubernetes RBAC.
Before building a password workflow, check whether RDS IAM database authentication is supported by the database engine and application. With IAM authentication, the workload obtains a short-lived authentication token using its IAM role instead of storing a long-lived database password. That can eliminate password rotation and reduce secret-management overhead, though it may require connection-pool configuration and is less portable than ordinary database credentials.
If IAM authentication is not suitable, use Secrets Manager with tightly scoped workload identity, environment-specific secret paths, encryption with KMS, and an operator or CSI provider to deliver the credential.
A common setup is to store the real value in AWS Secrets Manager and commit only a reference to it. For example, a secret might be named something like `/prod/orders/ database` or `/prod/orders/db-credentials`, with separate naming and KMS permissions for each environment.
External Secrets Operator can read that AWS secret and synchronize it into a Kubernetes Secret. The application then consumes that Kubernetes Secret as environment variables or a mounted volume, just as it would any other Kubernetes configuration. The repository contains the ExternalSecret manifest, namespace, secret name, and references—not the actual password.
This also makes environment separation easier. Production and test can use different secret paths and encryption keys, while the workload identity in each cluster is limited to its own environment.
If credentials are rotated, plan for the transition carefully. Database clients may keep existing connections open, so the application should reconnect or reload credentials gracefully. Rotation systems often allow a short overlap between the old and new credentials.