I'm learning Kubernetes while building a three-tier application with a React frontend, Node.js backend, and Amazon RDS database. I understand that ConfigMaps are for non-sensitive settings and Kubernetes Secrets are for values such as database usernames and passwords, but I'm unclear about the usual production setup in AWS and EKS.
I know database credentials should not be committed directly to source control. A common design seems to involve AWS Secrets Manager, EKS workloads, IAM, and possibly tools such as EKS Pod Identity, IRSA, the AWS Secrets and Configuration Provider, or External Secrets Operator.
Could someone walk through a simple real-world example covering where the credential is initially stored, how a backend Pod is authorized to retrieve it, how IAM limits access to only the intended workload, whether a Kubernetes Secret is created or the value is injected at runtime, how rotation works, and what configuration is actually committed to the code repository? I'd also like to understand how secrets are organized when an organization has hundreds or thousands of applications and databases—for example, whether names such as prod/orders-db, prod/payments-db, and prod/users-db are typical.
3 Answers
The access chain is usually: a Kubernetes service account is associated with an AWS IAM role through IRSA or EKS Pod Identity; the backend Pod runs using that service account; and the IAM role is allowed to read only the specific Secrets Manager ARN it needs. For example, the orders backend role might have secretsmanager:GetSecretValue for /prod/orders/api-db and nothing for the payments or users secrets.
With the AWS Secrets and Configuration Provider, the secret can be mounted directly into the Pod at runtime, avoiding a Kubernetes Secret. With External Secrets Operator, the value is normally copied into a Kubernetes Secret first. Kubernetes RBAC should also restrict who can create or modify workloads and service-account associations, since someone able to change a Pod specification could potentially make it use a more privileged identity.
If the application supports it, consider using IAM database authentication for RDS instead of a long-lived password. The Pod assumes its IAM role and obtains a short-lived database authentication token, so there is no static database password to store or rotate. This reduces secret-management overhead, although it can require connection-pool changes and is less portable than ordinary database credentials.
If password authentication is required, keep the password in Secrets Manager, encrypt it with an appropriate KMS key, give each workload a narrowly scoped IAM role, and rotate it there. Rotation may temporarily involve both the old and new credentials, and the application must handle reconnecting or restarting when the Kubernetes-mounted value changes.
A common pattern is to store the credential in AWS Secrets Manager under a structured name such as /prod/orders/api-db. The External Secrets Operator then reads that AWS secret and synchronizes it into a Kubernetes Secret. The application only references the Kubernetes Secret through an environment variable or mounted file, so the application does not need AWS-specific code.
The repository contains the ExternalSecret definition, namespace and workload configuration, and a reference to the AWS secret name—not the actual password. When the value changes in Secrets Manager, the operator periodically updates the Kubernetes Secret. Applications still need to reload the value or restart gracefully, because an already-open database connection may continue using the old credential.
Environment-specific paths and separate IAM permissions help prevent accidental access across environments. For example, a staging role should be able to read only staging secrets, not anything under /prod/.
A practical naming scheme is to include environment, application, and purpose, such as /prod/orders/db or /prod/orders/third-party-api. Larger organizations often separate accounts or KMS keys by environment as an additional boundary, rather than relying on naming conventions alone.