What’s the best way to manage database credentials and grants for Kubernetes services at scale?

0
2
Asked By MellowCedar42 On

We run services on AWS using RDS, EKS, Redshift, and DocumentDB. Today, Liquibase applies schema changes with shared credentials from Parameter Store, while a Jenkins library creates per-application database users and stores their credentials in Secrets Manager. This works, but it is difficult to see which services use which accounts, and decommissioned applications leave behind database users and secrets.

I'm considering using Vault or OpenBao for credential lifecycle management, with a separate Git repository owning durable database roles, grants, and Vault configuration. Application schemas would remain in each service repository and continue to be managed with Liquibase. For PostgreSQL and MySQL, I'm looking at Terraform providers to manage users and grants, but I'm unsure how to handle Redshift and DocumentDB declaratively.

Has anyone operated Vault or OpenBao successfully for this use case, and how difficult was the initial setup? Should we revisit AWS IAM database authentication instead, even though it would require application changes? I'd also appreciate practical approaches for managing Redshift and Mongo-compatible database permissions, as well as lessons from production deployments.

4 Answers

Answered By PracticalBadger31 On

A common compromise is to use Terraform, or a dedicated reviewed provisioning job, for the initial database roles and grants, while Liquibase remains responsible for application DDL. Keep the bootstrap identities separate: one tightly controlled role for migrations and distinct runtime identities for applications. Store the grant definitions in one repository so changes go through review and decommissioning can explicitly revoke the corresponding role.

For PostgreSQL and MySQL, Terraform providers can work for this. For Redshift and DocumentDB, an idempotent SQL or API-based job is usually more realistic than forcing everything through Terraform. The important part is that the job has a source of truth, produces logs, and supports deletion as well as creation.

Answered By SecretMaple56 On

If Vault is selected, the Vault Secrets Operator can synchronize Vault-managed values into Kubernetes secrets, although that still leaves a secret inside the cluster and does not solve database-user cleanup by itself. Other secret stores can provide a similar injection pattern. The key questions are whether you need dynamic database credentials, whether developers can support IAM-auth changes, and who owns revoking roles when a service disappears.

For an AWS-only estate, native identity-based authentication is usually simpler to operate. Vault or OpenBao becomes worthwhile when you need a consistent credential lifecycle across multiple providers and database types.

Answered By OrbitingLynx7 On

Before choosing a secrets platform, separate two concerns: credential rotation and lifecycle/auditability. Vault can issue and rotate credentials, but it will not automatically know whether a decommissioned service is still using an account. Tie provisioning and revocation to the service lifecycle, and log every credential request with the workload or pod identity. A periodic reconciliation against currently deployed services can then identify unused users and secrets.

For engines without a good Terraform provider, a scheduled Lambda or Kubernetes job can read an approved grants repository and apply the SQL directly. It is less elegant than a provider, but still repeatable, reviewable, and auditable.

MellowCedar42 -

That makes sense. A Lambda-driven approach may be easier than maintaining a custom Redshift or Mongo-compatible database provider, although I may still investigate writing one later.

Answered By CloudHarbor88 On

I’d revisit IAM authentication before adding Vault, especially since the environment is already AWS-based. With EKS pod identity or IRSA, a workload can receive an AWS role, and RDS IAM authentication can avoid long-lived application passwords altogether. Removing the password also removes a large part of the rotation and orphaned-secret problem; revoking the role or policy cuts off access.

There is some application work involved, but a small code change may be less expensive than operating another credential platform indefinitely. Vault is more compelling when you need dynamic credentials across many clouds, database engines, or non-AWS systems.

MellowCedar42 -

The application changes are the main concern, but I agree it may be worth paying that cost once rather than maintaining credentials forever. I’m also looking into how to bootstrap the SQL users and grants separately from the application schemas.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.