I inherited a couple of in-cluster agent workloads that authenticate to internal APIs and a database using long-lived credentials stored in Kubernetes Secrets. There is also a static credential that many people on the platform team can read, and it rarely gets rotated because coordinating the change is difficult. Once these workloads reach the surrounding network, they can access much more than they actually need. I understand the case for short-lived, per-workload identities inside the cluster, but I'm looking for practical guidance on extending that model beyond the cluster. How are you limiting each workload to only the external services and operations it requires?
2 Answers
Use federated workload identity rather than distributing credentials. Bind an identity to the deployment or service account, then let the workload obtain short-lived tokens from your identity provider and exchange them for access to the API, database, or cloud service. Assign permissions to that identity with least privilege, including resource and operation-level restrictions where possible. The same pattern works for CI workloads as well as services running in the cluster.
A secrets manager such as Vault can issue dynamic, short-lived database credentials or generate secrets through an external-secrets integration. That is useful for systems that cannot consume federated identity directly, but it adds operational complexity. Prefer native IAM or workload identity whenever the target service supports it, and use a secrets manager as a compatibility layer rather than the default for every credential.
Generated secrets are still handy for legacy services or one-off keys, but make sure their lifetime, audience, and permissions are limited. They should not become another static credential copied into a cluster-wide Secret.
Most major cloud platforms provide an equivalent. For example, AWS has IAM Roles for Service Accounts, while other platforms connect Kubernetes service accounts to managed identities. The exact setup depends on the provider, but the important parts are short-lived tokens, an explicit workload binding, and narrowly scoped permissions.