About 7 months ago, I set up our production infrastructure on Azure using Terraform and a Service Principal that I created through the Azure CLI. The Service Principal has Contributor rights at the subscription level and a client secret with a one-year expiry period. The infrastructure includes various resources like Resource Groups, VNets, Subnets, VMs, NAT Gateway, AKS, Azure MySQL Flexible Server, and more. I've made some manual changes since then, and the environment has been active for around 6 months.
Recently, I discovered that integrating Azure Key Vault with AKS is much easier with Managed Identities. However, my AKS cluster is still using a Service Principal. I read that Managed Identities are recommended over Service Principals for Azure resources.
Now, I'm worried because the Service Principal's client secret will expire in about 5 months. I have a few questions:
- What will happen when the SP secret expires?
- How can I safely rotate or update the secret across all provisioned infrastructure, particularly for AKS, without causing downtime?
- For those who use Terraform with a Service Principal, how do you manage secret rotation and expiry in production?
- Is switching to Managed Identities the only long-term solution, or do many just opt for longer SP expiry periods and handle rotation manually?
Any insights from those experienced in production would be greatly appreciated!
2 Answers
We opted for a self-signed wildcard PKI certificate stored in a key vault for our AKS clusters and pods. The certificate lasts a really long time, which allows us to control internal connections without breaking things in our pods or clusters. When it’s time to renew, we just update it without having to change the key.
When the Service Principal's secret expires, you'll get 401 errors if you try to use it. It's best to add a new client secret before it expires and update any configurations that use it. Remember, the SP is generally only for deploying changes, not directly used by AKS itself. Moving towards Managed Identities is a good strategy too, as it simplifies things. We aim to rotate our client secrets annually and try to remember to do it ahead of the expiry date.
Since we're managing our infrastructure via Terraform, can we automate the secret updates through Terraform rather than manually changing it everywhere?
Using user-assigned managed identities with federated credentials is a great strategy. It minimizes the risk of credential theft since they aren't stored in plaintext anywhere. Plus, you won’t have to worry about renewing those credentials regularly across multiple resources.