I'm working on deploying my app using Helm on GKE and want to integrate external-secrets to manage my secrets from GCP Secret Manager. After installing and applying the charts for the external-secrets, everything works fine initially and the Kubernetes secret is created without issues. However, when I try to modify the ExternalSecret to reference an additional GCP secret and run a helm upgrade, both the SecretStore and ExternalSecret resources are wiped out along with the Kubernetes secret. I've found that the only way to get it back is to recreate the external-secrets pod in its namespace and then run the helm upgrade again. I'm new to Helm and Kubernetes, so is this expected behavior? Should I avoid modifying the ExternalSecret after the first upgrade? Any advice on managing this would be greatly appreciated!
4 Answers
You might want to update your CRDs separately from your Helm chart. Doing this can prevent those resources from being destroyed during an upgrade. Creating an isolated Helm chart for your external-secret resources might help keep things stable, especially since they have dependencies on the operator.
You mentioned that both resources have the same name—be careful with that! If they do, the later deployment will overwrite the former. Beyond that, there could be a patch issue where changes force k8s to recreate objects rather than update them. Ensure your .Values are set correctly and look into changing the spec.creationPolicy to 'Orphan'. That way, the Kubernetes secret won't be deleted when the ExternalSecret is changed, but you'll need to manage the cleanup of the old secrets yourself.
It sounds like your configurations are causing one or both of the resources to be inadvertently recreated, potentially leading to the deletion of secrets. If you're using 'helm upgrade --force', that can definitely mess things up too. Before upgrading, check if you're making any changes that aren’t easily patched. Sometimes Helm has trouble with certain updates, which can result in loss of resources.
I'm a bit confused about how you're structuring your Helm charts. Are you trying to install the External Secrets Operator within your own application chart? That can lead to some issues. Ideally, your ExternalSecret resources should be defined in the app's chart, and the operator should be in its own chart. If you're also changing values like serviceName during upgrades, that could lead to resource recreation—Double-check your .Values settings to see if changed values are perhaps causing this!
I'm indeed placing the ExternalSecret and SecretStore in my application's chart, while the operator is in another namespace. The serviceName remains consistent during upgrades, so it’s puzzling why this is happening.
The templates I shared are part of a separate Helm chart that includes deployments and services. Should I consider splitting the external-secret resources into their own isolated chart instead?