I messed up while using ArgoCD with a Longhorn app that I deployed via Kustomize. This morning, I inadvertently changed the name of the ArgoCD app from `longhorn` to `longhornnn`, which led to ArgoCD initiating a deletion process for the app that no longer matches the name. Fortunately, the deletion is blocked due to incorrect Longhorn deletion flags that I set. However, now there's a `longhorn-uninstall` job that's failing and keeps retrying indefinitely. I've tried deleting the job, but it just keeps recreating. I attempted to run `helm list -n longhorn-system` to find relevant information or rollback options, but because it's managed via ArgoCD, Helm doesn't recognize anything related. Any suggestions on how to handle this situation?
2 Answers
It sounds like you're in a tricky spot! If auto-sync in ArgoCD is giving you grief, one immediate step could be to disable it temporarily. That might prevent further issues as you work on resolving this. But I get that disabling it doesn't really help in this particular mess you've found yourself in.
Before doing anything drastic, check if your Longhorn storage class is set to retain persistent volumes (PVs). This way, even if ArgoCD is trying to uninstall Longhorn, your data should stay safe. You could document your PVC manifests and then recreate Longhorn using the same Helm values. Just remember to patch your existing PVs to the new PVCs you've created afterward. Oh, and as a tip, be cautious about using GitOps tools like ArgoCD for platform-level components. It's safer to manage them directly with Helm instead!
I think advising against using tools like ArgoCD entirely is too extreme. They actually have features to prevent such mishaps. You just need to get used to the tools and their configurations before deciding to scrap them.
I see your point, but managing infra components through GitOps is doable if you do it right. Helm directly isn’t always the way; it can get messy without process in place. I’m in a similar boat with my home cluster and need to keep things tighter!
Disabling auto-sync might be good for the future, but it won't sort out your current situation. You'll need a different fix for that!