I've been wrestling with a frustrating issue involving ArgoCD and Crossplane for over a week now. While ArgoCD indicates that resources are 'Healthy' and 'Synced', Crossplane is failing to provision AWS resources and throwing a ton of 400 errors. We're talking about Lambda functions that won't update, RDS instances that are stuck, and IAM roles that are never created, all while ArgoCD displays a deceptive green dashboard that makes it look like everything's fine.
I've spent days searching online and haven't found any relevant discussions or solutions, which makes me feel like I'm navigating this problem in some sort of alternate reality where I'm the only one dealing with this. The core of the issue seems to lie in how ArgoCD's health checks are set up. They process status conditions in array order; if 'Ready: True' appears before 'Synced: False', ArgoCD assumes everything's good. I was able to fix the problem by changing the order of condition checks, but I'm baffled that this isn't a more widely acknowledged issue. Am I missing something obvious, or is this a problem many others are blissfully unaware of? Is anyone else even using health checks with Crossplane?
5 Answers
Glad you found a workaround! But seriously, posting on Medium as a 'Member-only story'? That's a bit annoying. Just share it openly so everyone can benefit.
I avoid Medium in general, but a member-only story? Forget it.
I dealt with this a while back. I assumed anyone using Argo would know to write custom health checks. It’s surprising how little this seems to be discussed.
Yeah, I feel like more people should be aware given how many folks are using these tools.
Exactly! It shouldn’t be an edge case, but here we are.
This issue highlights bad status reporting. Rather than trying to patch it, a bug report would be a more productive way to handle it.
Why post this as a Medium article? It’d be better addressed directly with a GitHub issue, so people can collaborate on fixing it.
The maintainers have said it’s not urgent, which is why I shared it there instead. Hoping this helps anyone who might run into the same issue!
You should definitely share it where the community can engage, like GitHub. It seems like a significant problem!
I think there’s a misunderstanding about GitOps and ArgoCD. ArgoCD is reporting that those resources are synced correctly, which they are. The failures in Crossplane happen later and are unrelated to Argo's checks. Instead of relying solely on ArgoCD for health checks, you should have proper monitoring set up for your resources.
Totally agree. If Crossplane is generating errors, that should trigger logs or alerts elsewhere. ArgoCD's job is to ensure your config in Git matches what's running in your cluster, not to monitor resource health.
Right! ArgoCD is not a health monitoring tool; investing in something like Grafana or Datadog for that makes way more sense.
Absolutely! It's frustrating to have valuable info behind a paywall.