I've been grappling with a particularly frustrating problem involving ArgoCD and Crossplane for several days now, and it's driving me up the wall. Here's the deal: ArgoCD shows my resources as 'Healthy' and 'Synced,' but Crossplane is actively failing to provision AWS resources. I'm getting a ton of 400 errors from AWS, yet ArgoCD is acting like everything is peachy. We're seeing issues like Lambda functions not updating and RDS instances stuck indefinitely, while the green glow of ArgoCD's dashboard feels like a cruel joke.
I've scoured the internet for days, looking for insights into this issue, but it's like I'm the only one experiencing it. The health checks in Crossplane are not functioning as they should because ArgoCD's Lua logic is processing status conditions in an order that can misrepresent actual health status. If an object is 'Ready' but also 'Not Synced,' ArgoCD just goes ahead and assumes everything is okay.
I've managed to fix the issue by reordering the condition checks to prioritize error conditions over healthy ones, but why hasn't anyone else encountered or documented this? Are folks simply ignoring health checks or monitoring AWS directly? Am I just particularly unlucky, or is there a deeper issue at play that no one is addressing?
Also, I wrote a more detailed article on the topic [here](https://medium.com/@heinancabouly/the-silent-failure-why-your-argocd-crossplane-resources-show-healthy-when-theyre-not-93185b648977?source=friends_link&sk=a21682df5e53d58f8e58e0609a255961). I've even reported it on GitHub in hopes that this problem gets recognized and fixed.
5 Answers
Thanks for the insights! We're considering a similar setup, and your article is actually really helpful. Have you thought about officially reporting this on GitHub? It looks like it might affect others down the line.
Why is this an article on Medium instead of a proper GitHub issue? It seems more valuable to propose a fix than to just share your frustrations.
I hear you! It would have been better to tackle it head-on on GitHub.
I think there's a bit of misunderstanding here about how GitOps works, especially with ArgoCD. The resources are technically synced, and ArgoCD verifies that. What you really need is a solid monitoring setup to catch the issues Crossplane experiences after the fact. GitOps doesn’t guarantee that everything’s healthy, it just ensures that your declared state in the cluster is what you expect it to be.
I get that, but can we at least agree that ArgoCD needs better health checks for these cases?
Exactly. Just because ArgoCD says it's synced doesn't mean Crossplane isn't throwing errors.
I've had some similar issues in the past, though I knew about Argo's health check behavior beforehand. It’s crucial to write custom health checks for your resources if the defaults don’t cut it.
Right? It seems like something more people should be aware of.
It's great to hear you found a workaround, but Medium as a platform? Honestly, it feels a bit pretentious, especially with member-only content.
Totally agree! Medium can be hit or miss for sure.
Yeah, member-only anything just pushes people away.
I did consider that, but the maintainers don't see it as urgent, unfortunately. Still, I’m hoping the article helps others.