I'm having a really frustrating experience with ArgoCD and Crossplane, and I'm starting to feel like I'm the only one. ArgoCD shows my resources as "Healthy" and "Synced," but in reality, Crossplane is failing to provision AWS resources and throwing 400 errors. This includes issues with Lambda functions not updating, RDS instances stuck, and IAM roles not being created—while ArgoCD's dashboard just insists everything is fine! I've Googled this for days, but I can't find anyone else talking about it. It seems like the health checks in the Lua logic are flawed because they can report something as healthy even when there are errors present. I managed to find a workaround by reordering condition checks, but I'm baffled that this isn't widely recognized as an issue. Am I just really unlucky, or is this a common problem that everyone else is ignoring?
4 Answers
Honestly, your understanding of GitOps and ArgoCD might have a few gaps. The resources are synced as per Argo's perspective. Crossplane failing afterwards is its issue, not Argo’s. GitOps ensures your cluster's objects match your expectations, not that all resources are healthy. Setting up proper monitoring is crucial; rely on tools like Datadog or Grafana to understand your system's health.
You hit the nail on the head! ArgoCD tracks sync status, but real health checks need monitoring tools.
Thanks for your detailed write-up! We're considering this tech stack for our project, and you've potentially saved me time and effort. Have you thought about submitting a GitHub issue for ArgoCD or Crossplane? This could be a problem for a lot of people.
I thought about it, but after discussing with maintainers, they don't see it as a major issue right now. That's why I wrote the article to help others.
Great to see you found a workaround, but I would suggest not publishing it on Medium as a "Member-only story." It feels a bit exclusive when it could help a lot of people.
100% agree, it should be accessible to everyone.
I avoid Medium too. If it's member-only, forget it!
This sounds similar to a problem I encountered years ago. I knew about ArgoCD's health check behavior before using Crossplane, so I tested and customized my health checks. I assumed everyone else was aware of the need for custom checks if the defaults don’t work properly.
I still think this isn't common knowledge enough for new users.
Totally! Using a proper observability platform will save you time and headaches.