I've been struggling with a frustrating issue for over a week now while using ArgoCD alongside Crossplane. The problem is that ArgoCD shows resources as "Healthy" and "Synced," even when Crossplane is failing to provision AWS resources. I'm getting 400 errors from AWS, but ArgoCD acts like everything is just fine, which is incredibly misleading! I'm seeing problems like Lambda functions not updating and RDS instances stuck without any updates, all while ArgoCD's dashboard stays green.
I've searched high and low for information on this issue but found nothing—no blog posts, no questions on Stack Overflow, and no issues on GitHub that address it. It feels like I'm in some alternate universe where nobody else is having this issue! I pinpointed that the problem lies within the health check Lua logic, which processes the status conditions in a specific order that allows for misleading health statuses.
I managed to fix the problem by reordering those checks to prioritize error conditions, but I'm baffled that no one else seems to have run into this. I want to know: are others using health checks with Crossplane? Are they just monitoring AWS directly? Or am I incredibly unlucky with my setup?
4 Answers
Glad you found a workaround! However, maybe think about putting your article somewhere less paywalled, like GitHub? Medium can be a bit limiting with its member-only content.
I agree, Medium can be tricky—sometimes best to just stick to open forums!
Sounds like you've encountered a common hiccup. Many users don’t realize that just because ArgoCD shows a resource as green, it doesn’t necessarily mean everything is perfect. Implementing custom health checks tailored to your resources might save you from future headaches. Plus, using proper logging and monitoring can provide insights into these issues.
Absolutely! Getting ahead of these issues with custom checks is crucial for a smooth workflow.
Exactly! Just because Argo says 'good' doesn’t mean it is. Custom health checks are essential for keeping an eye on Crossplane behavior.
This is some solid info! I'm considering migrating to a similar setup, so your insights are super valuable. Have you thought about reaching out to GitHub for an issue there? It sounds like there could be a lot of others affected without realizing it.
That makes sense. Either way, sounds like sharing this knowledge is a great way to help others!
I did consider it, but many of the maintainers felt it was more of a community issue. It’s frustrating, but I hope this can spark some more awareness!
Honestly, I think there’s a key misunderstanding here. ArgoCD is doing what it’s supposed to—it’s indicating that the resources are synced with what's declared in the cluster. The failures you're seeing are likely due to Crossplane not handling the errors properly. Have you looked into better observability or monitoring for your AWS resources? A proper monitoring tool could help track these issues more effectively.
Yeah, you make a good point. While ArgoCD will show synced states, it's not meant to capture every failure in the provisioning process. Definitely need monitoring systems like Grafana or Prometheus to dive deeper.
Exactly! If Crossplane is failing, you’d want alerts via proper monitoring rather than relying solely on Argo.
100% agreed, it's a hassle trying to access that stuff.