I've been grinding through a really frustrating issue with ArgoCD and Crossplane where everything looks perfectly healthy on ArgoCD's end, but in reality, Crossplane is failing to provision AWS resources, generating error messages. ArgoCD shows resources as 'Healthy' and 'Synced', while I'm being bombarded with 400 errors from AWS. It's incredibly baffling! Things like Lambda functions won't update, RDS instances are stuck, and IAM roles aren't being created. I feel like I'm the only one facing this issue since my Google searches are turning up empty. It seems like the health check logic in Crossplane is flawed—if the conditions are ordered incorrectly, ArgoCD thinks everything is fine when it's not. Has no one else noticed this? *Are people ignoring health checks with Crossplane?* I managed to fix the ordering of condition checks, but why isn't this a known issue? Am I really alone or just unlucky?
6 Answers
This definitely sounds like a bug, but sending a fix directly could be the way to go. It’s always better to contribute rather than just discuss issues.
Thanks for breaking this down! My team is considering switching to this stack, and your insights might save a lot of headaches! Have you thought about opening a GitHub issue for a more permanent fix?
I think you're misunderstanding how GitOps and ArgoCD work. ArgoCD is saying everything is synced when the resources match the desired state in Git. Problems with Crossplane occurring after the fact don’t reflect on Argo's performance. For effective monitoring, you should have a proper observability setup like Grafana or Datadog.
Exactly. Argo isn't a health monitoring tool – it focuses on synchronization states. Proper logging from your AWS resources should spot problems.
I see what you mean, but it does seem odd that Argo isn't giving a clear indication of resource health.
I've experienced similar issues a while ago. I had to create custom health checks because the default ones don’t work as expected with Crossplane. It's definitely not common knowledge for everyone using ArgoCD.
Thanks for sharing! It seems like we all need to be more proactive about these custom health checks.
It's awesome you figured out a workaround! But seriously, it would be better to share this in a more accessible place than Medium, especially since it’s locked behind a membership paywall.
Yeah, I totally agree! Embracing open-source contributions on platforms like GitHub is better for everyone.
I get that, but I was just trying to help anyone who might hit the same wall. I'll think about that next time!
Why isn’t this a GitHub issue instead of a Medium article? Seems like it'd get more traction there.
Yeah, it would! I think OP is worried that the maintainers see it as a low priority.
Yes, I thought about that. After discussions with the maintainers, it seems they’re prioritizing other things for now, though.