Hey everyone, I'm at my wit's end here. I've been tackling this super frustrating issue with ArgoCD and Crossplane and it seems like I'm the only one dealing with it. Here's the deal: ArgoCD shows all my resources as "Healthy" and "Synced," but Crossplane is failing miserably to provision AWS resources, throwing 400 errors left and right. Meanwhile, ArgoCD's dashboard is all green, acting like everything is peachy keen.
I've got Lambda functions that aren't updating, RDS instances stuck, and IAM roles not being created – all while ArgoCD gives me the false sense of security. I can't find anything online about this issue, which is bizarre.
The root of the problem seems to be in the health check Lua logic, where status conditions are processed in array order. So if `Ready: True` is listed before `Synced: False`, ArgoCD shruggingly determines everything's fine. I managed to fix it by reordering the conditions, prioritizing error checks first. Still, I can't shake the feeling that this can't just be my issue.
Is nobody else using health checks with Crossplane? Are you all just monitoring directly on AWS? Am I really the odd one out?
5 Answers
Nice job figuring out a workaround! But just a heads-up, posting your fix as a "Member-only story" on Medium isn’t the best way to share it – that limits who can see it.
Yeah, definitely feels off. Medium can get pretty restrictive.
Why is this not a GitHub issue instead of a Medium article? Seems like a substantial problem that could use some traction.
I asked the same thing! From what I gather, the maintainers are focused on other issues at the moment.
Seems like a miss – better to raise the alarm on GitHub where more people can see it!
Thanks for detailing your issue! We're considering a similar stack, and your insights could save us a ton of time. Have you thought about escalating this with a GitHub issue? It could impact a lot of users in the long run.
It’s definitely worth mentioning it, could shine a light on an unnoticed problem.
I did think about it, but after chatting with maintainers, it sounded like a niche issue they are juggling alongside other priorities.
Honestly, I think you might be looking at this all wrong. ArgoCD is showing that the resources are synced according to the state in your cluster. The failures in Crossplane are an issue, but Argo isn't meant to be your health monitor. It sounds to me like you need proper monitoring tools alongside GitOps to catch those errors earlier.
Right? ArgoCD does its job by ensuring the state matches what you declared, but if Crossplane is failing afterwards, that's on monitoring.
Exactly! Using tools like Datadog or Grafana can help you keep tabs on those AWS errors.
I had a similar issue a while back. I figured out how Argo's health checks work early on, so I made custom checks for my resources. Seems like a lot of folks might overlook this detail when starting out.
I really believe people need to be more aware about this. It's a crucial part of using Argo.
Agreed, it's pretty fundamental to understand how health checks operate!
Totally agree! Open access is key for collaboration.