Hey folks, I'm losing my mind over this frustrating GitOps problem I've been dealing with for the past week. My ArgoCD dashboard is showing resources as "Healthy" and "Synced" while Crossplane is actually failing to provision AWS resources. We're getting a bunch of 400 errors from AWS, but ArgoCD just keeps saying everything's fine, which is super misleading!
I've got Lambda functions not updating, RDS instances stuck, and IAM roles not created. It's like the ArgoCD dashboard is mocking us with its green status!
I've searched high and low online and found nothing addressing this issue. It seems like I'm the only one having this problem where ArgoCD's health checks aren't accurately reflecting the state of my cloud resources. The health check logic processes status conditions in a way that if the "Ready: True" condition appears before the "Synced: False" condition, ArgoCD treats it as healthy, completely ignoring failures.
I've managed to fix the issue by changing the order of condition checks so that error conditions are evaluated first, but I can't understand why this hasn't been flagged as a known problem. I feel like I'm either missing something huge or everyone else is just blissfully ignoring their failed deployments.
So let me know, have you all encountered anything like this? Are people not really using health checks with Crossplane? Or am I just really unlucky with my setup?
6 Answers
Glad you found a workaround! But honestly, posting it on Medium behind a paywall is a bit frustrating. You could've shared it more openly.
Yeah, I mean, nobody wants to hit a paywall for useful info. Just share it freely!
This really sounds like a bug. The 'ready' condition should accurately reflect the health status. Instead of a workaround, reporting it might lead to a proper fix.
I've dealt with this problem a while back and fortunately had prior knowledge about Argo health checks. I thought everyone knew they should write custom health checks if the defaults don't work. Especially since this seems common across different setups.
I don’t think this is well-known enough! More people should be aware to save headaches.
You should share your solutions more, maybe start a discussion about it!
Honestly, I think you misunderstood how GitOps and ArgoCD work. ArgoCD is syncing resources based on their desired state; it's not about ensuring everything is perfectly healthy. If Crossplane is having issues later, that’s separate and should be monitored with proper observability tools like Datadog or Grafana.
Right, and if Crossplane has errors, that won't change what ArgoCD reports. It's essential to have a solid alerting system in place.
Exactly! ArgoCD shows when the resources are synced, but for health checks, you’ll need additional monitoring tools.
This should definitely be a GitHub issue rather than just a Medium article. Have you considered opening one? It might help others who face similar issues.
I thought about that, but it seems like they're focusing on other matters right now. Posting it on Medium seemed like the next best option.
True, but still worth it to create a GitHub issue eventually. Other people might find it helpful!
True, Medium's not the best place for this kind of stuff, especially with the member-only model.