I'm really at my wits' end here. I've spent almost a week battling what feels like the worst GitOps nightmare ever. So here's the rundown: ArgoCD shows resources as "Healthy" and "Synced," but Crossplane is failing to provision AWS resources, throwing out 400 errors like it's nothing. We're talking about Lambda functions that won't update and RDS instances stuck indefinitely. Meanwhile, ArgoCD keeps insisting that everything is fine, which feels like a cruel joke.
I've searched everywhere for similar issues – blogs, Stack Overflow, GitHub – but it's like I'm stuck in an alternate universe where no one else seems to be running into this. The issue stems from how the health check Lua logic processes the status conditions; if `Ready: True` comes before `Synced: False`, ArgoCD just ignores the fact that resources are failing.
I managed to fix it by adjusting the order of the condition checks so that errors are prioritized. I'm perplexed that this isn't more widely recognized as a problem. Am I really the only one experiencing this, or are others just ignoring ArgoCD's health checks in favor of direct AWS monitoring? I genuinely need to know if there's a broader issue here or if I'm just particularly unlucky. I documented my solution for anyone else who might face this as I feel like I'm blazing a trail here!
5 Answers
Thanks for sharing this detailed write-up! We're considering moving to a similar setup, and your insights could save us from a lot of trouble. Have you thought about pushing this as a GitHub issue? Might benefit more folks if the maintainers saw it.
It's still worth a shot! Even if it's a niche issue, visibility can help others.
I gotta agree, the status checks should reflect actual readiness. If things are in a bad state, a fix might be more effective than patching up Argo's reports.
It's great that you found a workaround, but I'm curious why you opted to share it on Medium behind a membership wall. It seems like it would benefit more people if it were easily accessible.
I totally get that! Honestly, I avoided Medium at first, but I thought maybe it would reach more folks there.
Yeah, visibility is key! Member-only content doesn't help the community as a whole.
Honestly, I've faced this before. I knew about ArgoCD's health check quirks beforehand, so I set up custom health checks right from the start. I assumed everyone would do the same, but maybe not?
Yeah, those custom health checks are crucial if you want reliable monitoring.
I think you're on to something. This info definitely seems to be less common than it should be!
Listen, I think you're not seeing the whole picture with GitOps and ArgoCD. The resources are actually synced, which is what ArgoCD verifies. If Crossplane fails later on, that’s a different issue. You should have separate monitoring for actual service health outside of your CI/CD tools.
Exactly! ArgoCD does its job well by confirming sync status, but real-time health should come from dedicated monitoring tools.
Right! It’s about managing expectations regarding what ArgoCD does versus what monitoring should be in place.
I did consider that, but after discussions with maintainers, it seemed they think it’s not a pressing issue right now.