I've been dealing with an incredibly frustrating issue while trying to use ArgoCD with Crossplane, and I can't help but wonder if I'm the only one facing this problem. For the past week, I've been debugging what seems to be a major flaw in how these tools communicate. Despite ArgoCD reporting that resources are "Healthy" and "Synced," I'm getting 400 errors from AWS because Crossplane isn't able to provision resources effectively. This includes problems like Lambda functions not updating and RDS instances stuck in limbo, while ArgoCD maintains its facade of green lights.
I dug into this issue and found that the health check logic might be the crux of the problem. Essentially, if `Ready: True` is listed before `Synced: False` in the conditions array, ArgoCD will consider everything to be fine, ignoring the actual failures happening behind the scenes. This revelation has left me baffled—after all my searching, I can't find anyone else who seems to have encountered this issue.
I took the liberty of fixing the Lua logic by reordering the condition checks to prioritize error conditions first, but I can't shake the feeling that others must be experiencing something similar. It makes me question whether most users are overlooking health checks with Crossplane or if they're just monitoring AWS directly instead of relying on ArgoCD's status.
5 Answers
Thanks for your write-up! I'm considering switching to a stack involving ArgoCD and Crossplane, so your insights will definitely save me time. Have you thought about submitting a formal GitHub issue? This could help a lot of others.
I ran into this problem a while back and expected everyone to know about the health check quirks. It’s odd that this isn't more widespread knowledge.
I get that feeling too! A lot of people seem unaware of these behaviors.
Definitely! I've been exploring similar setups and hope more people share their experiences.
Why put this on Medium instead of filing a GitHub issue? Seems like it could be a big deal for others too!
I thought about it, but Crossplane maintainers said it's community-driven and not a high priority. Just trying to help others find this issue!
Honestly, it sounds like there's a misunderstanding of how GitOps and ArgoCD work. ArgoCD is right in saying the resources are synced. The issues with Crossplane don't affect that verification. You should rely on other monitoring tools to alert you about actual health issues. ArgoCD is for deployment, not health checking!
Right? So many people confuse syncing with health. A solid observability setup is key.
Yup! ArgoCD just reflects the state as specified in the YAML. Supplement it with proper monitoring.
Glad you found a workaround! But seriously, posting it on Medium as a member-only story isn't the best way to share this info. Just make it open!
Exactly! Medium can be pretty exclusive with the member-only stories. It's frustrating.
The member-only stuff really turns me off. Just share the knowledge!
I did think of that, but they seem to prioritize other issues first. Just sharing to lessen the burden for those who might face the same problem.