Is Anyone Else Experiencing this ArgoCD + Crossplane Silent Failure Issue?

0
4
Asked By TechWiz347 On

Hey everyone, I'm really frustrated and I hope I'm not alone in this! I've been struggling with a major GitOps headache involving ArgoCD and Crossplane for about a week now. ArgoCD is showing that everything is 'Healthy' and 'Synced,' but at the same time, Crossplane is completely failing to provision AWS resources, throwing 400 errors left and right. It's like ArgoCD is telling me 'Everything's fine!' while I'm dealing with Lambda functions not updating and RDS instances stuck.

I've searched high and low for documentation or discussions on this issue but found virtually nothing. It seems like no one else has encountered these silent failures with ArgoCD's health checks not accurately reflecting the state of the resources. The Lua logic ArgoCD uses for health checks seems to evaluate conditions in a way that prioritizes healthy states, completely overlooking actual issues.

I've managed to find a workaround by reordering the condition checks, but I'm baffled that this isn't a commonly recognized problem. Did I just find a flaw no one else is aware of? Is this a common oversight in how people monitor these systems? Please tell me I'm not the only one facing this!

5 Answers

Answered By SystemSage On

Thanks for documenting your experience! I'm considering using the ArgoCD + Crossplane setup as well. Hopefully, your findings keep others from going through the same frustrating issues.

AstroCoder -

Stay tuned to GitHub; often, the community maintains those integrations!

TechWiz347 -

Glad to help! I’d love to prevent someone else from hitting the same wall.

Answered By AstroCoder On

You might want to send a patch to ArgoCD or Crossplane about the health check issue! That could prevent this from happening to others in the future.

TechWiz347 -

That’s a good idea! I’ll look into it.

DataDrivenDev -

Definitely worth pursuing! Patches help the whole community.

Answered By DevOpsGuru On

I've also faced similar issues in the past, and it's true that many folks don’t know about the quirks in Argo's health checks. It's not documented well! Writing custom health checks can definitely save you headaches. Thanks for bringing this to light!

TechWiz347 -

I appreciate it! I figured I couldn't be the only one but it seems like this isn't talked about enough.

CrossplaneFan88 -

No problem! I hope your article helps others recognize the need for custom health checks.

Answered By DataDrivenDev On

It's great that you found a workaround, but have you thought about reaching more people than just Medium? Maybe consider posting a GitHub issue for visibility that could help the community. It's frustrating when issues like this seem to fly under the radar.

TechWiz347 -

Yeah, I didn’t intend for it to be a paywall, just wanted to document it. But I guess I should’ve opened an issue!

CuriousDev32 -

I totally get where you're coming from. Medium articles can be behind a paywall, which doesn't help spread awareness!

Answered By SysAdminJoe On

Honestly, I think there’s a misunderstanding here. ArgoCD is indicating that the resources are synced, which is true. However, if Crossplane fails afterward, that's another layer of failure. GitOps is more about keeping your declared state consistent rather than monitoring health. You might want to set up real monitoring and alerting systems for the resources themselves.

TechWiz347 -

That makes a lot of sense. I guess I've been relying too much on ArgoCD for health statuses!

CloudNinja99 -

Exactly, that’s a critical distinction! ArgoCD will show synced resources even if something’s off. Using tools like Grafana or Prometheus for observability feels essential.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.