I'm currently facing issues with FluxCD when trying to sync a large number of Helm releases that are all hosted as artifacts in AWS ECR. It seems that during the sync process, Flux is hitting ECR too aggressively, leading to a 429 Too Many Requests error, which causes issues with reconciliation and results in degraded deployments. I'm wondering if anyone has found a clean solution for this without moving away from ECR, or if it's just the consensus that using Helm in ECR doesn't scale well for FluxCD.
2 Answers
Have you thought about requesting a limit increase for the API quota you're hitting? Typically, many API rate limits can be increased, which might help you avoid those 429 errors.
I’m a Flux maintainer, and I think I understand the issue you're facing. If you're reusing the same HelmRepository across hundreds of HelmReleases, that could create multiple HelmChart objects, which taxes your CPU and Memory significantly. Instead, creating the HelmChart manually and referencing it from multiple HelmReleases might improve efficiency. Also, consider switching to using an OCIRepository, as it’s generally more efficient than the legacy HelmRepository approach, especially with the new enhancements in recent Flux releases.
I missed that detail too. I guess I better catch up on the latest updates, thanks for sharing those links!
So that's why they recommend it! Would be nice if that info was clearer in the docs!