I'm building a personal SRE project with guidance from a mentor. The EKS cluster was created with Terraform, Flux was bootstrapped through Terraform, and the setup uses reusable modules and a remote backend. I now want to deploy Prometheus, Grafana, and Alertmanager through Flux.
My mentor suggested looking at kube-prometheus-stack and the Prometheus Operator/kube-prometheus projects. The recommendation was to manage the applications with Flux HelmRelease resources while keeping the monitoring-related CRDs separate instead of allowing the Helm release to install and manage them. I'm trying to understand the correct repository structure and deployment order, and whether the CRD manifests should be sourced from the Prometheus Operator project or copied from another location.
1 Answer
Separating the CRDs from the HelmRelease is a sensible approach. Keep the CRD manifests in a dedicated Flux-managed directory or Kustomization, apply them before the HelmRelease, and let the HelmRelease manage Prometheus, Grafana, Alertmanager, and their configuration. This avoids Helm upgrades or removals unexpectedly modifying or deleting shared CRDs and the custom resources that depend on them.
That makes sense. For the CRD manifests, should I take them from the Prometheus Operator/kube-prometheus repository, or is there a better source? I found manifests there and wondered whether I should copy them into my Flux repository.