I'm a full-time software engineer and run a small business on the side. While I'm not primarily focused on DevOps, I have some experience with technologies like Kubernetes, Rancher, RKE, ArgoCD, and others from my past jobs.
Currently, my business runs on a Rancher-provisioned RKE cluster, utilizing a mix of ArgoCD and Rancher apps (primarily installed via Helm) for deployments. We back up our data with Velero and store it in S3 nightly. Recently, we faced a significant issue when the cluster got corrupted, leading to a complicated manual restoration process using Velero. This unfortunate experience, along with the difficulties in upgrading to RKE2, has prompted me to look into Infrastructure as Code (IaC) solutions.
I'm experimenting with Pulumi combined with cloud-init for setting up core infrastructure and transitioning all Rancher apps to ArgoCD for a centralized GitOps workflow. I'm seeking thoughts on whether this setup is reasonable and where the responsibilities of Pulumi end, and those of ArgoCD begin. Specifically, I'm considering:
- Using Pulumi to provision k3s via cloud-init, then setting up Rancher.
- After the Rancher node is set, utilizing the Rancher provider to create an RKE2 cluster, allowing Rancher to manage the provisioning.
- Once the cluster is up, implementing ArgoCD projects/apps for deployment.
I'm keen to hear if this sounds like a solid and sustainable setup!
3 Answers
It seems like the Velero restoration woes stem from issues with etcd or Rancher state rather than the tool itself. If I were in your place, I’d lean towards using Terraform or OpenTofu for infrastructure and GitOps for managing apps, treating clusters like cattle instead of pets. What's your thought process on your recovery time objectives (RPO) and current threat model?
I’m using GitHub Actions and Terraform with plans to switch to OpenTofu. I previously faced downtime after migrating to Istio from ingress-nginx, and it messed up my cluster a bit, but my setup is now pretty straightforward. I think your proposed workflow looks good, but just be careful that Pulumi doesn't interfere with anything within ArgoCD. If it’s interacting with your microservices, that might indicate a misconfiguration. Otherwise, it sounds solid to me!
Your approach with Pulumi and ArgoCD is pretty much the industry standard: Pulumi or Terraform is responsible for everything up to the cluster with ArgoCD taking over once it's up. I’d suggest that Pulumi manages the infrastructure outside Kubernetes (like VMs, networking, etc.), and ArgoCD should handle Kubernetes manifests - apps, cert-manager, ingress controllers, etc. Also, if you’re going full IaC, consider skipping Rancher completely and just go with k3s or RKE2 directly. Rancher can introduce complexity that may not be necessary for a single cluster setup.
You mentioned that if it's interfering with microservices, it could be an issue. What do you suggest for provisioning ArgoCD itself? Would you still use Pulumi for that too?