Hey there! I'm in the process of designing a brand new Kubernetes architecture and I'm looking for some guidance on best practices to ensure smooth growth and scalability.
Here are some specific areas where I'm seeking advice:
- **CI/CD Tools**: What do you think is the best approach? Should I go with ArgoCD, Jenkins, or a combination of both?
- **Repositories**: Is it better to have a single repository for all DevOps/IaC configurations, or should I consider:
- A dedicated repo for ArgoCD to consume, with multiple pipelines pushing versioned manifests?
- Or multiple repos, each monitored by ArgoCD for deployments?
- **Helmfiles**: Should I utilize well-structured Helmfiles with a focus on manual deployments, or should I automate fully?
- **Directory Structure**: What's an ideal, scalable repo structure for GitOps and IaC practices?
- **Best Practices**: What patterns should I adopt to build a solid foundation for GitOps and IaC that's well-structured, versionable, and ready for the future?
### Some context:
- I have 4 years of experience in infrastructure, starting with datacenters, telecom, and ISP networks, and currently work as an SRE/DevOps engineer.
- Right now, I manage a self-hosted k3s cluster (6 VMs on a 3-node Proxmox cluster) used for testing and development.
- The plan is to migrate entirely to a full Kubernetes setup: development and staging will be self-hosted (eventually moving from k3s to vanilla k8s), while production will be on GKE.
- Currently, our production workloads are primarily containers, serverless services, and microservices, with only a few VMs.
My goal is to create a fully Kubernetes-native environment with clean GitOps/IaC practices that can scale as we grow. What are your recommendations regarding CI/CD design, repo strategy, directory structures, and GitOps patterns? Thanks for the help!
6 Answers
Have you checked out resources like the GitOps CI/CD GitHub repository? It's packed with ideas and structures for repository management. I mean, it really differs from team to team, but I find keeping all Kubernetes resources in a single repo simplifies things. It avoids confusion and makes changes easier to deploy.
Yeah, I've heard that model can keep everything organized!
When it comes to CI/CD tools, I'd recommend choosing one orchestration system to avoid chaos—Argo or Flux could be great options. Try to keep your number of repositories to a minimum; many teams find success with a "platform" repo for cluster setup while handling apps in separate repos for better access management.
For Helmfiles, some folks find them unnecessary—consider relying on Kustomize instead, paired with ArgoCD for deployment. And definitely do not overlook the importance of keeping your staging environment close to production—this makes a world of difference when you're testing deployments!
That's true! I think reducing the complexity by simplifying your repo structure will save you headaches down the road.
Definitely! It's also important to educate your team on CI/CD processes to prevent confusion later.
For your directory structure, think of organizing it around your stages (dev, prod, etc.) and leverage Argo’s app-of-apps pattern to manage different applications within each stage. It's crucial to align your directory structure with your deployment process, and ensure that it scales as you grow. You don't want to rediscover these practices as your environment expands!
I think there's a lot to consider when planning your environments. I wonder about using self-hosted for dev and staging while production is on GKE—shouldn't staging match production closely to catch issues earlier? Also, what kind of workloads are you dealing with? If they're heavy in databases or processing, your deployment strategy might need extra care.
For CI/CD, if you don’t have a huge number of deployments, using Helmfile alongside GitHub Actions could work fine. But with many moving parts, ArgoCD and some rendered Helm manifests could be more effective. Just be aware, setting this up can be quite the undertaking!
Agreed! Staging should be as close to production as possible to avoid surprises. It's smart to plan for heavy workloads up front, too.
Absolutely, the smoother you set up staging now, the better your rollout will be later on.
Thanks for the tip! A centralized repo does sound appealing for productivity.