Hey folks,
I'm managing a bunch of clusters across various environments and applications at my company, and while things are running smoothly for now, I've started to get a bit worried about how we're setting everything up and keeping things organized. Not every developer here is comfortable with AKS, or even infrastructure for that matter. Most of them just want an easy way to run their applications without diving into maintenance or complicated configs.
So I've been brainstorming a concept for a shared cluster where developers can host their workloads and easily request the services they need. We typically have three different environments for our apps—DEV, QA, and PRD—and I really don't want to mix these in a centralized cluster setup. Each environment should have its own separate cluster for better isolation and so our Platform team can test changes before they hit production (we do have a separate dev-test cluster for this).
I want to make things as straightforward as possible for developers while keeping security in mind. Ideally, I'd like them to create all necessary resources with a self-service approach using predefined templates (like TerraForm or ARM). This would include things like:
* Cluster namespace
* Databases
* Configuration Management (like App Configuration)
* Event Systems (e.g., ServiceBus or similar tools)
* Identity & Access Management (permissions, etc.)
Currently, I've laid out a concept that involves managing resources while using something like Git with PR approvals to oversee deployments. Here's what my current setup includes:
* SQL database creation in a central SQL server
* Namespace and service accounts creation using Workload identity
* Group and RBAC configurations
* A Terraform module for namespace management (with plans for Terragrunt later)
* DNS and Certificate integration (using app service routing at first)
Here are my main questions:
* Do you have any concerns about the shared cluster approach with a central team managing it?
* Are there tools that support projects creating their own necessary resources, especially for external services (like Azure)?
* Any recommendations on critical things to keep in mind with this approach?
I appreciate any advice you can offer!
3 Answers
It sounds like you're looking for an Internal Developer Platform (IDP) that lets your end-users request the resources they need on demand. This could streamline your process by empowering the devs instead of making them depend on central management for every little thing.
For a multi-tenancy setup, you might consider using Capsule. It's designed to create tenancy boundaries across any Kubernetes distribution. Plus, check out their ResourcePools for effective resource management. To abstract the user-facing API, you could look into Crossplane; it allows users to order services within their own tenants, all while managing everything themselves. This keeps your setup clean and efficient.
Starting off, I'll stick with Kubernetes' default resource quotas, which should work for our initial needs. But I'll definitely explore Crossplane as it looks promising!
Don't just separate by namespaces; think about quotas and resources as well. You could use node labeling to assign specific resources to teams. For example, Team A could have resources on 2 nodes, while Team B uses 4, but they both share a common space for services like Redis.
What’s the advantage compared to just using resource quotas? Wouldn't that conflict with Kubernetes' general design? I was considering dedicated node pools for more critical apps.
Yes, that sounds awesome! Thanks for the suggestion.