Hey there! We're currently developing a large e-commerce SaaS platform aimed at automating processes across various marketplaces. We've hit a bit of a snag regarding customer instance management. Essentially, our architecture comprises both customer-specific and shared services, with each customer having their own PostgreSQL database and several message brokers. At the moment, we have to manually execute Ansible scripts for each new customer to set up these specific services, initiate the database instances, create message broker topics and queues, etc. I'm curious about the best approaches to run multi-instance systems in a production environment using Kubernetes.
Additionally, we utilize GitLab for our CI/CD pipeline, and one of our developers proposed leveraging it as the entry point for deploying customer instances. I'm not so sure that's a great idea, but I'm struggling to articulate my concerns. If anyone has real-world examples or suggestions, I'd love to hear them!
5 Answers
You might want to consider whether you truly need separate physical instances of PostgreSQL and message queues for every single customer. It might be feasible to use a logical database setup, where each customer has its own database with distinct credentials or dedicated topics in the message broker.
You could manage your service manifests by storing them in Git and letting ArgoCD handle the deployment. Check out this [Cloud Native PostgreSQL](https://cloudnative-pg.io/) tool, which simplifies database creation for deployments. For message brokers like Kafka, the [Strimzi](https://strimzi.io/) operator can be super helpful. Overall, this could really streamline your process:
1. Customer signs up
2. System generates the necessary YAML files
3. Push them to Git
4. ArgoCD applies the configuration
5. Operators manage everything else.
Alternatively, you could create a single Custom Resource Definition (CRD) for "customer instance," allowing your operator to generate all resources needed (deployments, services, PostgreSQL cluster CRDs) without muddling your customer signup service with too much Kubernetes knowledge.
Have you thought about using a Helm chart? Could make the deployment process a lot smoother!
I really think you should avoid turning your CI/CD pipeline into a production API. GitLab CI is fundamentally designed to push code updates, not to handle customer provisioning. If you suddenly get a wave of signups—like, say, 500 in one hour—your GitLab queue might get backed up, your runners could crash, and you’d be looking at a lengthy sign-up process. Plus, tying your business logic to your development tools is generally not a good practice. Automate everything you can and make sure your customer sign-up process doesn't require any manual intervention, or it’ll feel more like a consulting service with a login page!
Absolutely, convincing my colleague about that is a challenge too!
There are multiple ways to approach this! We developed our system using an operator model. Our API sends a manifest (Custom Resource Definition), which then takes care of initializing everything—databases, caches, message queues, services, job servers, and so on. We keep these components mostly decoupled from each other, using init containers only when absolutely necessary for startup.
Totally! We did something similar by developing an operator and adding a user-friendly web interface for our customer support team. They log in with OAuth, and all their changes get pushed to Git along with their username for accountability. Argo monitors the repo and handles deploying changes to production automatically!
We’re actually aiming to offer both options. For larger clients, we want a dedicated infrastructure, possibly an entire node, while for smaller ones, we plan on managing multiple databases within the same PostgreSQL cluster. Currently, we’re managing it within one cluster, but scalability is tricky for larger clients. Customer data—including connection details—resides in a separate database and can easily be managed through Vault for configuration.