Hey everyone! I'm new to Kubernetes and I'm curious about an architecture idea I have. I'm thinking of setting up a Kubernetes cluster that interacts with a message queue—each message would contain the name of a Docker image. The goal is for Kubernetes to dynamically create pods using the images defined in the messages. I know this might not be the conventional approach, but it's intended for a cluster of worker nodes that will execute user jobs. Each worker would run the job, then terminate and clean up afterward. Any advice, tools, or articles you can recommend would be super helpful! Also, just to clarify, my aim is to run custom user-written Python code, and I want users to have the freedom to import any packages they like. That's why I thought it would be easier to allow users to configure their environments rather than constantly managing the execution environment for each worker.
10 Answers
Running an external message trigger to execute containers based on user input is a big security concern. Instead, consider giving users access to your Kubernetes API, then use tools like Kyverno or Gatekeeper to manage which containers can run. Implementing quotas will also help protect your cluster from potential abuse.
You should definitely explore Knative eventing along with RBAC for your use case! That could simplify a lot of management for you.
Why not use a generic autoscaling worker instead of spinning up a new Docker image for each user job? These workers can simply pull jobs from the message queue and scale based on metrics. It's simpler and avoids the hassle of custom setups.
I did think about having an autoscaled cluster, but managing package dependencies in user code seems tricky. I thought letting the users set up their environment would be more flexible. What do you think?
Don’t forget, you'll need to implement some code that listens to your message queue and creates Jobs or Pods in Kubernetes. If Python is your jam, check out the Kubernetes Python client documentation [here](https://github.com/kubernetes-client/python/blob/master/kubernetes/README.md#getting-started). You’ll find functions for creating both jobs and pods, which can help manage your workflow.
Thanks! That might be just what I need—to run some jobs for Python code temporarily and then clean up!
You might also want to look into ArgoCD deployed on your cluster. It syncs with a Git repository and you can process Helm charts automatically based on changes. That could streamline deployments for you!
You'll likely want to build a simple application that takes your "messages" and converts them into Kubernetes jobs using the Kubernetes API. Check out the Kubernetes documentation on jobs [here](https://kubernetes.io/docs/concepts/workloads/controllers/job/). Depending on how big your setup is and your security needs, running jobs in the same cluster could be risky.
I’m just looking for a proof of concept right now, but that’s definitely something to consider later. Thanks!
Custom Resource Definitions could also be a great fit for your setup. You might find this documentation useful: [Custom Resource Definitions](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/).
Thanks, I’ll read more about that!
Don’t forget to look up "Docker in Docker" (DnD). Scaling with KEDA might involve a deployment that has a Docker sidecar to handle those cases effectively!
I’ll do that, thanks!
Have you checked out KEDA? It offers scaled jobs which might fit your needs!
I’ll look into that, thanks!
If you're expecting to scale, consider using Argo Events alongside Argo Workflow. They could help manage job submissions and execution seamlessly.
Can you clarify more on the security risks? I thought containers isolated it. I’ll also keep your suggestions about quotas in mind!