We have an AI-related Python app running with Uvicorn in Kubernetes, and its memory needs keep growing. Jenkins builds an image from GitLab and pushes it to Harbor, then Argo CD deploys it.
We are considering a dedicated Linux server but still need automatic deployment, a domain name, backups and a quick way to recover. What setup would you recommend?
Before moving it out of Kubernetes, consider adding a high-memory node to the cluster and dedicating it to this app. Taint the node and add a toleration to the app, then use a node label and affinity to make sure the app lands there. You can keep the existing Jenkins, Harbor and Argo CD flow.
That isolates its resource use without giving up the orchestration you already have. A single dedicated node is still a single point of failure, though. Rescheduling elsewhere only helps if another suitable node has enough memory.
I'd look at the memory curve before choosing the server size. AI workloads can genuinely need a lot of RAM, but that doesn't rule out a leak or avoidable duplication.
Check how many Uvicorn workers are running and whether each worker loads its own copy of the model or other large data. Compare memory after startup with memory under load, and watch whether connections or sessions leave it climbing after traffic drops. That should give you a better capacity target than just raising the limit each time.
If the separate server is still the plan, keep building the container in Jenkins and publishing it to Harbor. Have a deployment step on the server pull a specific image version and start it with a restart policy, using a container manager or systemd. Argo CD manages the Kubernetes deployment, so a standalone host needs its own deployment mechanism.
Put a reverse proxy in front of Uvicorn for the domain and TLS, and add a health check and monitoring. Keep the previous image available and test rolling back to it. Also be clear that restarting a process on one server is not the same as surviving that server failing.
8x~staplejar_307:
Use an immutable image tag or digest for deployments so the rollback points to the exact build you want to restore.
Separate the application deployment from data recovery. The container image and server configuration help you rebuild the service, but they are not backups of any persistent data the app uses.
Identify what needs backing up, where it lives and how much data loss is acceptable. Store backups away from the server and test restoring them, including any configuration and secrets needed to bring the service back. A spare server or a tested rebuild procedure will matter if the machine itself dies.
Answers are reviewed before publication.
Ravlin-7p:
A dedicated node sounds worth exploring. I had been assuming the app needed to leave the cluster to get its own hardware.
a.614~birdbath:
Check that the rest of the cluster has enough spare memory for a replacement pod if you expect it to run elsewhere during maintenance or a failure.