I have a staging cluster with several microservices that typically runs 24/7, but I've noticed it just sits idle at night and on weekends, which feels like a waste of resources and money.
I initially tried using a cronjob alongside a kubectl script to scale down the deployments at night, but that setup became unreliable over time due to eviction issues and conflicts with other cronjobs. This led me to build a more complex solution—a visual flow builder that manages scaling as a Kubernetes operator. It automatically reduces the scale of my deployments in the evening and restores them in the morning. However, I'm curious whether others also have solutions for automating this or if simple scripts have worked adequately for them. What approaches do you all use to balance cost-saving and functionality in staging environments?
5 Answers
I built a custom Python script that runs as a cron job and handles putting our cluster to sleep. It saves state in a ConfigMap and also backs it up to S3. Waking it up was tricky—I ended up going with a Lambda function over EKS or keeping a cheap node running 24/7. It’s been working really well, and I feel positive about our cost savings!
Check out GoKubeDownscaler. It's a handy tool that lets you set up an off-hours schedule via a ConfigMap or argument. It can quickly scale down Kubernetes resources when not in use. Plus, you can exclude specific namespaces and resources from being affected, which can be super beneficial if you have teams across time zones. Just a heads up, I help maintain it, so feel free to ask any questions!
Thanks! This looks awesome! But my main interest is understanding if there's a real need for cloud-native automations like what you're doing.
You might want to consider using tools like Karpenter or the Cluster Autoscaler. They can help to optimize the scaling process in your staging environments and might work better than scaling down deployments manually.
Why are we feeling so shouty about this? Could we tone it down a notch?
I’ve seen ScaleOps do pod-level scaling, which enhances the efficiency of Karpenter and CAS. Definitely worth checking out!
Honestly, if your staging environment is supposed to mimic production, it should stay up. However, if you're looking to save money, you might want to consider deleting the cluster and recreating it instead of just scaling down the deployments. This can help manage costs more effectively. It just depends on how much time it takes to recreate it.
Recreating the cluster daily sounds extreme. How small are your staging clusters for that to work?
How long does it take to set that up? Are you using something like FluxCD?
Instead of using a cronjob for scaling down, try utilizing native job resources and ensure you have a Pod Disruption Budget set. This way, your resources can scale down properly without interruptions, or you can leverage auto-scaling or KEDA triggers as well.
Why do you need to backup your ConfigMaps? Do you also put the K8s API servers to sleep?