We're looking for a way to automatically scale down performance-lab workloads when testing isn't running. We have hundreds of namespaces and more than 8,000 workloads, so manually managing schedules or creating configuration for every workload would be difficult. The goal is to reduce cloud costs, ideally in combination with cluster autoscaling so unused nodes can also be removed. I'm considering kube-green, kube-downscaler, KEDA, and similar tools. Has anyone used these at this scale, and what approach worked best?
4 Answers
KubeElasti is another option intended to scale existing workloads without requiring application changes. It can scale a target back up when traffic arrives, queue requests while the workload starts, and handle health probes so probes don’t unintentionally trigger scaling. It may be worth evaluating if the environments need on-demand wake-up rather than only fixed schedules, although you would still want to confirm how it handles automated configuration across thousands of workloads.
Knative could scale workloads down based on incoming traffic, but it usually requires application-side changes and workloads that tolerate cold starts well. KEDA and Knative make more sense for cloud-native services that can start quickly, whereas scheduled tools such as kube-green or GoKubeDownscaler are likely a better match when the main requirement is turning entire test environments off outside testing hours.
The original kube-downscaler project has been replaced by an actively maintained Go rewrite called GoKubeDownscaler. It was designed for shutting down many namespaces through a global schedule, with overrides available at the namespace or individual resource level. It supports recurring schedules, specific time ranges, and more complex conditions. For a large performance-lab environment, this kind of centralized scheduling may be a better fit than configuring every workload separately. kube-green can work too, but its scheduling and scaling behavior are somewhat different and generally more namespace-focused.
That’s helpful—centralized scheduling and per-namespace overrides are exactly the sort of controls we need. I’ll investigate the Go rewrite further.
KEDA is a common choice for event- or metric-driven scaling, and kube-green is also fairly popular for scheduled shutdowns. However, KEDA may require creating a scaling object for each workload, which could become difficult to manage across 8,000 workloads. You’d also need to consider the number of monitoring queries and the extra load on Prometheus.
That was my concern too. Changing the applications to support Knative would probably be a much larger project than adding scheduled workload scaling.