Hey everyone! I'm working on a Kubernetes operator that manages SLOs with Custom Resource objects, and I've got an idea I'd like to bounce off you, especially those with SRE experience. I'm considering implementing a feature that blocks any edits, updates, or deletions on workloads that have used up their error budget once it falls below a certain threshold. I was thinking about adding annotations to allow overrides if necessary. I'm curious to know if this approach makes sense. All feedback is welcome—thanks!
3 Answers
I think it’s a cool idea, but hard-blocking in the operator could lead to problems during incidents when you really need to deploy fixes. Many teams prefer gating new rollouts in CI or using admission checks based on error budgets. Just ensure any overrides you put in place are clear and can be tracked.
It sounds like there's a misunderstanding of error budgets. They’re meant for teams to manage their service reliability, shifting focus from new features to stability. Just blocking fixes isn't going to help; teams need to be proactive in addressing issues instead.
I’m a bit confused. Are you suggesting stopping developers from deploying fixes for services that are having issues? That doesn't seem like a good idea.
Got it, sounds like this might not be the best approach for the operator. Maybe focusing on blocking edits to certain workloads but allowing for specific labeled ones could be better. I'll pivot to working on alert integration next.
Thanks for the insight! I realize now that blocking edits might not be effective for the operator. I’ll keep working on enhancing the alerting part.