I'm dealing with a situation where some Kubernetes jobs are creating pods that end up stuck in the `Terminating` state. I've noticed the pods have the following finalizer added to them, likely through a mutating webhook associated with the jobs:
```
finalizers:
- batch.kubernetes.io/job-tracking
```
These jobs aren't cleaning up properly, and I can't seem to delete the pods—even a force delete just hangs due to this finalizer. The problem is that I can't remove the finalizer on the pods since they are immutable. I've come across some older bugs that seem related, but they are not recent. We are running Kubernetes version 1.30.4.
Interestingly, this issue has only cropped up on one particular cluster. Some older reports mentioned this might happen if the cluster is overloaded. Has anyone else encountered this problem or have suggestions on how to resolve it?
3 Answers
Have you thought about using `kubectl patch` to try and remove that finalizer? That might help you out of this `Terminating` limbo.
You could also use a command like `kubectl patch
I managed to delete the pods by removing both the finalizer and the `qosClass` field first. However, it feels like there's a deeper issue here. The documentation states that this finalizer should be automatically cleared by Kubernetes.
I noticed you mentioned you're using k8s v1.30.4. That version is pretty old and it's no longer supported. There have been many fixes related to job tracking with finalizers over recent versions. Have you considered upgrading? It might resolve your issue quite easily.