Hey everyone, I'm running into a frustrating issue where my pods get stuck at creation with a message stating that the 'PersistenceVolumeClaim is being deleted.' This usually happens when we're rolling out a restart of our deployments during patching. Several of my deployments share the same PVC that's bound to a PV reliant on remote file systems. Occasionally, I see this problem arise where the new pods cannot start. To resolve it, I find that I have to scale down all the pods to zero, which leads to unwanted downtime. What's puzzling is that we never send any delete requests to the API server, and the PV reclaim policy is set to 'Delete.' In theory, the rollout restart shouldn't remove all pods simultaneously, so the PVC shouldn't be getting deleted. We deploy our pods to a cloud provider, so I don't have full visibility on how the API server is responding to calls. I'm suspecting some API calls may be getting processed out of order. Has anyone else experienced a similar issue?
1 Answer
It sounds like the PVC shouldn't be getting deleted just because the pods are scaling down. Usually, the PVC getting deleted means that something triggered it, possibly a misconfiguration or a specific API call. Are you certain no other processes are interacting with it?
I think you're on the right track. If all the pods using the PVC disappear, it might trigger an automatic deletion of the PVC, depending on your configuration. Have you checked any documentation from your cloud provider?