Accidentally Deleted All Namespaces in Production! How Can I Recover?

It's a major bummer, but really, you should restore from the backups you mentioned. Unfortunately, stuck terminating objects usually won’t just magically reappear. You should redeploy your applications and PVs, and then restore everything from your backups. If you have etcd backups, those can be lifesavers!

Let this be a lesson in disaster recovery! It’s good to have stringent checks in place for production environments, like limiting permissions and having a monitoring system. I’d recommend implementing a better GitOps process for next time to avoid this kind of chaos.

To recover, take any etcd snapshots you have. That could allow for a restoration of your system to a previous state. If the Persistent Volumes are still intact, you might also be able to recover data directly from the node they’re tied to, but it’s risky. Good luck!

First off, that's a tough situation! Unfortunately, once you've deleted namespaces in Kubernetes, there’s no built-in way to "undelete" them. It's crucial to have a good backup system in place, like a GitOps setup, to easily recreate your apps. For the volumes stuck in terminating state, if you set your reclaim policy to "retain", you can create new Persistent Volumes (PVs) and link them to your old Longhorn volumes. Fingers crossed for you!

I totally get the panic. Sounds like a rough moment! If you bought the server recently and are experimenting, it really shouldn’t be too important, right? Maybe starting from scratch could be cleaner and faster at this point. Just a thought!