How can I spread Longhorn replicas across three physical Proxmox hosts?

0
1
Asked By MellowPine42 On

I have three physical Proxmox servers running a K3s cluster. Each server has a control-plane VM, and servers two and three also run worker VMs. I want to deploy Longhorn with three volume replicas so that each replica is stored on a different physical server. That way, losing one complete Proxmox host will not take down the data. I am confused about how Longhorn schedules its components and replicas, especially because one physical server currently has only a control-plane VM. I have tried node labels and tolerations, but some Longhorn UI and driver pods remain stuck or enter an error state. What is the correct topology and configuration for this setup?

4 Answers

Answered By QuietRaven18 On

Longhorn system components and volume replicas are separate concerns. You can use node selectors, taints, and tolerations to control where managers, UI, CSI drivers, and other Longhorn pods run. For replica placement, label nodes by physical host and disable soft replica anti-affinity if you want a hard requirement that replicas cannot share a node: `defaultSettings.replicaSoftAntiAffinity=false`. Also configure the StorageClass with three replicas. Check the pod events and Longhorn logs rather than relying only on the pod status when the UI or driver is failing.

Answered By CopperLynx63 On

For production, I would avoid depending on a control-plane-only VM as a storage node unless you deliberately tolerate its taint and have verified the disk setup. The cleaner design is to run at least one worker/storage node on every physical server, then schedule Longhorn storage across those three workers. If one host has no worker or Longhorn disk, Longhorn cannot guarantee one replica per physical host, regardless of topology labels.

Answered By OrbitMango7 On

Treat the physical servers as failure domains, not the individual VMs. Label the Kubernetes nodes according to the host they run on, for example `topology.kubernetes.io/zone=server-1`, `server-2`, and `server-3`. Then configure the Longhorn StorageClass with `numberOfReplicas: 3` and enable replica zone awareness or replica auto-balancing. Longhorn can then place one replica in each zone when suitable storage is available. Make sure each physical host actually has a Longhorn disk exposed to a Kubernetes node; labels alone do not provide storage.

Answered By NimbleCedar5 On

You can reduce the default replica count if your workload accepts the risk, but three replicas are appropriate when the goal is surviving a complete host failure. Keep in mind that Longhorn replication is not a backup: accidental deletion, corruption, or an application-level problem can be copied to every replica. Configure recurring backups to storage outside the three Proxmox servers as well.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.