I help maintain shared Kubernetes clusters across Amazon, Azure, and Google cloud environments. Right now, workloads use each provider's native storage, but application teams choose from whatever StorageClasses are available and sometimes run into compatibility or performance issues. The clusters also autoscale, so worker nodes can be added or removed frequently.
I'm considering Ceph with OSDs backed by PVCs on a dedicated, stable node pool whose size does not change with normal cluster autoscaling. The goal is to provide a consistent storage experience across clouds without depending directly on each provider's storage implementation. We previously tried Longhorn, but it caused problems when moving workloads from on-premises environments to the cloud, and a managed enterprise storage platform was too expensive.
Has anyone deployed Ceph this way across multiple cloud providers? Is the operational and performance overhead worth it, or would it be better to expose a small set of standardized StorageClasses and map them to each cloud's native storage?
3 Answers
Ceph can work, but it brings substantial operational overhead in public cloud. For a reliable production cluster, especially with write-heavy databases, you would likely need several dedicated OSD nodes and careful planning around replication, disk duplication, write amplification, and network traffic. You could end up paying for redundancy on top of storage redundancy already provided by the cloud provider. A lighter storage operator may be worth investigating, but native storage behind a clear abstraction is probably the simpler option here.
Use StorageClasses as the portability layer rather than trying to make every cloud look like the same storage platform. Keep the interface small—perhaps `default`, `fast`, and `slow`—and define their performance and availability expectations clearly. Then use templates and policy enforcement to guide application teams toward the right choice. Ceph will not by itself fix unclear guidance or inconsistent workload definitions, and in a public cloud it may amount to paying twice for storage capabilities you already receive from the provider.
If the main goal is consistency between environments, you may not need Ceph at all. Define your own StorageClass contract, with names such as `default`, `fast`, and `slow`, and configure each cluster so those classes use the appropriate native backend. Avoid exposing provider-specific names like `ebs` or `ceph` to application teams. You can also enforce which classes are allowed with admission policies such as Kyverno or another policy engine.
That makes sense. The stable pool was mainly intended to keep the storage disks around while autoscaled worker nodes come and go, but standardizing a few StorageClasses may solve the application-facing problem more simply.
I agree that Ceph may be more than we need. Reducing the choices to two or three clearly defined classes and enforcing them through templates or policy should address most of the problems without adding another storage system to operate.