I've set up GitLab with MinIO on Longhorn, and I'm currently using a PVC that has a 30GB capacity. Longhorn reports around 23GB being used, but in the MinIO UI, it only displays about 200MB of actual data. I'm curious about why there's such a large discrepancy between the PVC usage reported by Longhorn and the data displayed in MinIO. Could this be due to metadata, snapshots, or leftover files? Has anyone encountered a similar situation or have tips on how to troubleshoot this issue? Thanks a lot!
3 Answers
It's possible that snapshot overhead and unclaimed deleted blocks are inflating the PVC usage. Make sure to check both Longhorn snapshots and the MinIO temporary directories, as well as diving into the PVC filesystem usage for clarity.
Longhorn shows the actual usage at the host disk level, not the filesystem level of the volume. This means if your app keeps writing data in small increments, eventually all the blocks will be used, and Longhorn will report high usage. It could also involve snapshots causing the reported size to exceed the volume's max size. To get a clear view, you can monitor the PVC usage with Prometheus for more accurate filesystem data.
One common issue with MinIO running on Kubernetes is the way storage is set up. If you have multiple MinIO volumes tied to the same storage class, you might see logical usage reported as lower because of MinIO’s erasure coding. Longhorn manages storage across nodes, which can amplify data sizes significantly due to this coding and striping. So, although your logical data usage looks low, the physical is quite high. Just keep that in mind when setting up your storage!
Absolutely! I have a scheduled trim job for each volume that helps keep things in check.