I'm building a highly available homelab to learn more about on-prem infrastructure. The setup is a three-node Proxmox cluster, with each node running VMs for DNS, HAProxy, a Patroni database node, and Apache web servers. HAProxy handles load balancing, reverse proxying, TLS termination, and configuration deployment through infrastructure-as-code as web servers are added or removed.
The web servers need to serve a collection of images. I originally stored them on a NAS, but that created a single point of failure. I later moved the storage to Ceph for high availability, although performance is noticeably limited because the nodes are connected over 2.5 Gbit/s networking.
I'd like to experiment with image caching, both to improve read performance and to learn which approaches are practical. The options I'm considering are:
- Using HAProxy's built-in cache directly
- Adding a caching proxy such as Varnish or Nginx between HAProxy and Apache
- Using Apache's mod_cache or a filesystem cache on the shared storage
For anyone with experience running these in production or homelabs, which approach would you recommend? Are there important limitations or design considerations, especially around cache size, disk storage, cache invalidation, and keeping multiple cache nodes consistent?
3 Answers
If the only goal is high availability, caching may not be necessary because it doesn’t remove the dependency on the origin storage. It mainly reduces latency and repeated reads.
For experimentation, Varnish and Nginx are both sensible options. Varnish is a good general-purpose HTTP cache and can handle more than static images, while Nginx may be the simpler choice if the requirement is specifically to cache image files. Either way, restrict caching to appropriate paths and define how objects are refreshed when files change.
HAProxy caching is worth testing, especially for small or moderately sized objects. You can choose which requests are eligible for caching, inspect cache statistics, and compare hit rates with other designs. Just be careful with large files, because keeping them in RAM can consume resources quickly, and separate HAProxy nodes won’t share cached objects.
A practical experiment would be to test HAProxy’s cache against Nginx or Varnish using the same image set, then compare latency, memory or disk usage, cache-hit ratios, and behavior after a cache restart. There’s no reason you can’t run more than one design while learning.
That’s a good point. I’ve been focusing heavily on Varnish, but comparing it with HAProxy’s built-in cache would give me a much better understanding of the trade-offs.
HAProxy does include a built-in cache using its cache configuration and HTTP request/response rules. It stores data in RAM, and each HAProxy instance maintains its own cache, so it’s best suited to small, frequently accessed objects such as thumbnails, icons, or CSS files. It probably isn’t the right choice for a large image library.
For larger objects, Nginx with proxy_cache or Varnish with a disk-backed cache are more typical choices. Keep in mind that caching only improves speed; if the cache misses, the request still has to reach the origin storage. If availability is the main objective, consider replicating the images onto local disk on each web VM or using distributed object storage. Then caching becomes a performance optimization instead of something required for the service to remain available.
That makes sense. I’ve already moved the images from the NAS to Ceph, so the storage layer is redundant even though it’s slower than I’d like. I’ve started experimenting with Varnish and it’s been a useful learning exercise.
The caching is mainly for learning and improving read performance. The lab itself is already an educational project, so trying several approaches and comparing their behavior sounds worthwhile. Varnish and Nginx are now at the top of my list.