Hey everyone! Today we hit a snag with cache busting while performing rolling updates, and I'm curious how you all manage similar situations. Here's a quick rundown of our setup and upgrade process:
1. We have an nginx load balancer that sits in front of several upstream web servers.
2. Caching is enabled on the load balancer for static files like CSS and JS, based on the URL and parameters.
3. When we update the CSS files, we bump the version in the HTML, for example changing style.css?v=1.0.0 to style.css?v=1.0.1. This change busts the cache, and the new file is cached correctly.
However, during a rolling update, we may have one web server upgraded to 1.0.1 while another still serves 1.0.0. If a client hits the site and gets directed to the upgraded server but the CSS request goes the old server, they end up getting the outdated file (v1.0.0) instead. This creates a cache mismatch.
I'm considering a few solutions:
1. Delete the cache on the load balancer after each deployment (but that seems counterproductive).
2. Disable caching before starting the deployment and re-enable it afterward.
3. Disable caching for versioned assets altogether, relying solely on the parameters for busting browser cache.
What do you think? Any other suggestions or thoughts on this issue? Keep in mind our web servers are immutable containers, so we can't just update the CSS files beforehand.
1 Answer
To avoid this caching issue, you might want to temporarily take all but one server offline during updates. Start by upgrading one server completely before bringing it back online, then do the same for the others one by one. It prevents cross-version problems but does introduce some downtime.
That's true, but it could work. Instead of total downtime, consider updating one server, then quickly toggle the others offline to upgrade them all in batches. Might streamline the process a bit!