I'm curious about how to effectively handle update cycles for systems that need to be online 24/7, particularly in fields like emergency services or other critical infrastructures. What methods do you use for scheduling these updates? Do you prefer to do late-night updates once a month, or perhaps quarterly scheduled updates? Also, how do you convince management, who might be resistant, about the necessity of these updates?
5 Answers
The approach we take includes a monthly maintenance window during late nights, but we also incorporate redundancy with a separate Disaster Recovery site. We usually run updates on the DR site first, then failover, ensuring almost seamless service continuity and updates with minimal downtime.
Ultimately, communication is essential. Bring the department heads into discussions early, set expectations about outages, and frequently remind everyone about the need for maintenance. When pushback occurs, ask for valid reasons against the schedule; usually, having managerial backing helps clear the way for necessary updates.
Redundancy is key! If something has to be online all the time, you should definitely have at least two (or more) systems running. This allows you to perform updates without significant downtime, as users will never notice if one system is being worked on while the other one is live.
Absolutely! In Site Reliability Engineering, we say "One is Zero"—you need at least two.
Generally, I have better luck getting my changes approved if I point out regulatory requirements we need to meet. Explaining the potential penalties to management when they resist makes it clear that we must go ahead with updates, even if they have concerns about scheduling.
When it comes to critical systems, you should always ensure there’s enough redundancy so users aren't aware of the updates happening. In our setup, we do maintenance during business hours when more staff are available to help if anything goes wrong. It might not matter if an outage happens at 10 AM or 10 PM, so having support on hand is crucial.
Right! I've found that updating during the day makes it easier to get vendor support in case things go south.
This is a solid strategy. Making sure to regularly use the DR site keeps it from going stale.