Most of the cooling in our server room suddenly failed around 23:00. Several systems shut down after their inlet temperatures reached about 43°C, including a virtualization host and multiple storage nodes. I went to the office and found that the breaker supplying roughly two-thirds of the AC capacity had tripped. It immediately trips again when reset, apparently through its differential protection. I'm calling an electrician and the HVAC installer, but what temporary measures are safe and effective in the meantime? Would fans, open doors, or portable AC units help, and what equipment should be shut down first?
4 Answers
The long-term fix is redundant cooling or a standby spot-cooling connection, plus environmental sensors tied to the building alarm system. Temperature and humidity alerts, along with water-leak and power monitoring, give you time to respond before servers start shutting down. Also review whether the cooling circuit, power distribution, and emergency procedures have enough redundancy for a single failure.
Keep the room closed if opening doors or windows would bring in hot, humid, or contaminated air. If the surrounding area is cooler, controlled exhaust through the doorway can help temporarily, but someone should monitor temperature, humidity, and airflow continuously. Shut down equipment in stages before it reaches its thermal protection limit rather than waiting for automatic shutdowns.
Use large fans to move hot air out of the room and bring cooler air in, but only if the building layout allows it and you can avoid pulling in dust or humidity. A portable AC unit or rented spot cooler is better if you can obtain one quickly; make sure its exhaust is routed outside and that its electrical load is safe. Fans circulating air inside the room without removing heat will only delay the temperature rise.
Portable units may not keep up with a room full of racks, especially on a hot day, so treat them as temporary assistance while reducing the server load.
Do not keep forcing the breaker on. A breaker that trips again—especially on differential protection—indicates a fault that needs a qualified electrician or HVAC technician. In the meantime, shut down nonessential equipment and follow your disaster-recovery or emergency shutdown procedure. Reduce the heat load rather than risking equipment or people.
That makes sense. I stopped trying to reset it and began shutting down systems that were not essential.
We have some monitoring, but this exposed a gap in the alerting and cooling redundancy. I’ll add that to the follow-up work once the immediate fault is repaired.