Most of the cooling in our server room suddenly failed around 23:00. Several servers shut down after their inlet temperatures reached about 43°C, including a virtualization host and multiple storage nodes. I also found that the circuit breaker for roughly two-thirds of the cooling equipment had tripped, and it immediately trips again when reset, apparently because the differential protection is activating.
I have already shut down equipment that is not essential and contacted the electricians and HVAC contractor. Until they arrive, what are the safest temporary ways to keep the room cool? Would exhausting air with large fans, opening doors or windows where possible, or bringing in portable AC units be effective?
4 Answers
Do not keep forcing that breaker on. A breaker or residual-current device trips for a reason, and repeatedly holding it closed could create a shock or fire hazard. Have a licensed electrician and the HVAC contractor diagnose it. In the meantime, shut down anything nonessential and follow your emergency or disaster-recovery procedure.
The temperature alarm did its job. Many server platforms begin warning in the high 30s Celsius and may shut down around 43°C, particularly when that is the inlet temperature. Keep monitoring inlet temperatures and humidity continuously, and shut down equipment in a controlled order before it reaches the thermal cutoff. Afterward, consider redundant cooling, standby portable units, environmental sensors with automatic alerts, and testing the emergency plan regularly.
Use fans to move hot air out of the room rather than simply circulating it. If the building layout allows it, crack open doors or windows on the exhaust side and put large fans at the doorway pulling air outward. This is only a short-term measure, and you need to make sure you are not spreading heat into another occupied or sensitive area.
That works reasonably well for a small room with only a few racks, but it may not keep up with a full server room. Portable AC units or rented spot coolers are much better if you can safely provide power and route their hot-air exhaust outside.
Portable spot-cooling units are the best temporary option if you can source them quickly. Check their actual cooling capacity, electrical requirements, condensate handling, and exhaust routing before putting them in the room. A normal household portable AC may not keep up with several racks, especially on a hot day, so treat it as supplemental cooling rather than a guaranteed solution.
A single portable unit may not be enough. We once had supplemental cooling during an HVAC failure, but it still could not keep up with multiple storage systems on a hot day, so sizing and a planned shutdown sequence matter.