Since early this morning, six of our servers have completely frozen and needed a hard reboot. Most of the affected systems are hosted in AWS, but at least one on-premises server has had the same problem. Windows Event Viewer and system logs do not show anything obvious immediately before the lockups. We have not installed any recent Windows patches because updates are currently on hold. Is anyone else experiencing similar freezes, or have ideas for what infrastructure or software issue to investigate?
3 Answers
The pattern sounds similar to a virtualization or host-infrastructure problem. Check the cloud provider's account-specific health events and compare availability zones, instance hosts, and affected hardware. For some instance freezes, stopping and starting the machine can move it to another hypervisor and clear the problem.
This could still be related to a recent security fix or endpoint protection update rather than the normal Windows patches. Check whether any CrowdStrike or other security-agent changes were deployed today, and compare the affected machines with systems that stayed online.
Since multiple systems are affected but there are no useful Windows log entries, I would look for a common dependency: endpoint security, monitoring agents, recent configuration changes, network or storage infrastructure, and hypervisor events. Preserve crash dumps or console output if possible before forcing the next reboot, since a hard reset can remove evidence.
That was one of the first things we checked, but the issue is not limited to AWS. One of our on-premises data centers had a server freeze as well.