My IBM x3650 M4 Disk Rebuilding for Days – What Should I Do?

You can log into the server's IMM2 interface and check the logs there; it should show you any rebuild progress, if there is any. Since you're using vSphere, another option is to install MegaCLI on ESXi. It has commands that allow you to query the rebuild status directly.

I recently dealt with a similar scenario on a Windows server with Hyper-V. One drive failed and there were two others showing predictive failure. The rebuild was incredibly slow—like 0.1% every few hours. We decided to revive a secondary host and migrated all VMs before the rebuild to avoid downtime. Eventually, we suspected the controller was the issue and replaced it, which improved the rebuild speed significantly. If you've routinely lost drives every year, it might be worth investigating the controller's condition.

As for the safest next step, I’d suggest getting a new array ordered as soon as possible. While you're waiting for it to arrive, ensure your backups are restorable. It's crucial to be prepared in case things go south.

You should probably use the RAID tools for your RAID card to check on the rebuild progress. Just a heads-up, RAID rebuilds can be risky, especially if you're running RAID 5. The disks are usually worn down evenly, and rebuilding them can put a lot of strain on them. I'd recommend verifying your backups before proceeding with any rebuild. If you're in a cluster, check your HA settings. You might also want to consider temporarily migrating your VMs to another server before you start the rebuild, or even replacing all the disks if you're concerned about multiple failures.