Why Does My ESXi VM Freeze After 30-60 Minutes with GPU Passthrough?

I'm working on setting up GPU passthrough with ESXi 8.0 U2, and I keep hitting a snag. My VM boots perfectly fine with the GPU assigned, but about 30 minutes to an hour later, it just freezes up completely. When this happens, the VM becomes unresponsive in the vSphere UI, and the only way to recover it is by powering it off. Sometimes, even after shutting it down, I can't get the VM to power back on without rebooting the whole host. Here's a bit about my setup and the troubleshooting I've attempted: I'm using Asus 870e Rog hardware, and I'm testing with NVIDIA A2 and A16 GPUs, all passed through via PCI passthrough. I'm running ESXi 8.0.0 U2. I've made a few tweaks to the VM configuration like disabling svga, playing with the hypervisor flags, and toggling the hot add settings. I've also observed that nvidia-smi doesn't show any info on the host, which I expected since I'm using passthrough. The VM freezes when it's idle or after some usage, but not right after booting. Additionally, I've found logs about the TPM 2.0 device not having the TIS interface active and some NVRM entries. So, I'm really puzzled about what might be causing this freeze after running fine for half an hour or so, and why I need to reboot the host to get it back online.