Is it a good idea to disable VBS?

Can you explain what happens if I just disable core isolation instead of VBS? Does that keep some security while also giving me better performance?

I'd recommend leaving VBS on and just disabling core isolation if you're feeling the performance hit. Disabling VBS entirely can risk your security, and while gaming might see a slight improvement, 2-5% isn't a huge gain for potential vulnerabilities. Just keep in mind that VBS helps protect against various attacks.