How can I test new desktop environments without cluttering my home directory?

Create a separate test user account with a mostly untouched home directory. You can use it to see how applications behave with default settings, troubleshoot whether something is a real bug or just a personal configuration issue, and try software that may create lots of files. When you are finished, you can simply delete and recreate that account. User switching usually makes this convenient without logging out of your main session.

Use a virtual machine when you want to test a complete desktop environment or a larger collection of software. That keeps the experiment isolated from your real installation, and you can reset the VM or restore a snapshot afterward. A separate user account is generally lighter and faster for testing individual applications.

If you install packages through a package manager that supports it, choose the option for complete removal when uninstalling. That may remove system-level configuration files, but it will not reliably clean every per-user file under your home directory, so it works best alongside a test account or virtual machine.

Nix can help, but the package manager alone does not manage all files created in your home directory. Home Manager is needed for declarative user configuration, and it makes removing the configurations it manages much easier. Unmanaged files created by applications can still remain, though; an impermanence-style setup can help prevent unwanted user data from surviving reboots, but it requires more configuration.