Is undervolting a 5800X3D worth it if my temperatures are already okay?

Not really necessary if you're happy with the temperatures and noise. Undervolting might reduce power and heat, but there's no guarantee you'll notice a gaming performance gain. Leaving a stable system alone is a perfectly reasonable choice.

I'd consider it if you want a quieter system, especially in that warm room. Less CPU power can mean less heat for the cooler to deal with, so the fans may not need to work as hard. The actual difference depends on the workload and your fan curve.

You might also see better sustained boost in a demanding all-core workload, but don't count on extra frames in every game.

If you want to experiment, check whether your motherboard BIOS exposes Curve Optimizer for the 5800X3D. Some boards do, and PBO2 Tuner is another option. A negative curve setting isn't the same thing as a fixed voltage reduction in millivolts.

Start conservatively, make one change at a time, and compare temperatures, power and performance against stock. Test both heavy workloads and lighter use. An unstable setting can show up as an occasional crash or an error long after a quick stress test passes. If anything acts strange, reduce the offset or return to stock.

The downside is mostly the time spent tuning and checking stability for what may be a small gain. Instability can also cause calculation errors or data corruption, not just obvious crashes. If tweaking is fun, go for it. If not, stock sounds fine here.