How should I license 17 Windows Server VMs across these hosts?

Start with the hosts each VM is actually allowed to run on, not the number of vCenter instances. With Standard core licensing, fully licensing a host's physical cores gives it rights to run up to two Windows Server VMs. You can stack another full set of core licenses on that host for each additional two VMs.

If all 17 VMs can run on any of the five hosts, each host needs nine full sets. For the core counts given, that's 1,152 core licenses in total, or 72 sets of 16 core licenses. If the VMs are restricted to smaller groups of hosts, calculate each host for the maximum number that could run there instead.

Also check the licensing minimums, including 16 cores per server and 8 cores per processor. The math assumes your stated core counts meet those minimums.

The 256 core calculation works for the fixed placements you described, assuming those are the maximum VM counts on each host. Host 1 needs three sets for five VMs, host 2 needs one for two, host 3 needs three for six, and the two 16-core hosts need one each.

It stops being enough if a move puts more Windows Server VMs on a host than its stacked licenses cover. Check your placement rules and failover settings before relying on that cheaper number.

Price Datacenter alongside Standard for the hosts with lots of potential VM placements. Once all required physical cores on a host are licensed for Datacenter, it provides unlimited Windows Server VM rights on that host. It can be simpler than stacking Standard, but the break-even point depends on your pricing and how many VMs each host needs to cover.

Migration and license reassignment are separate questions. A VM moving to another host doesn't automatically mean you can move the underlying Standard licenses with it whenever you like. Check the reassignment rules and any applicable agreement benefits with your licensing provider before designing around frequent moves.