Hey everyone! I'm trying to figure out how to transfer a massive 4TB of data from a Windows 2019 File Server at our data center A to a Windows Server 2022 File Server at data center B. We've got secure IPSec VPN tunnel between the two locations, and I've been using the Quest Secure Copy Tool for this. However, I've noticed that while transferring the large data, the speeds only reach about 15 to 22 MB/s. Interestingly, when I send over a 1GB test file, it can manage around 70-90MB/s. Is there any way to improve this transfer rate or suggestions for faster methods, scripts, or commands that I could use? Thanks!
4 Answers
Ensure that the server at data center B has SSD storage. You can use Resource Monitor to check for disk queue lengths during transfers; a high length could indicate that the disk isn't keeping up. Also, it’s good to check for packet loss or latency issues because those can really throw a wrench in your speeds, too.
It seems like the slower transfer speed could be due to the fact that the 4TB is likely broken into multiple files, whereas your 1GB test is a single file. Each small file needs to establish its own connection, which can lead to a bottleneck. If the Quest Secure Copy Tool supports multi-threading, using that could help speed things up. Alternatively, you could try using Robocopy which has multi-threading capabilities. Another idea is to compress your data into larger zip files before transferring, which can also improve speed.
If the distance isn’t too far, consider a physical handoff. Copying everything onto an SSD and physically driving it to data center B could save you a ton of time. This is especially useful if the internet connection isn’t that reliable.
Checking the latency between the two data centers could be insightful. If there's significant latency and you're not using a WAN accelerator like Riverbed, that might be what's limiting your speeds. Long-distance connections can cause issues that slow down transfers over time, especially for larger files.
That's a solid plan! I’d do that too, but in Alaska the internet can be quite unreliable and power outages happen often. It's sometimes easier to just drive it over.