I'm currently working on a healthcare app where I need to migrate a massive amount of historical patient data from one system to another. The challenge is that I have 28 folders, each containing 100,000 files. When I copy each folder from System A to B, System C takes about 20-28 hours to process and ingest those files, which isn't ideal for our live environment. Medical devices are sending real-time patient data, and whenever I create a backlog by copying those files, this new data gets delayed for a day or more.
I'm looking to implement a script that copies a certain number of files, pauses for a few minutes, and then continues the process. I've found a script that does this but it's quite old and I'm wondering if there are more modern alternatives available after almost a decade of PowerShell updates. My goal isn't to speed up the copying process, as the bottleneck lies with the vendor's software which I can't alter. I simply need to send files in a way that doesn't interfere too much with live data processing.
5 Answers
You might want to stick with robocopy, which allows you to set an Inter-Packet Gap (IPG) to manage transfer delays. Start with a high IPG value and adjust down until you see performance issues. Also, monitor System C to identify when it’s less busy, then try to increase the number of files sent at those moments to minimize bottlenecks.
Consider implementing a monitoring system on the destination folder to pause file transfers if the number of files exceeds a certain limit. This lets you adjust your copying rate based on the current workload of the live system, which is a more flexible approach than just using a fixed delay.
Robocopy is definitely the right tool for this. If you're open to alternatives, you might look into doing partial restores from backups. It can help in balancing the live workload and might test your recovery processes as well.
Using robocopy with the /MT option for multi-threaded copies can speed up your transfers, but honestly, focusing on pacing is crucial. Balancing your batch sizes with the current load on System C is what's really going to help reduce the impact on live data.
If you control the environment, make sure to check if the NAS is fast enough to handle multiple data streams. Sometimes the underlying storage can be a bottleneck too. Also, for your process, caching the directory listing can speed up operations significantly.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically