I'm updating a PowerShell script that scans an extremely large file server and processes directories in parallel with ForEach-Object -Parallel. A complete run can take many hours, so printing every processed item would create too much console output and slow things down. In the older single-threaded version, I stored a script-scoped timestamp and printed the current directory once every five minutes, then scheduled the next progress update. The parallel runspaces cannot safely read or modify that script-scoped variable. What thread-safe approach should I use to share progress between workers and display one status line approximately every five minutes?
3 Answers
The System.Collections.Concurrent types are a good fit here. Each worker can put the directory it just processed into a ConcurrentQueue, and a separate monitoring loop can periodically drain the queue or simply read the latest available item and write a status line. That keeps console output out of the worker hot path and avoids unsafe access to shared state. You can also bound how much history you retain if you only need the latest few entries.
If the work done per directory is substantial, Write-Progress may be enough to show the most recently completed item. Just avoid calling it for every item if the workers finish very quickly, because frequent console updates can become a bottleneck. A timed reporting loop or throttled updates would be safer for a long-running scan.
I only need one console update about every five minutes. Parallel workers make it easy to accidentally print for every item, so I’m looking at throttling the reporting rather than writing directly from each worker.
PowerShell’s ForEach-Object documentation includes an example of sharing a thread-safe variable reference between parallel runspaces. That pattern may be simpler than coordinating mutexes, monitors, or semaphores yourself. For this use case, a synchronized state object containing the latest directory and a timestamp can work well, provided updates and reads are protected appropriately.
I’ll test that approach. I’m still getting familiar with the different synchronization objects, so using the documented pattern should be less error-prone than building the locking logic from scratch.
That sounds promising. I may keep the queue limited to roughly 5–10 entries since I only need a small amount of recent progress information.