I have a Bash script that finds directories matching a condition, enters each one, and runs an update command. The script works, but there are several thousand directories and each update takes a little while, so processing them sequentially is too slow. What is a good way to run the updates concurrently while limiting the number of active jobs, or alternatively divide the directories into batches and process those batches in parallel?
4 Answers
`xargs` can do this without requiring another parallel-processing utility. Have `update-this-dir.sh` accept a directory path and change into that directory itself, then run: `find . -type d -iname '.config' -exec dirname {} ; -print0 | xargs -0 -n 1 -P 10 -t update-this-dir.sh`. Here, `-n 1` gives each invocation one directory and `-P 10` allows up to ten concurrent processes. Using null-separated paths is safer than newline-separated output when directory names contain spaces or unusual characters.
Tools such as `fd` can also execute matching commands concurrently, but the important design change is to pass the directory explicitly instead of relying on the caller's current working directory. That makes parallel execution safer and avoids different jobs interfering with one another. Also verify that the update operation is safe when several directories are processed at the same time, and choose the concurrency level based on disk, network, or service capacity rather than simply using the largest possible number.
GNU Parallel is designed for exactly this. Generate the directory list with null separators and pass it to a worker script or command with a fixed job count. For example, make the update script accept the directory as an argument, then use something like `find . -type d -iname '.config' -exec dirname {} ; -print0 | parallel -0 -j 10 update-this-dir.sh`. The `-j 10` option keeps at most ten updates running at once.
You can also implement a simple worker limit directly in Bash. Start each directory update in a subshell, keep its process ID, and wait whenever the number of active jobs reaches the limit. The basic pattern is: `(cd "$dir" && printf 'nn%sn' "$(realpath .)" && update-this-dir.sh --file ./name.txt) &`, followed by storing `$!` in a PID array and waiting for older entries before launching more. After all directories have been submitted, run `wait` for the remaining PIDs. Keep each job's output in a separate log if mixed terminal output would be confusing.
If the directory list is already stored in a file with one path per line, the equivalent is `< list_of_dirs xargs -n 1 -P 10 -t update-this-dir.sh`. The input redirection goes before `xargs`; using `>` there would write output instead of supplying input.