I have a Bash script that finds directories matching a condition, enters each directory, and runs update-this-dir.sh --file ./name.txt. There may be several thousand directories, and running the update sequentially takes a long time. What is a good way to process them concurrently while limiting the number of active jobs—for example, keeping 10 running at once or dividing the directories into batches? The solution should handle paths safely, avoid competing changes to the parent shell's working directory, and ideally preserve useful output and exit statuses.
3 Answers
If you want to stay in Bash, launch each directory operation in a subshell so every job has its own working directory, and keep a PID queue. A basic pattern is: max_jobs=10; pids=(); while IFS= read -r -d '' dir; do (( ${#pids[@]} >= max_jobs )) && { wait "${pids[0]}"; pids=("${pids[@]:1}"); }; ( cd "$dir" && printf 'nn%sn' "$PWD" && update-this-dir.sh --file ./name.txt ) & pids+=("$!"); done < <(find . -type d -iname '.config' -exec dirname {} ; -print0); wait "${pids[@]}". For larger jobs, a tool such as GNU parallel or xargs is generally simpler and gives better control over output and failures.
You can do this with xargs as well. Make update-this-dir.sh change into the supplied directory itself, then use: find . -type d -iname .config -exec dirname {} ; -print0 | xargs -0 -n1 -P10 -t update-this-dir.sh --dirpath. The -P10 keeps up to ten processes active; when one finishes, xargs starts another. Using null-delimited paths avoids problems with spaces and other special characters in directory names.
GNU parallel is designed for exactly this. Have the update script accept the directory as an argument instead of relying on the caller's current directory, then run a fixed number of jobs at once. For example: find . -type d -iname .config -exec dirname {} ; -print0 | parallel -0 -j 10 --halt soon,fail=1 update-this-dir.sh --dirpath {}. The -j 10 setting limits concurrency, and GNU parallel also provides options for grouping or labeling output.
Be careful with the input redirection: use < list_of_dir when list_of_dir is a file, or pipe a command into xargs. Also use -print0 and -0 if paths may contain whitespace or newlines.