How can I run a command in each directory in parallel with Bash?

November 16, 2026 4 answers
Asked by t.753+hexpocket

My script finds directories containing a .config directory, changes into each parent directory, and runs an update command. There are a few thousand of them, so sequential execution is slow. How can I run several updates at once without starting them all at the same time?

Answers

_m47+ledgerpiston.8

xargs is a good fit for a fixed concurrency limit. If you can change update-this-dir.sh to accept a directory path and do its own cd, feed it NUL-delimited paths and use xargs -0 -n 1 -P 10 update-this-dir.sh. That runs up to ten instances at once, each with one directory argument.

Keep the paths NUL-delimited all the way from find. Converting newlines to NULs is not safe for directory names that contain newlines. Also check the options on your system, since xargs implementations can differ.

_v68+mapcrayon.2:

Make sure the update script exits if cd fails. Otherwise it could run the update in the wrong directory.

Reply to this answer
_b95~relaynotch.4

GNU Parallel can manage the worker limit for you without splitting the directory list into batches. Feed it NUL-delimited directory paths, set the job count to something like 10, and have each job change into its assigned directory before running the update.

That's an extra dependency, but it saves you from writing the job scheduling yourself. Pick a modest job count first and see whether the storage or the update command becomes the bottleneck.

Reply to this answer
7a+fusepiston_361

You can do this in Bash too. Run each update in a background subshell, with cd and the command inside that subshell. The parent shell then stays in the starting directory instead of hopping around as jobs run.

Keep an array of background PIDs using $! after each launch. When the array reaches your limit, wait for one PID and remove it before starting another job. After the loop, wait for every remaining PID. Record nonzero wait statuses if you need the script to report failed updates.

Waiting for the oldest PID is simple, although a slow oldest job can leave a slot idle while other jobs have already finished. A wait-for-any approach can keep the slots busier, depending on the Bash version available.

_a62+claywicket.5:

Parallel output can get mixed together too. Separate log files per job can make failures easier to trace.

Reply to this answer
p.416+buttonshuttle

Whatever launches the jobs, check whether the same parent directory can appear more than once in the input. Running two updates against the same files at once could cause trouble. Deduplicate the paths or use a proper atomic locking mechanism if duplicate work is possible.

A check for a lock file followed by creating it isn't atomic, so two workers can both pass the check. A named pipe alone doesn't provide a pool of parallel workers either.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions