I'm writing a Bash script that processes a list of tasks in parallel, with a fixed limit such as four jobs at once. I want to implement this without GNU parallel while still handling real-world failures correctly. The script should track every child's exit status, aggregate failures, respond to SIGINT and SIGTERM by stopping active work, and avoid leaving orphaned or zombie processes. I've experimented with background jobs, wait -n, job control, and traps, but I'm unsure how to structure the worker tracking and cleanup so that no exit codes are lost. What pattern would you recommend for controlled parallel execution in Bash?
5 Answers
Pure Bash can do this, but the bookkeeping is the difficult part. Keep an array of PIDs, start at most N jobs, and reap them with wait. A trap should set an interruption flag, stop starting new work, and terminate the process groups or tracked children. Then wait for every remaining PID so they are reaped, recording each status before returning an aggregate failure code. Be careful not to use a single wait result as the overall status, because it only represents one child.
For a pure-Bash design with many tasks, a fixed set of worker loops can be easier than repeatedly spawning and polling jobs. Feed tasks through a queue, have each worker claim one task at a time, and write its result and exit status to a separate temporary file. The parent remains in the foreground to catch signals, stops assigning work after cancellation, terminates workers, waits for them, and reads all result files. Use cleanup traps for temporary files and make the trap idempotent so it cannot recursively trigger itself.
A common mistake is assuming job control solves process management. Job-control commands are mainly for interactive shells; a noninteractive script should explicitly track PIDs and use wait. Also avoid storing tasks or statuses in unquoted strings, since spaces and unusual characters can corrupt the queue. If the requirements expand to output capture, retries, process trees, and portable signal behavior, Python or another orchestration tool will generally be more maintainable than a large Bash implementation.
A basic structure is: launch jobs in the background, store each PID, and whenever the number of active PIDs reaches the limit, wait for one of them and remove it from the array. In Bash versions that support it, wait -n can reduce the polling, but you still need a reliable way to associate the completed status with the corresponding PID. On interruption, set a cancellation flag, send TERM to all active jobs, optionally follow with KILL after a short grace period, and call wait on every PID during cleanup.
Also consider whether each task starts its own subprocesses. Killing only the immediate child may leave descendants running, so launching each task in its own process group and signaling the group is safer when the workload creates subprocess trees.
If external tools are allowed, xargs with its parallel-process option is usually much simpler and is commonly available even where GNU parallel is not. Another practical option is to generate a temporary makefile and invoke make with -j N; make handles scheduling and propagating failures. These approaches avoid reimplementing queues, worker tracking, and signal handling in shell.
That makes sense. I’m specifically avoiding external parallel tools to learn the mechanics and to support minimal environments, but using xargs or make would be a sensible production choice when those dependencies are acceptable.