How can a Bash batch resume safely after interruption?

If the workflow naturally has stages, GNU make can provide a simple checkpoint model: outputs are targets, inputs are prerequisites, and successful files are rebuilt only when prerequisites change. Each command should still write to a temporary file and rename it after validation. This is a good fit for deterministic file transformations, but it does not by itself solve non-idempotent external actions, distributed locking, or reliable state tracking on NFS.

For anything beyond a small one-off script, SQLite is a reasonable middle ground. Keep one row per input with fields such as status, progress, output path, hash, and timestamps. Workers claim pending rows transactionally, update progress, and mark a row complete only after the output has been validated. A stale running row can be reclaimed after a timeout. This handles retries and queries much more clearly than trying to make a text journal provide transactions.

A plain journal can work if you define recovery rules explicitly. Treat the output rename as the commit point, and make the journal an audit record rather than the sole source of truth. During startup, scan journal entries and outputs, validate hashes, and reconcile any valid output that has no journal entry. Write the journal append with locking, flush it if durability matters, and assume either side of a crash may have happened. Do not use a temporary file from another filesystem, since rename is only atomic within one filesystem.

Parallel workers need separate temporary names and a single coordination mechanism for claiming work. A lock around the entire batch is simple but prevents useful parallelism; a database or per-item claim files are better if workers must run concurrently. On NFS, advisory locks, close-to-open caching, rename visibility, and durability guarantees can vary by configuration, so test the exact environment. If correctness is important, local storage or a real job/database system is safer than relying on shell-only coordination.

The key concept is idempotency: make rerunning one item produce the same result and avoid duplicating external side effects whenever possible. Then a crash is much less dangerous. Use a temporary file in the same directory, write and validate it, and rename it into place. On restart, validate the final output rather than trusting that its mere existence means success. A crash after the rename but before the journal append leaves an unrecorded output, but that can be treated as a completed item if validation succeeds, or regenerated safely if the operation is idempotent.