I have a Bash script that processes thousands of independent inputs. It may be interrupted after some outputs have been created, and simply skipping every existing output is unsafe because a file might be truncated or incomplete. Restarting everything is also wasteful and could repeat external side effects.
One design I'm considering is to write each result to a temporary file in the destination directory, validate it, rename it into place atomically, and then append the input ID and output hash to a journal. On restart, the script would trust only journal entries whose current output hash still matches. A lock would prevent overlapping runs, and traps would remove only temporary files belonging to the current process.
Are there important failure cases with this approach, particularly with parallel workers, NFS, or a crash between the rename and journal append? Is there a simpler checkpointing design that stays understandable without effectively becoming a database application?
4 Answers
The best simplification is to make each operation idempotent. Give every input a deterministic output name, and make rerunning an already completed input produce the same result without repeating an external side effect. Then a crash merely causes a retry instead of creating a correctness problem.
For non-idempotent external actions, keep an explicit durable operation ID and make the external system deduplicate that ID if possible. A shell lock can stop accidental overlapping runs on one host, but it cannot undo an operation that succeeded just before the process died.
For a small, mostly sequential batch, a manifest with states such as `pending`, `running`, and `done` may be enough. Update it only after a validated output has been atomically renamed, and on startup reset stale `running` entries. Process inputs in small batches if you want easy operator checkpoints.
Parallel workers need unique temporary names, exclusive ownership of each input, and a single well-defined completion update. Avoid relying on shell `>>` appends as a transaction. On NFS, locking, close-to-open visibility, rename behavior, and cache consistency can differ between clients, so test the exact setup; a local SQLite database or a coordinator on one host is safer than a shared hand-written journal.
If the workflow has more than a couple of states, SQLite is often simpler than inventing a journal format. Store one row per input with an ID, status, attempt number, output path, checksum, and timestamps. Workers claim pending rows in a transaction, update progress, and mark a row complete only after the validated output is installed.
On startup, treat rows stuck in a running state as interrupted and make them retryable. SQLite gives you filtering, uniqueness, and transactional updates without requiring a separate database server. It also makes parallel workers much easier to coordinate than multiple shell processes appending to the same text file.
The basic file pattern is reasonable: write a uniquely named temporary file in the same directory, validate it, and use `mv` to rename it into place. A same-filesystem rename prevents readers from seeing a partially written destination.
The journal should not be treated as the only source of truth, though. If the process crashes after the rename but before recording the journal entry, a restart can scan the final file, validate it, and reconstruct or confirm the checkpoint. If the journal append happens first and the output is missing, the entry must be rejected. In other words, make recovery reconcile the manifest with the actual outputs.
Use a per-input identity and expected format or checksum, not just file existence. Also be careful about durability: an ordinary successful `write` or rename may still be sitting in a filesystem cache when the machine loses power.
That crash window is why I’d make the output self-validating and let a restart repair the journal. The journal is useful for speed and bookkeeping, but the output plus its validation rule should determine whether work is really complete.