I'm setting up a periodic maintenance job where Persistent=true might trigger an overdue run shortly after boot, potentially close to the next scheduled run. The service can also be started manually, so relying on the timer schedule alone doesn't guarantee that only one instance is active. I'm comparing letting systemd serialize a single service unit, using flock on a dedicated file descriptor, and creating an atomic lock directory containing the owner PID and process start time. A plain PID file seems risky because PIDs can be reused and stale files may remain after a crash. For a second invocation, is it better to exit successfully, report an error, or wait with a timeout? If using a filesystem lock, how can stale locks be detected and cleaned up without introducing a race?
3 Answers
For this case, flock is usually the simplest and safest choice. Put the actual job behind a small wrapper that opens a dedicated lock file and holds the descriptor for the entire run. The kernel releases the lock automatically if the process exits or crashes, so you don’t have to decide whether a PID in a leftover file is still trustworthy. A second invocation can exit cleanly with a log message when overlapping work is harmless; use a wait timeout only if the missed run should get a chance to proceed shortly afterward.
If you need a lock that works independently of systemd, use an atomic operation such as creating a directory. Creation succeeds for exactly one contender, while others see that it already exists. Store enough metadata to identify the owner, such as a PID together with the process start time, but treat that metadata only as a diagnostic and stale-lock check—not as the lock itself. If the owner is genuinely gone, clean up carefully and retry. Moving the suspected stale directory to a temporary name and rechecking it before deletion helps reduce races. Still, this is more complicated than flock and is easier to get subtly wrong.
Systemd can serialize starts of the same service unit, but a lock in the job wrapper is more defensive because it also covers manual launches and other callers. Whichever approach you use, make the non-owner behavior explicit: cleanly skip when duplicate work is acceptable, fail loudly when it indicates an operational problem, or wait only with a bounded timeout so a stuck job cannot block everything indefinitely.
That’s the least painful approach in practice. It also works when the job is started manually, as long as every entry point uses the same wrapper and lock.