For a periodic maintenance job, Persistent=true can trigger a missed run shortly after boot, sometimes close to the next scheduled invocation. The service might also be started manually, so relying on the timer schedule alone does not guarantee that only one copy is active. I'm comparing letting systemd serialize a service unit, using flock on a dedicated file descriptor, and creating an atomic lock directory containing the owner PID and process start time. A plain PID file seems unsafe because PIDs can be reused and stale files may remain after a crash. For a second invocation, is it better to exit successfully, fail clearly, or wait with a timeout? If using a filesystem lock, how can stale locks be detected and removed without introducing a cleanup race?
2 Answers
If you need a lock that works without flock, create a directory or file using an atomic operation that fails when it already exists. Store enough owner information to validate it, such as the PID and process start time. If acquisition fails, treat the lock as active unless you can reliably prove the owner is gone. For cleanup, avoid blindly deleting a possibly reused lock: rename the suspected stale lock to a temporary name, verify it is still the same stale lock, and only then remove it. Otherwise, exiting or retrying with a timeout is safer than aggressive stale-lock cleanup.
For most jobs, flock is the least complicated and most reliable choice. Put the real command behind a small wrapper that opens a dedicated lock file and holds the descriptor for the entire run. A second invocation can either exit cleanly with a log message or wait for a bounded time, depending on whether overlapping work should be skipped or queued. Since the lock is held by the process, it is released automatically if the process exits or crashes.
This also keeps the locking independent of how the job was started, so manual runs and timer-triggered runs follow the same rule.
A PID by itself is not sufficient because the operating system can reuse it. Including the process start time, or using an OS-provided advisory lock when possible, avoids mistaking a new process for the original owner.