Our Python backend runs across Kubernetes nodes, with Postgres and Redis already in use. We have scheduled cleanup and reporting jobs, and want to manage them in the backend rather than with Kubernetes cron jobs.
Would Postgres advisory locks be a good fit? We may eventually support user-defined jobs, so the design needs room to grow.
Postgres advisory locks are a reasonable starting point for a small set of scheduled jobs. Give each job its own stable lock key, try to acquire it before running, and skip or reschedule if another worker holds it. Separate keys let unrelated jobs run at the same time.
Watch the connection lifetime. A session-level advisory lock stays with the database session, so make sure the job holds the right connection and releases the lock even on errors. If the connection is lost, Postgres releases the lock, but the old worker might still be running. A lock alone does not guarantee exactly-once effects.
For user-defined jobs, a jobs table could be a better fit than hardcoding a lock around each task. Workers can claim available rows with SELECT FOR UPDATE SKIP LOCKED inside a transaction, then record who claimed the job and when the claim expires.
You'll need a recovery policy for workers that die. If a claim expires and another worker retries, the original worker might still be alive, so a lease by itself doesn't prevent overlapping execution. Use a claim version or fencing token where the downstream writes can enforce it, and make retries safe where possible.
Also decide how scheduled occurrences are created. A unique constraint on the job and scheduled time can prevent multiple schedulers from creating the same occurrence.
A task queue is worth considering if the user-defined jobs become a real requirement. Separate scheduling from execution, put work on the queue, and let workers consume it. Celery is one Python option that supports scheduled tasks as well as workers.
A queue doesn't automatically give you exactly-once execution. Workers can fail after doing the work but before acknowledging it, so check the retry and acknowledgement behavior and make job effects safe to repeat.
What's the reason for moving away from Kubernetes cron jobs? Keeping scheduling there may be the least complicated option for the fixed jobs. A dedicated scheduler pod is another possibility, though you still need to think about restarts and overlapping runs rather than assuming one replica makes duplicates impossible.
Redis can coordinate a lock too, but be careful with expiration. A paused worker can keep running after its lock expires and another worker acquires it. Releasing a lock should also check ownership so an old worker can't delete a newer worker's lock.
Since the durable data is already in Postgres, using it for coordination may be simpler than introducing a separate locking authority. Redis is still an option if you can handle stale workers and retries explicitly.
Answers are reviewed before publication.
unreasonably-good-at-folding:
For work that can outlive a lost connection, consider how to stop a stale worker from writing. Idempotent operations or fencing enforced by the resource being updated can help.