How can I redesign CronJobs that no longer finish within the time limit?
Asked by
MistyCedar42
I'm responsible for several scheduled jobs that poll a partner's live data for changes. The amount of data has grown significantly: it no longer fits comfortably in memory, even after scaling vertically, and the jobs now take longer than the required 30-minute window. Increasing the time limit isn't an option because other dependent processes need them to finish on schedule. The system was originally built without much technical oversight, has already crashed, and is now considered critical. What's the best way to redesign the processing so it can handle the growing volume while keeping the data current and completing on time?
Answers
This probably needs an architectural change rather than simply adding more CPU or memory. Process the data in batches or chunks instead of loading everything at once, and consider splitting the work across workers with a map/reduce-style approach. If the partner supports it, track a cursor, timestamp, version, or change token so each run only handles new or modified records instead of repeatedly scanning the entire dataset. You’ll also need to make the processing idempotent so retries don’t create duplicates or corrupt state.
Reply to this answer
First determine whether the job is doing a full refresh or actually polling for changes. If the data is continuously changing, ask whether the partner offers an incremental API, change feed, webhooks, or another way to receive updates. A full live-data scan every 30 minutes may simply be impossible at the current scale, so the requirements and upstream integration may need to change.
Reply to this answer
Since this is a critical system that already fails under load, measure it before rewriting everything: identify the slowest stages, memory-heavy operations, database queries, and the amount of data processed per run. Then set up checkpoints, bounded queues, monitoring, alerting, and recovery for partial failures. It’s also important to communicate that the original design no longer matches the volume and that meeting the deadline may require changes to the data flow, not just a larger machine.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions