job optimization

How can I redesign CronJobs that no longer finish within the time limit?

I'm responsible for several scheduled jobs that poll a partner's live data for changes. The amount of data has grown significantly: it no longer fits comfortably in memory,…