How can I scale a data-polling CronJob that exceeds memory and runtime limits?

Be careful with simply adding more replicas. If every replica polls the same range, you can create duplicates, rate-limit failures, and race conditions. Use partition ownership or a work queue, store checkpoints, and make writes idempotent. Also monitor lag, chunk duration, memory usage, API rate limits, and failed partitions so you know whether the system is keeping up.

The main fix is usually to stop treating the job as one large in-memory batch. Process the data in pages or chunks, persist progress after each chunk, and make each chunk restartable and idempotent. A cursor, timestamp, or partner-provided change token is generally better than repeatedly scanning the entire data set. That keeps memory bounded and lets the work resume if a run is interrupted.

If the complete workload still cannot finish in 30 minutes, split it into multiple independent workers instead of extending the single CronJob. A scheduler or queue can dispatch partitions by customer, date range, or partner-record key, with a final aggregation step once all partitions finish. Add limits so parallel workers do not overwhelm the partner API.

First measure where the time is going. You may be able to reduce the workload substantially by using incremental polling, requesting only fields that changed, adding indexes to local queries, and avoiding repeated deserialization or duplicate processing. If the partner API only supports full exports, consider importing the export into durable storage and processing it asynchronously rather than making the CronJob do everything synchronously.