We build Python automations that interact with LLM APIs, internal and external services, Outlook, Chrome, shared drives, SharePoint Online, and other tools commonly used in business workflows. One automation monitors Outlook emails, starts a DocuSign signing request, and on later runs checks the signing status. Once the document is signed, it retrieves the file and sends it as an attachment to the original email for internal review. This process needs to run every 30 minutes. Beyond Windows Task Scheduler, what tools or patterns do you use to schedule workflows with dependencies, retries, state, and external API failures? I'm also considering building a more capable alternative to Task Scheduler for these use cases—would that be worthwhile?
4 Answers
If you want something embedded directly in a long-running Python service, APScheduler is a relatively simple option. If you already need distributed workers, Celery with Redis and Celery Beat can handle scheduled jobs and queues, though it may be more infrastructure than one workflow needs. For a Windows- and Microsoft-focused setup, Azure Logic Apps, Azure Functions, or Power Automate are also worth considering because they have native Outlook, SharePoint, and other business-service connectors.
I’d start with an existing orchestration tool rather than building a scheduler yourself. Prefect, Dagster, and Airflow all support schedules, dependencies, retries, logging, failure handling, and a UI for rerunning or troubleshooting jobs. Prefect and Dagster are especially approachable if you want to stay close to normal Python. Airflow is powerful, but it generally fits Linux-based deployments better, while Prefect can also work in Windows-heavy environments.
Please be careful about creating a custom scheduler from scratch. The basic timer is easy; the difficult parts are persistent state, retries, partial failures, idempotency, API rate limits, locking, timeouts, and recovering after a crash. A lightweight cron job with a lock and structured logging may be enough for a simple workflow, but once the number of automations grows, Prefect, Dagster, or Airflow will save a lot of maintenance effort.
For external APIs, I’d prioritize resilient workflow design regardless of the scheduler. Store the DocuSign request ID and processing state in durable storage, make each step safe to retry, use timeouts and exponential backoff, and record enough detail to resume after a partial failure. A scheduled cloud function with EventBridge or a similar service can work well if you’re already in the cloud; otherwise, a small service using APScheduler or a workflow orchestrator is more practical than building a complete scheduler.
Exactly—the orchestration logic is where most of the complexity appears. A loop and a timer are easy, but state persistence, token refreshes, permission problems, timeouts, and partial failures need deliberate handling so one bad API call doesn’t stop every later run.
Thanks, that’s helpful. I’ll explore those options before deciding whether there’s any real gap worth filling with a custom tool.