I'm looking for suggestions on the best Azure service to run a daily, long-running batch job using Python. The job involves extracting data from Microsoft Graph for around 35 users, including things like Teams chats, emails, and meeting events. The script has to manage pagination and handle Graph API throttling with retry logic. It produces several CSV outputs and uploads these files to Azure Blob Storage. Ideally, this task would be scheduled to run once a day and typically takes over 10 minutes.
I would prefer not to use Azure Data Factory, and Azure Functions don't quite meet my needs due to runtime limitations. I'm aiming for a cost-effective and low-ops solution, and I'm open to container-based approaches. Currently, I'm considering either Azure Container Instances triggered by Logic Apps or Azure Container Apps Jobs with built-in scheduling.
I'd also love some advice on best practices related to scheduling, retries, avoiding duplicate runs, monitoring, alerting, and authentication methods, like whether to use Managed Identity or client secrets for Graph and Blob access. What would you recommend?
4 Answers
I think Azure Container Apps would be a solid choice for your needs. They support event-driven architectures, which can help you avoid constantly hitting the Graph API unnecessarily. It also has built-in scheduling, which fits well with your daily job.
Consider using Azure Automation. It's convenient and can handle your job's requirements, though it does have a 3-hour runtime limit. Just ensure your job fits within that timeframe!
Before we dive deeper, could you clarify what runtime limits you’re encountering with Azure Functions? Plus, running one long job might not be necessary. Have you thought about breaking it down into smaller tasks to improve efficiency? You could take initial IDs and gather info as small tasks, maybe even managing your progress in a storage account or through a queue.
If your goal is to extract Teams messages, just make sure that’s the core of the job. Users often underestimate what the data extraction entails, so keep that in mind as you refine your process!
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically