I'm currently working with four background workers that are set up to continuously poll our database every ten seconds to check for new tasks. These tasks involve processing XLS file ingestions, which can take several hours. However, our infrastructure guy configured them as Container App Jobs, which seems more suited for tasks that start, run, and finish, rather than for a service that continuously polls. I'm curious if there's a better Azure service for this purpose, like Azure Functions or something else, and what risks we might face by keeping the current setup.
5 Answers
Another idea could be to set up a polling job with a scheduled trigger that sends messages to a queue. Alternatively, modify your existing code that inserts tasks into the database to send messages to a message broker directly. You could also adapt your jobs to listen for messages using an event trigger. More on scheduled jobs [here](https://learn.microsoft.com/en-us/azure/container-apps/jobs?tabs=azure-cli#scheduled-jobs)!
You might want to check out Durable Functions! They can be triggered on a timer and are designed to handle long-running tasks. Also, consider Azure Data Factory or Azure Batch as options. There is some discussion around Azure Container Apps and long-running jobs, and while it can work, you might face config issues. Here's a [link](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=in-process%2Cnodejs-v3%2Cv1-model&pivots=csharp#async-http) for more info on Durable Functions.
I'm curious, what’s triggering the updates in your database? Understanding that could help point towards a better solution.
If you haven't already, look into Change Event Streaming with Azure SQL. You could send insert events to an Event Hub and then process them using Azure Functions. It's definitely more complex but worth considering for a scalable design. Check out [this article](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/change-event-streaming/overview?view=sql-server-ver17) for details.
You could have your SQL DB make an API call to a REST endpoint for processing tasks. However, I’m not a big fan of having databases make API calls. I’d suggest rethinking the architecture to make use of Azure Logic Apps, which might handle this type of workflow better. More info [here](https://learn.microsoft.com/en-us/azure/logic-apps/concepts-schedule-automated-recurring-tasks-workflows).

Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically