What’s the Best Azure Service for Continuous Polling in Background Workers?

0
12
Asked By CloudySky123 On

I'm currently working with four background workers that are set up to continuously poll our database every ten seconds to check for new tasks. These tasks involve processing XLS file ingestions, which can take several hours. However, our infrastructure guy configured them as Container App Jobs, which seems more suited for tasks that start, run, and finish, rather than for a service that continuously polls. I'm curious if there's a better Azure service for this purpose, like Azure Functions or something else, and what risks we might face by keeping the current setup.

5 Answers

Answered By SolutionSeeker99 On

Another idea could be to set up a polling job with a scheduled trigger that sends messages to a queue. Alternatively, modify your existing code that inserts tasks into the database to send messages to a message broker directly. You could also adapt your jobs to listen for messages using an event trigger. More on scheduled jobs [here](https://learn.microsoft.com/en-us/azure/container-apps/jobs?tabs=azure-cli#scheduled-jobs)!

Answered By TechGuru42 On

You might want to check out Durable Functions! They can be triggered on a timer and are designed to handle long-running tasks. Also, consider Azure Data Factory or Azure Batch as options. There is some discussion around Azure Container Apps and long-running jobs, and while it can work, you might face config issues. Here's a [link](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=in-process%2Cnodejs-v3%2Cv1-model&pivots=csharp#async-http) for more info on Durable Functions.

Answered By QueryMaster On

I'm curious, what’s triggering the updates in your database? Understanding that could help point towards a better solution.

Answered By DevWizard88 On

If you haven't already, look into Change Event Streaming with Azure SQL. You could send insert events to an Event Hub and then process them using Azure Functions. It's definitely more complex but worth considering for a scalable design. Check out [this article](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/change-event-streaming/overview?view=sql-server-ver17) for details.

Answered By DatabaseDude On

You could have your SQL DB make an API call to a REST endpoint for processing tasks. However, I’m not a big fan of having databases make API calls. I’d suggest rethinking the architecture to make use of Azure Logic Apps, which might handle this type of workflow better. More info [here](https://learn.microsoft.com/en-us/azure/logic-apps/concepts-schedule-automated-recurring-tasks-workflows).

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.