I'm working on a web app that includes a public endpoint for users to submit requests. I've got an ingestion service that solely focuses on sending these requests to a database, which helps ensure reliability and minimizes downtime since it only deals with basic data operations. However, I'm now trying to figure out the best way to process these requests after ingestion.
I know there are various approaches I could take. For instance, I could utilize a relational database to periodically check for new entries and process them. Alternatively, I could implement a change data capture (CDC) mechanism with Kafka or a similar queue system to have workers actively listening for new requests. If I decide to use something like DynamoDB, I'm curious about the best methods for processing new entries effectively.
There are many potential solutions out there, and with expected traffic around 50 requests per second peaking at that rate but averaging around 2-3 per second, I'm looking for a straightforward and reliable approach that aligns with these requirements.
3 Answers
Why the separate ingestion processor? If you're only looking at few seconds for processing, doing it synchronously might be easier. But I get the idea—it’s like securing your potential income; keeping requests in the system ensures no loss, especially if there's a bottleneck downstream.
I hear you, but there are security considerations with keeping requests like this. Just be mindful of the risks involved when you're separating these components. It's important to ensure that ingestion is handled securely.
I wouldn't get too complicated with this. Just store the requests in a basic relational database. Then, set up a separate service to handle the processing. After the web server saves a new request to the database, it should notify the processing service that there's something to handle. For added reliability, let that processing service regularly check for new entries, just in case the notification doesn't go through.
If your needs grow significantly down the line, you could think about adding Kafka to the mix. But honestly, based on your current traffic numbers, keeping it simple right now is the way to go.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically