How can I speed up multiple API requests for my project?

You might want to check if you really need to make those requests in a serial manner. If they can run in parallel, that could save you a lot of time. Also, what tools are you currently using for making these requests? A map-reduce framework could be helpful in that regard.

Are your external requests dependent on each other? If not, running them concurrently should definitely help cut down on the processing time. Also, if it makes sense for your data, think about fetching the external data from different sources on a schedule (like hourly), caching that, and then making runtime requests to your centralized cache. Just be careful with caching data from APIs that don't allow it for commercial use! And make sure your front end updates as data comes in to keep the user engaged.

To improve the flow with multiple APIs and cut down on response time, consider parallelizing your requests instead of sending them one at a time. You could also look into streaming processing to return partial results quickly, and implement caching to avoid making repeated calls. Breaking down the process into stages using queues and workers can really help speed things up. Combining these techniques should give your project a nice boost in both speed and overall user experience.

Consider taking in a normal request first, then using queues or events to handle the process. This approach will let you scale things up easily. In the end, you could have a subscriber endpoint and push updates to users via websockets, creating a smoother experience.

If you need to manage state throughout the process, a workflow engine like Temporal could be a good choice. If your needs are simpler, just using a regular job queue might suffice! It really depends on your project requirements.