I'm a beginner trying to automate some processes at work by interacting with our company website through a web browser. Unfortunately, I can't use any extensions or plugins, so I'm coding everything myself. I have a single iframe setup that's allowing me to navigate, scrape data, and enter information, but it's only working one iframe at a time. I'm looking to run multiple iframes at once so that I can reduce the time it takes to process everything from around 2 hours to maybe just 45 minutes when using four iframes. Currently, my asynchronous functions wait for each task to finish before starting the next one, leading to a sequential process instead of parallel execution. I want to know if this is expected behavior, or am I misusing iframes? Also, could the organization's system restrictions be causing this bottleneck? I'm hoping to understand how I can achieve true parallelism with iframes. Any tips or advice would be greatly appreciated!
5 Answers
Have you considered that maybe focusing on getting a high-paying job in scraping or automation elsewhere might be a better investment of your time? Just a thought! If you're managing to do all this with an iframe, your organization may lack adequate security measures.
Remember, using `await` pauses execution until the promise is resolved, which is why you’re seeing this sequential behavior. To avoid this, gather all your async operations into an array of promises and use `Promise.all()` on that array to start them concurrently before waiting for results. This approach maximizes your efficiency.
You might want to tweak your `doParallel` function. Instead of a regular for loop that awaits each promise consecutively, utilize `Promise.all()` to handle all your requests in that function. This omission is likely what's causing your iframes to cycle through their operations sequentially. By launching all the promises together and waiting for them to resolve at once, you can trigger processing in Parallel.
Iframes are designed to display different sites, but there is a workaround. You can create multiple iframe elements at once, perhaps even making them really tiny (like 1x1 pixels) and injecting them into the DOM simultaneously. Make sure each iframe has its own context for your requests. However, efficiency might still be a concern as they behave similarly to separate tabs resource-wise.
To fetch multiple pages concurrently, you shouldn't be awaiting each operation before starting the next. Instead, you should queue them all up and use `Promise.all()` to handle them all at once. That way, you launch all your requests simultaneously instead of waiting for each one to complete. Just keep in mind that concurrency is different from parallelism, especially in single-threaded environments like JavaScript. You might still hit limitations, especially related to your organization's server and its I/O processes. If you want true parallelism, consider spawning multiple tabs instead.

Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically