Why Are Some Tasks in ProcessPoolExecutor Marked as ‘Running’ While All Workers Are Busy?

I’m intrigued as well but need more details. How did you determine it was *running* before execution? You might have just caught the transition when it was handed off to the worker for execution. If your tasks are long-running and still getting marked as running before they actually start, that would be unusual!

I can’t give a definite answer, but looking at the source code might help! I’ve found diving into the implementation of ThreadPoolExecutor beneficial when I faced similar issues. The code isn't overly complex, and understanding the library can provide insights that just asking won't. It’s a major perk of using open-source libraries!

That does sound strange! Let’s say you have 6 tasks and 3 workers. If the first 3 tasks are running and all workers are occupied, when a 4th task is initiated, there’s a chance it could be marked as running prematurely. Can you share a snippet of your code that shows how you set up the workers?

This behavior might actually be a performance optimization. Since inter-process communication can be slow, the executor may try to keep one task in progress and another queued per worker. When a task is delivered to a worker, it's marked as running even if it hasn't been executed yet. This way, when the worker finishes its current task, it can immediately start the next one without waiting for a status update. It makes sense that you’d see it marked as running because the parent process can't confirm cancellation after a worker has pulled a task from the queue.