I'm developing Marcel, a shell written in Python, and I'm facing challenges with concurrency options. Initially, I implemented Jobs using threads, but it lacked proper termination handling, especially with commands being interrupted by ctrl-C. Then I turned to the multiprocessing module with the fork option, which works well on Linux. However, I hit a wall with MacOS due to its incompatibility with fork. The alternative, spawn, introduces significant complexity due to the need for pickling state, which slows things down. I've also considered forkserver but found it poorly documented. I'm looking for fresh insights or alternative approaches that could work effectively for a portable shell. Any suggestions?
5 Answers
Check out Xonsh, another Python shell. They have their methods to handle concurrency that you might find useful.
Why not explore coroutines? Task objects in asyncio are designed to be easily cancellable, which might suit your needs perfectly.
Using threads might still be an option if you manage to check a shutdown flag regularly. Nevertheless, signal delivery to threads can be tricky. You might want to consider using asyncio, which simplifies concurrency and includes cancellation functions. Just remember, blocking operations will need to be run in background threads! But, honestly, if you’re building a shell like Marcel, you might not need concurrency at all; Python’s subprocess module can handle spawning processes well enough for a typical shell, much like traditional Posix shells.
I suggest going with asyncio and implementing stop events. It allows for clean terminations with ctrl-C if you handle everything correctly, including task cleanup.
Just to echo the async conversation, I'd really recommend looking into trio or anyio over asyncio. They have much cleaner APIs for threading and cancellation options.
I typically don’t agree with sweeping statements, but I did read something alarming in the asyncio docs. They suggest that tasks can be garbage collected during execution because the loop doesn’t maintain strong references. That raises a lot of questions about compatibility!