I'm curious if there are any high-level libraries available that can help manage persisted subinterpreters within a single process. My goal is to load a complex series of classes into a subinterpreter that remains active, allowing me to send commands to it from the main interpreter. Ideally, I want something where the subinterpreter isn't constantly being destroyed and recreated.
3 Answers
Are you sure you actually need a subinterpreter? You might find multi-threading or multi-processing code could achieve the same results for you.
What do you mean by "persisted" subinterpreter? Generally speaking, subinterpreters don't have strong support since they're not much more advantageous than using subprocesses. For instance, libraries built with pyo3, like Pydantic, usually refuse to work in a subinterpreter.
By persisted, I mean I want to reuse the subinterpreter instance without it being destroyed and re-initialized. I thought subinterpreters were lighter compared to subprocesses? My workload is quite light per thread, but it will be quite frequent.
You should check out `anyio`, which has built-in support for subinterpreters. Its documentation has some useful insights on how to get started!
I'm open to other suggestions! I initially chose subinterpreters to avoid IPC/pickling overhead with multiprocessing, but I’d look into persisted subprocesses if they’re supported better. Switching to a free-threading version would be ideal, but some libraries I use won't support that just yet. Thanks for your thoughts!