Python environments let different projects use incompatible versions of the same library. I understand why that works, but I'm wondering whether Python could instead install multiple versions side by side and let each package use a compatible version automatically, similar to how Flatpak handles application dependencies. That seems like it could avoid the confusion of activating the correct environment before running a project. Is this technically infeasible, or would it simply introduce different problems?
4 Answers
A single global installation would also make upgrades risky. Updating a library for one project could silently break another project that relies on older behavior. Separate environments prevent those projects from affecting each other. Tools such as lockfiles and project managers can make creating and selecting environments less manual, but they do not eliminate the need to keep incompatible dependency sets apart.
Python could theoretically provide more automatic dependency isolation, and some package managers already support installing separate dependency trees. The tradeoff is that you still need to decide which dependency tree should be used when launching a program. Project environments are a relatively simple boundary: each project gets one consistent set of packages. The confusing part is mostly the tooling and workflow, not that environments are fundamentally required by the language.
The main problem is that dependencies aren’t always isolated just by their package files. Suppose your application uses libraries A and B, and both depend on C, but on different versions. A might create an object using C’s types and pass it to B. Even if both versions of C are installed, B may reject the object because its class came from a different copy of the library. You can sometimes make this work, but it becomes fragile and difficult to debug. This is why systems such as npm and Rust allow multiple dependency versions but can still produce confusing conflicts.
Flatpak is solving a somewhat different problem. It normally bundles an application with its dependencies, while sharing carefully defined runtimes for lower-level components. The application is expected to run within that bundle, so its libraries are not generally mixed with libraries used by another application. Python packages, on the other hand, frequently exchange objects with one another inside the same running process. Two copies of a package can therefore conflict even when both installations are technically present.
That makes sense. I was mainly thinking of avoiding the frustration of forgetting which environment a project needs, but I can see how automatic side-by-side versions would move the complexity into runtime errors instead.