I'm containerizing a FastAPI-based machine-learning API and was surprised by how quickly the image grew after adding dependencies such as PyTorch and ONNX. Multi-stage builds, slim or distroless runtime images, and ordering layers so dependency installation stays cached reduced the image size by roughly 70% and made rebuilds much faster. The remaining challenge is that the ML libraries themselves can be enormous, especially when CUDA binaries and native extensions are included. What approaches are working well in production? Do you prefer python-slim, distroless, or another base image, and how do you reduce ML container size without making deployment and debugging unnecessarily difficult?
4 Answers
The biggest size problem usually isn’t Python itself—it’s the ML stack. PyTorch, CUDA components, and other native libraries can add hundreds of megabytes before the application is even copied in. For production, it helps to keep build tools and pip caches out of the runtime stage, install only what the service needs, and consider ONNX Runtime when you don’t need the full PyTorch stack for inference.
A multi-stage build with Debian-based python-slim is a practical middle ground. Build wheels and native dependencies in one stage, then copy only the application and required runtime packages into the final image. I’d also use uv for faster, more controlled dependency installation, keep development packages out of the production requirements, and inspect the finished image layers to find unexpected files. Alpine can look smaller, but musl often causes trouble with Python ML packages because prebuilt glibc wheels may not work and dependencies might compile from source.
The musl issue is easy to underestimate. Once large native packages stop using prebuilt wheels, build times and image complexity can increase enough that Debian slim is actually the easier choice.
Distroless images can reduce the attack surface and remove a lot of unnecessary userland, but they’re not always pleasant for debugging Python services with C extensions. If you need shell access during incidents, python-slim plus a dedicated non-root user is often a better operational compromise. For smaller non-ML services, compiling with tools such as Nuitka can help, but it won’t eliminate the size of heavyweight ML runtimes.
Don’t overlook ordinary image bloat. Use --no-cache-dir when installing with pip, maintain a strict .dockerignore so .git directories, local virtual environments, and caches aren’t copied into the build context, and avoid full Debian images unless you need them at runtime. Dependency creep also matters, so audit the dependency tree and remove convenience libraries that aren’t actually required. A non-root user is a good baseline even when you stay with slim rather than going fully distroless.
That matches what I’ve seen. Once the core ML dependencies are installed, removing build artifacts and caches becomes important, and ONNX deployment can avoid shipping a much larger training-oriented stack.