I've been working on an Ubuntu Server machine with strong GPU capabilities, primarily using docker with docker-compose.yml files. Currently, I'm focused on a specific container, ollama:rocm, and I'm interested in trying out Docker Model Runner. It seems like it could be the place for reliable LLMs and I'm eager to dive into it, but I'm uncertain if it's fully available or still in the works. Although it mentions GPU support, I haven't found clear information on which GPUs are compatible and what the performance comparisons are.
Furthermore, are there any implementations for Ubuntu Server that support AMD GPUs that I may have overlooked?
1 Answer
Hey! Just to clarify, Docker Model Runner is currently only available for Apple Silicon Macs using Docker Desktop. They’ve got plans to add support for Windows with NVIDIA GPUs soon, and eventually for other GPU vendors as well as Docker CE for Linux. They're rolling this all out over the next few months to gather feedback and improve, so it's definitely coming but not ready yet for your setup.
Thanks for breaking that down! It’s a bummer for those of us on Linux, though. But at least it sounds like they’re working on it.