Hey everyone! I'm a sysadmin who just dockerized my first open-source project, a Log Analyzer utilizing local LLMs. While I've managed to get everything up and running, I'm questioning if my method is suitable for production or just a patch.
Here's the setup: My host machine runs Ollama (serving Llama 3) on port 11434, while my container runs a Python FastAPI app that sends logs to Ollama for analysis. Currently, in my docker-compose.yml, I pass the host URL through an environment variable. For Mac and Windows, I use host.docker.internal, and for Linux, I've learned to add host.docker.internal:host-gateway.
Is this approach standard? Should I instead run Ollama within its own container using a bridge network? I aim to keep the image size minimal (around 400MB), so incorporating Ollama directly seems like a bad idea. I'd appreciate any feedback on my Dockerfile or Compose setup to ensure I'm distributing this correctly!
3 Answers
Just so you’re aware, performance on Ollama with macOS on M1 hasn’t been stellar before—might want to check for any updates. Also, offering an optional flag to start an Ollama container could strike the right balance for flexibility. The openwebui is cool, but it tends to be a hefty image.
You should run Ollama inside Docker for better encapsulation. It’s quite straightforward—just use `docker run` with Ollama. You can expose a few ports and maybe add a volume for model storage. Leveraging existing Docker containers in your compose file could simplify things significantly.
I get why you’re connecting to the host's Ollama, especially with GPU passthrough on Mac and Linux needing extra work. It might be beneficial to offer a full setup option in your docker-compose file for those who’d prefer a standalone container with Ollama included.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically