Hey everyone! I'm working on a llama.cpp server setup inside a Docker container, and I'm running into an issue. When I try to access the server through my browser, I get a 500 Internal Server Error. The model loads correctly with GPU support, but I really need help figuring out how to fix this error. Any guidance would be appreciated! Thanks!
3 Answers
Check your server logs to see what's causing the 500 error. That can provide clues about what's going wrong.
It sounds like we need a little more info to help out. Can you share your Dockerfile? Without it, we’re just guessing.
I figured it out! The issue was with the port configuration. My Dockerfile uses the standard llama.cpp server with GPU enabled, I just needed to adjust the path to my models.
The problem likely lies with how your image is set up inside the container, not Docker itself. You might want to look into that.
Thanks for that! I found the right port and it's working now! Quick question though, once I get everything running, how can I start making requests to the server besides using the chat interface? Sorry if this is a dumb question! I'm still learning.