I want to learn how to serve models and understand things like KV caches, vLLM, and GPU scheduling. Is there a sensible order to learn this stuff?
Where should I start learning AI infrastructure and model serving?
Answers
Start with one model and one inference engine before adding a cluster. Run an open model locally with vLLM or TGI, send it requests, and watch GPU memory usage. Then try requests with different prompt and output lengths.
Once that works, read about KV caching and continuous batching alongside the engine docs. The concepts tend to make more sense when you can relate them to a running server. After that, containerize the setup and move it to Kubernetes.
Reply to this answer
For the Kubernetes side, make a small lab and deploy an inference engine there. Treat it as a sequence of experiments: get the GPU available to a workload, start the server, send requests, then change the workload and observe what happens.
The vLLM and Ray documentation are useful references as you go. Kubernetes community channels can also help when you hit a setup problem that the docs don't quite answer.
Reply to this answer
Don't skip the hardware sizing question. Picking a cloud VM for an open model can be confusing, especially when you're still learning GPU memory management. Use a specific model and deployment as your example rather than trying to memorize a universal VM recipe.
Reply to this answer
If setting up hardware is a blocker, a browser-based infrastructure simulator can be a useful side exercise. Breaking nodes or servers in a simulated environment gives you a way to explore failure behavior, though it won't replace running an inference engine.
Reply to this answer
Add your answer
Answers are reviewed before publication.
6e+parcelcog_571:
Would you learn GPU scheduling and the NVIDIA operator before deploying the engine on Kubernetes?
_b63!clampdial.5:
Yes, at least the basics of how Kubernetes exposes and schedules GPUs first. Then the operator documentation can help with the NVIDIA-specific setup before you deploy the server.