How I Achieved Fast Cold Starts with Llama 3.2 on AWS Lambda

I've managed to run Llama 3.2 on AWS Lambda with cold starts consistently under 500ms. Initially, I was hesitant to invest in a dedicated SageMaker endpoint or a reserved EC2 instance for an internal chatbot that only gets about 40-60 hits daily, which felt like wasting money. At first, Lambda's cold start times were unbearable—loading a ~4GB model took around 40 seconds, making it useless for chat. However, after experimenting over the weekend, I found a workable solution. Two key strategies made a significant difference: First, I learned that allocating more RAM directly affects CPU performance in Lambda. My initial setup with 4GB of RAM resulted in slow inference speeds, but once I increased the RAM to unlock 6 vCPUs, I saw speeds jump to 18-22 tokens per second. Second, I found a way to bypass disk access during the cold start phase. Although SnapStart can help, it currently has limitations in Python container images and disk space. By using `memfd_create` to stream model bytes from S3 directly into RAM, I was able to trick `llama.cpp` into thinking it was reading from a file without touching the disk, effectively killing cold starts. I've written more about the full architecture and code snippets separately. If anyone's interested, I can share the link in the comments. Also, has anyone tried using Durable Functions instead of Step Functions for orchestration? I'm tempted to simplify state management but concerned about potential costs.