Would you use Lambda's GPU instances for inference?

November 22, 2026 5 answers
Asked by Veldnaro.3q

AWS has added G-series GPU instances to Lambda Managed Instances for inference and other workloads. Where would this fit compared with Bedrock, SageMaker endpoints, or GPUs on ECS or EKS?

Answers

Rovqueno-5z

Event-driven batches seem like the obvious thing to test. Image or document jobs arriving through S3 or SQS might be a poor fit for a GPU endpoint that sits idle most of the day.

The appeal is less endpoint management and the ability to handle spikes. I wouldn't assume it's cheaper without checking the actual managed instance pricing and comparing it with the endpoint's idle time.

5v!copperratchet_629:

Measure the full job time too, including getting the model ready, not just the GPU inference time.

Reply to this answer
m.835+brackenbadge

For steady traffic, a GPU service on ECS or EKS could still make more sense. High utilization changes the math, especially if reserved capacity or Savings Plans apply. I'd compare the total cost at expected utilization rather than assuming serverless wins.

Reply to this answer
Tavdari-5w

Cold starts are the question for me. A multi-GB model has to be available and loaded when a fresh environment starts, and that delay could wipe out the benefit for latency-sensitive requests.

Test with the real model artifact and traffic pattern. A small model processing an occasional batch is a different proposition from a large model serving interactive requests.

Reply to this answer
Trelzavi-6p

Bedrock is worth considering if it supports the model and task you need. But not every inference workload is an LLM. A custom TensorFlow model, for example, is a different case from calling a supported model through Bedrock.

Reply to this answer
_x68!mivdaro.5

Don't skip the CPU comparison. Small traditional ML models or embeddings might run cheaply on CPU Lambda, particularly when a batch can be split into lots of parallel jobs. A GPU only helps if the workload gets enough benefit to justify it.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions