I need to upload files already stored in S3 to the YouTube API. Each transfer may take a while, but should finish within Lambda's maximum duration.
Which is likely to be cheaper for a high volume of uploads? I'm also considering SQS with ECS workers that scale down to zero when the queue is empty.
Start by pricing the actual transfer time, not just the file size. With Lambda, you pay for execution duration and the memory configured while the upload runs. With ECS, the cost depends on whether you use Fargate or EC2, the resources allocated, and how much time capacity sits idle.
For intermittent jobs, Lambda is worth testing because it doesn't need a worker fleet waiting around. If uploads are frequent enough to keep containers busy, ECS may come out cheaper. Benchmark a representative batch and include network and request charges in the comparison.
SQS plus ECS is a reasonable setup if you want more control over worker concurrency. Queue each upload, have workers pull jobs, and scale task count based on backlog. Scaling to zero avoids paying for idle tasks, though you'll need to account for startup delay when new work arrives.
Set the queue visibility timeout with long transfers in mind, and make retries safe. A worker can fail after starting an upload, so you don't want a retry blindly creating duplicates.
I'd keep the user-facing request separate from the transfer either way. Put the job on a queue, return a job ID, and let the caller check its status later. You can add a notification mechanism if needed, but there's no reason the caller has to hold an HTTP connection open for the whole upload.
One distinction here: presigned S3 upload URLs solve the client-to-S3 part, not the S3-to-YouTube transfer you're asking about. Since the files are already in S3, you still need something to read and send their contents to YouTube.
Whichever worker you choose, leave room for slow transfers and retries rather than relying on every upload finishing just under Lambda's time limit.
Answers are reviewed before publication.
5m__drainkey.72:
Scaling on queue depth also needs a sensible concurrency limit so a burst doesn't overwhelm the YouTube API or create more tasks than you intended.