I've been trying to pull images from Amazon ECR to my local machine using a 500mbps fiber connection. While pushing images to ECR saturates my upload speed at nearly 500mbps, and pulling from Docker Hub also maxes out my download speed, pulling from ECR is disappointingly slow, only reaching about 50-100mbps. Is there a reason for this huge difference? Do pulls from ECR involve extra steps like decompression or something similar?
3 Answers
Don't forget that ECR uses S3 under the hood for storage. When you run a Docker pull, ECR actually redirects your download request to S3 buckets to retrieve the image layers. So if you're feeling slow speeds, it might just be the S3 data retrieval that's causing the holdup.
This issue is pretty common. Basically, AWS ECR isn't CDN-fronted like Docker Hub or the public ECR via CloudFront, which means you're pulling directly from a single AWS region. If your connection to that region isn't great, you'll see those low speeds of around 50-100 Mbps.
To improve your pulling speed, try these tips:
- Increase the parallel download limit by setting `"max-concurrent-downloads": 10` in your `~/.docker/daemon.json` and then restart Docker.
- Consider pulling from a nearby region by replicating the repo, or use ECR Public if it's an option.
- Try to avoid having too many small layers in your images.
- You might want to check your ISP's routing to AWS for possible throttling by running a traceroute to the layer URL.
Typically, while pushes are fast, ECR pulls can be slower due to the lack of an edge cache.
I get that, but it's still strange that uploads are so much faster. Shouldn't they also be impacted by the lack of CDN?
Absolutely, I've seen the same thing. For example, it took me about 29 seconds to pull a 700 mb image from the same region, and I even have VPC endpoints enabled for Docker. It's just part of the experience with ECR sometimes.
Yeah, routing might cause delays even before reaching the AWS backbone.