I've spent the last two weeks optimizing my CI/CD pipelines, focusing on my NX monorepo. I implemented several strategies including utilizing the NX build cache system with S3, leveraging the native S3 cache for CodeBuild jobs, enabling ECR container caching, and even creating a custom build image. Initially, these optimizations reduced my build times significantly from about 7-10 minutes down to 1-2 minutes. However, I'm now facing a frustrating situation where the pipelines have reverted back, taking even longer than before—between 7-13 minutes. Most annoyingly, it seems to take at least 5 minutes to provision a CodeBuild container. I'm really trying to understand what's happening here!
3 Answers
It sounds like you're hitting some typical CodeBuild provisioning delays. Honestly, it can be pretty slow at times. Have you checked what instance types you're using for your builds? That might be part of the problem, especially if you're using older instance types that don't provision as quickly.
Have you considered the size of your cache? If it's grown too large, CodeBuild may struggle. It compresses and decompresses the entire cache as one big blob when moving it in and out of S3, which can add time. I found that caching things like npm packages directly from npm was often faster than relying on a cache like this, unless the build was inherently slow and the cache very small.
This happens before any cache is used, so I don't think cache size is the main issue.
Weird that you're experiencing such slow starts. For reference, I can do basic Docker build/push to ECR in about 30 seconds with the provisioning taking around 11 seconds. Which instance type are you using?
I'm also on BUILD_GENERAL1_MEDIUM. Seems like the issue isn't isolated to just me.
Thanks for the insight! I'm using BUILD_GENERAL1_MEDIUM instances right now. They were fine a few weeks ago, so I'm puzzled.