I'm working on setting up an enterprise-level Azure DevOps with self-hosted agents using VM Scale Sets (VMSS). One issue we're facing is the long scale-out time, which can take between 10 to 20 minutes to provision and start a new VM, making it tough to get jobs processing quickly when agents are busy. Our aim is to minimize this wait time as much as possible.
Here's some context for what we're dealing with:
- We have agents hosted and registered through Azure DevOps agent pools.
- VMSS is used to give us flexibility.
- This setup is for a CI/CD proof-of-concept that might go into production.
- While reliability and cost are important, the focus is really on speeding things up.
I'm looking for ideas or best practices to reduce this scale-out delay. I'm considering keeping some agents warm/idle, using pre-configured VM images with the agents installed, or exploring different scaling strategies. How do others manage to quickly pick up jobs with self-hosted Azure DevOps agents at scale? Any real-world tips or lessons learned would be greatly appreciated!
5 Answers
We switched to using Azure Container Instances for our pipelines. The instance gets created, which takes about a minute to set up. Then, our jobs are handed off to the self-hosted agent. After the job is done, we just blow away the ACI, which keeps things pretty efficient!
The issue might lie in how you're setting things up. We utilize VMSS with a custom Ubuntu image, and our wait times are about 3-5 minutes, with Windows agents taking around 5-7 minutes tops. We build our golden images monthly with a custom Packer script. Just curious, what does the 10-20 minute bootstrap involve for you? Are you installing software after the VM starts?
VMSS can be effective, but it sounds like you've explored other options too. We recently switched to managed DevOps pools, which are a managed service that does a lot of the heavy lifting for you. With some tuning, we managed to get the average startup time for new agents to about 2 minutes with a Windows image and even quicker with Linux. You might want to check that out!
Have you checked out Managed DevOps Pools? They're pretty awesome—startup time can be around 5 minutes, and you can have standby agents during business hours only. Plus, you only pay when the VM is running, and the agent installs automatically. Here's a resource for more info: [Learn about Managed DevOps Pools](https://learn.microsoft.com/en-us/azure/devops/managed-devops-pools/?view=azure-devops).
Building a golden image that includes all your necessary tools and dependencies is a solid approach. If scale-out time is a major concern, keeping some warm or standby instances could really help. Just keep in mind, for added capacity, it might take some extra time to spin up new instances. We find that around 10 minutes for a start is acceptable given our focus on cost optimization.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures