Hey everyone! I'm currently using x64 Ubuntu machines through GitHub-hosted runners for our workflows and I'm on the lookout for alternatives that might offer better cost efficiency and performance. I've come across a few options and would love to hear about anyone's experience with them. The platforms I'm considering are: Blacksmith, Ubicloud, BuildJet, WarpBuild, runs-on, and Namespace. Specifically, I'd like to know about the startup time or cold start latency, job execution performance, pricing, how complicated the integration with GitHub Actions is, and if there are any unexpected limitations that I should be aware of. If you've switched from GitHub-hosted runners or have any benchmarking insights, please share!
6 Answers
Have you considered using local runners? They could be a solid option if self-hosting is on the table!
I’ve done a comparison between runs-on, terraform-aws-github-runner, AWS CodeBuild, and persistent EC2 runners. I found that runs-on was super easy to set up and performed well, while CodeBuild was a bit of a headache and had its limitations. You should definitely explore the options you mentioned; it’s the first I’m hearing of some of them!
I suggest checking out EC2 spot instances for your runners. They're much cheaper and can be spun up temporarily during your workflows. Depending on your needs, I've found them to be a solid alternative to GitHub runners. If you're working in Azure, they have a similar spot SKU option that also turned out to be cost-effective for us.
Good point! Just be aware that spot instances aren't available all the time, which could complicate pipeline triggers.
With WarpBuild, we manage runners in your AWS/GCP/Azure accounts, including spot instances, so we'd handle that for you!
We currently use Actions Runner Controller (ARC) on EKS, and it's been pretty great. The startup time is around 5 minutes until the pod is up and running, so we always keep a few runners available to lower this latency. Performance-wise, we've seen them run 1.5 to 2 times faster than GitHub-hosted ones. The best part? It's free! Just keep in mind the admin overhead of setting up and managing your own runner image and cloud infrastructure costs. Overall, the complexity is fair; the GitHub docs are decent, although it would help to have some K8s experience. As for limitations, I can't think of any major ones.
I totally agree with you! About the startup time, are you using a custom image? I find my runners with the stock image start running actions in about a minute! Also, I second the recommendation for ARC; it's super reliable.
We're actually using ARC on Kubernetes too, and it works wonders for us!
We're using AWS CodeBuild action runners and the startup time is usually just 1-2 minutes. It integrates well with our AWS setup for internal tests and I believe it supports dynamic runner adjustments through additional configs. However, I have ran into a limitation with a certain Python action because of unsupported architecture. Definitely check AWS's pricing model for estimates—it's essentially serverless!
Self-hosting your runner could be the way to go. It’s not too hard, and it allows you to use some robust hardware at a lower cost compared to cloud hosting. We bought a Mac Mini specifically for this purpose, and it’s been great!
We’ve set up our own runners too, and it’s been cost-effective for our needs!
Exactly! Plus, self-hosting gives you better control over performance as well.
Yep, self-hosted is definitely a great choice!