Hey everyone!
We're using x64 Ubuntu machines through GitHub-hosted runners for our workflows and thinking about possible alternatives to improve our cost and performance. Have any of you tried runner platforms like Blacksmith, Ubicloud, BuildJet, WarpBuild, runs-on, or Namespace? I'm particularly interested in factors like startup time, job performance, pricing, integration with GitHub Actions, and any unexpected limitations. I'm looking for insights from anyone who has made the switch or benchmarked these platforms against GitHub-hosted runners. Your experiences would really help us decide whether to migrate or stick with our current setup.
Thanks a lot!
9 Answers
We’ve been hosting GitLab runners on EKS; it’s very cost-effective and only requires licensing fees. Also, I’m a fan of Buildkite for CI/CD and self-hosting on AWS.
We’ve been using AWS CodeBuild as our action runners and it's pretty good. The startup time is usually around 1-2 minutes, which fits our need. It also supports dynamic runner OS/resources with just a few config tweaks. Just be aware of the AWS pricing model—it’s serverless, so you pay only for what you use.
Have you ruled out local runners?
You mean self-hosted?
We use Actions Runner Controller on EKS and it’s been great. The startup time is about 5 minutes for the pod to be ready, so we keep a minimum number of runners always active to minimize that latency. In terms of performance, it’s 1.5 to 2 times faster than GitHub-hosted runners. And best of all, it's free! However, setting up a Kubernetes cluster and managing your own runner image can take some administrative resources. The GitHub documentation is decent, though having some Kubernetes experience is essential. As for limitations, I haven’t encountered any.
Ditto.
Are you using a custom image? My runners with the stock image start running actions in about a minute from nothing. I also second ARC on Kubernetes; it's easy to set up and very reliable.
Self-hosting your runner could be a low-cost solution. It's not difficult and allows you to leverage serious hardware for much less compared to hosting in the cloud.
Yeah, and it offers significant savings! We recently got a Mac mini for our on-prem hosting.
Absolutely! Self-hosted runners can save a lot on recurring expenses.
Instead of the third-party platforms you mentioned, have you thought about using EC2 spot instances for your runners? They can be spun up temporarily during the workflow and tend to offer a cheaper and more performant option compared to GitHub's runners, depending on your requirements.
Founder of WarpBuild here - we can help manage runners in your AWS/GCP/Azure accounts, including spot instances.
If you're using Azure, you can do the same with Azure spot SKUs. Even running 24/7, it turned out cheaper for us than longer GitHub Actions jobs. ARC runners on K8s with spot node pools were even cheaper!
Thanks for the insight! Spot instances are not on-demand; how do you handle their availability for pipeline triggers?
As the founder of WarpBuild, I can attest that we offer the fastest single core performance for our runners. Our startup time and cold starts are rapid due to warm pools. Integration is straightforward, usually just a couple of clicks to set up. We prioritize security, which is why we opted out of a migration wizard that required code read/modify permissions. We have functionality for orchestrating VMs in your cloud for greater scale and customization, along with powerful docker builders and automated caching. Feel free to check us out at https://www.warpbuild.com/compare/github for benchmarks!
We've had great luck with runs-on; it adds about 20 seconds to our times but is very reliable. It practically works out of the box. We only had to tweak the labels so they were unique enough to avoid job overlaps, and the developer is responsive, which is a plus.
I did a proof of concept comparing runs-on, terraform-aws-github-runner, AWS CodeBuild, and persistent EC2. Runs-on was super easy to set up while CodeBuild was disappointing. The terraform module has a learning curve and the persistent setup requires regular maintenance.
That fits the CodeBuild pricing model. I didn't have the exact numbers, but it's very accessible. Just watch for quirks; I've noticed some third-party actions have compatibility issues due to AWS's architecture.