Anyone Tried Distributed Compilation with Kubernetes?
Asked by
CuriousCoder172
I'm working with three bare metal nodes, each having 8 cores, and sharing storage through NFS. My personal setup is a bit weaker; I have a modern Intel laptop with 12 cores (10 e-Cores and 2 p-Cores). I've been looking into distributed compilation tools like distcc, ccache, sccache, and icecream. I'm curious if anyone here has successfully set up distributed compilation in Kubernetes. My goal is to compile for both Windows x86_64 and Linux aarch64 using cross-toolchains. Before diving in deeper, could you share your experiences? I think sccache workers could fit as DaemonSets, and the scheduler as a Deployment, but I'm unsure about getting the toolchains in place and what other hurdles I might face. Any insights would be appreciated!
Answers
Distributing the compilation jobs might not be worth it since you'll have to transfer many small files over the network. Consider creating a pod specification you can launch whenever needed and keep your ccache on an NFS volume for better performance.
Reply to this answer
My advice is to remember that Kubernetes is mainly for managing your compute resources, not for dictating how to compile your code. If you think about how you'd handle this without Kubernetes, that can guide your approach here.
Reply to this answer
You can incorporate the toolchain directly into your Docker image or use a sidecar container to share the toolchain binaries with the cache tool. I wrote a blog post on sharing large files between containers that might help: [Mounting Large Files](https://anemos.sh/blog/mounting-large-files/). For sccache, you'll want the workers accessible from your client, which could be done via a `NodePort` service with `externalTrafficPolicy: local`. Just sharing some thoughts as I haven't done this exactly, so take it with a grain of salt!
Reply to this answer
If you're using compilation caching, is it still slow when compiling locally? Just curious, as this might point to some inefficiencies in your setup that could be alleviated.
Reply to this answer
Add your answer
Answers are reviewed before publication.
CuriousCoder172: Thanks for the pointers! This sounds pretty reasonable. I had initially considered running the Icecream daemon as a Deployment with workers on NFS share, but the sidecar approach is a new angle for me. I’ll check out your blog post during my commute! :)