I'm trying to figure out the best way to utilize the CPU resources of some old 7th Gen Intel computers we've got. These computers are not being used anymore, but we have a backlog of data analysis work that could really benefit from using their processing power. Ideally, I want to SSH into one of these computers, perform some data wrangling, and then run a complex algorithm that distributes the CPU load across the other three computers over a private LAN. Is there any Linux software or configuration that can help me achieve this? I'm not sure what the exact terms are to look up, but I'm mainly using R and Python for this analysis. Even with 1GB network cards, it would be useful if I could run the processes over a weekend and check back on Monday.
1 Answer
You’re looking for concepts like distributed computing or grid computing. The idea is that you need software that either handles the task distribution automatically or that you can wrap your applications in software that takes care of it for you. A good starting point is the Beowulf cluster, which is all about getting multiple computers to work together effectively. Just be aware that setting this up can be quite challenging! Check out this article for a deep dive into it: https://en.wikipedia.org/wiki/Beowulf_cluster
I think both R and Python have libraries that can help with distributed computing, so you should be okay with those! It’s just about finding the right setup.
Got it! So if I go this route, do you think it’ll work well with R and Python, or should I look for specific tools?