I'm trying to run a parallel machine learning algorithm using K-Means on my PC, which has 12 threads. I got the code from GitHub, and while the sequential version runs perfectly and even outperforms the author's execution time, the parallel version takes over 10 minutes, which seems excessive. I've already enabled all the threads in the system configuration, but it hasn't made a difference. Is there anything else I need to configure? My specs are: CPU - AMD Ryzen 5 4600H with Radeon graphics, RAM - 20 GB, Architecture - x64. Here's the code I'm using: [Parallel K-Means Clustering](https://github.com/ChristineHarris/Parallel-K-Means-Clustering)
3 Answers
It's hard to pinpoint what could be causing the slow execution without more details on the code and the environment setup. The GitHub link you provided shows that the original code benchmarks on a setup with 4 CPUs, each with 20 threads. Your machine might not be optimized for the same performance. Have you checked the Task Manager while running to see if all CPUs are being utilized? It’s worth trying to simplify or break down the k-means logic to test multiprocessing separately to understand any bottlenecks.
It sounds like you might be running into the limitations of Python's multithreading. Python doesn't truly support multithreading, so what you're using is multiprocessing, which can behave differently due to how memory is managed. This means that the execution time may vary based on various factors like your OS, the Python version, and how data is handled between processes. Consider removing VSCode from your workflow and testing it directly in the terminal or using a different IDE. Sometimes, integrated development environments can add unnecessary overhead.
I actually tried using Google Colab for this and it worked flawlessly. I was just curious about my machine's performance!
Just because the code is from GitHub doesn't mean it will work seamlessly on every machine. When you mention the parallel version taking a long time, did you run the exact file from GitHub or modify it? Sometimes, those scripts include test cases that are optimized for specific setups. I'd recommend double-checking that you are indeed running the unaltered version first, using tools like Task Manager to diagnose CPU usage for better insights.
I made some small changes to fit my data, but it's frustrating that I can't see the performance on VSCode. I'll definitely take a look at Resource Monitor again.
I did tweak a few things to fit my dataset, and running it on Colab brought good results. I'm still puzzled by VSCode performance, though; I wanted to compare it directly on my machine.