I'm working on a physics simulation that currently runs locally in Python. Each iteration takes roughly four minutes, and the full workload may involve running the program about 1,000,000 times on an HPC cluster. The code relies heavily on SciPy, NumPy, Pandas, npmath, and diskcache/memoization, with many repeated calculations across iterations. Since several of these libraries already use optimized C, C++, or Fortran internally, I'm unsure whether rewriting the Python code in C or C++ would provide a meaningful speedup. Would it be better to focus on benchmarking, parallelization, batching jobs, caching, or I/O instead?
2 Answers
SciPy and NumPy already perform most numerical operations through highly optimized C and Fortran code. If the Python layer is mainly coordinating inputs, calling library functions, and handling outputs, rewriting that glue code in C or C++ may provide little benefit. First profile the program to find the real bottlenecks, then focus on distributing the workload across the cluster, using efficient batches, and minimizing unnecessary I/O and data movement.
At this scale, the total runtime is the biggest concern. Four minutes per run multiplied by a million runs is an enormous amount of compute, even with parallel jobs. A rewrite might save substantial time if profiling shows that Python-level loops or object handling dominate, but you should also investigate whether the runs can be batched, parallelized, or reorganized to avoid repeating work. The cluster allocation and scheduling limits may matter as much as the language choice.
The plan is to use the cluster and submit batches of roughly 100 to 1,000 jobs at a time. We expect to have access to the available compute for about two months, so I’m trying to estimate whether that will be enough.
That’s what I suspected. I’ll benchmark the individual stages first and see where the time is actually going before deciding whether a rewrite is justified.