I'm diving into Cython for the first time while optimizing my Python project, which is similar to `zipgrep` and `ugrep`. It streams through archive files and searches patterns without holding much in memory. I've done some benchmarks: it's about 15-30x faster than `zipgrep`, but 2-8x slower than `ugrep`, which I expected since `ugrep` is in C++. I've tried cythonizing using `Cython.Build` with setuptools and Nuitka, but the performance remains unchanged. Did I compile them wrongly even though they built successfully? I'm wondering: is it worth manually writing `.c` files or switching parts of my code to `cdef`, or will Python's overhead always keep it slower than `ugrep`?
2 Answers
The goal of Cython is mainly to speed up the most critical sections of your code while leaving the rest in Python. This is a strategic way to optimize performance without overcomplicating your code.
If you're not using static types in a `.pyx` file, `cythonize` might not give you much benefit. A lot of users have found better success using Numba's `@jit(nopython=true)` for performance boosts. It's definitely worth considering!
I see what you mean! I'm thinking about switching to cdef files; do you think that would actually help enhance the speed significantly?
I’m still learning about all this optimization stuff, but it's great to know! What’s the most efficient way you've seen experienced developers optimize their code?