Dealing with Multi-threading Issues in Pandas DataFrames

How did you find out that the threads were hanging? It might be worth double-checking if it was a pandas issue or if it could be due to improper use of the lock. Sometimes, what you’re facing could be a deadlock situation, which typically points to issues in your own code rather than pandas itself.

Pandas isn't designed to be thread-safe, so that could explain the behavior you're seeing. However, saying you'll never use pandas again based on this experience seems a bit harsh—you might want to give it another shot once you figure out the threading issues.

Without logs, it’s tough to say for sure if pandas is the issue. It might be worth trying out Polars or another data handling solution as well. Just avoid having multiple threads writing to the same file at once; that can cause major headaches!

Are you sure you're using threads correctly with the GIL (Global Interpreter Lock) on? You might want to look into using subprocesses instead if you're trying to process multiple DataFrames concurrently. I stick with Polars for this kind of work.

Have you considered switching to Polars? It’s a newer library that’s designed better for these types of tasks. Personally, I’ve moved away from pandas completely and haven't hit any major limitations yet—it's pretty easy to convert between the two if you ever need to.