I have a fairly complex data pipeline written in Polars. It completes in about 1.2 seconds on my 12-core local workstation, but takes roughly 13 seconds on a 128-core EC2 instance. Setting POLARS_MAX_THREADS=12 on the server did not initially solve the problem. Both systems use DDR5 memory, and I read the input through a tmpfs mount, so I expected their memory-based performance to be reasonably similar. The pipeline uses the streaming engine and processes many text files before collecting a LazyFrame. What could explain the much slower runtime on the larger machine?
3 Answers
Storage can still matter even when the application uses a tmpfs mount. The files may first need to be read into the page cache, temporary files may be created on an underlying filesystem, and shell commands or intermediate writes can bypass the assumptions you are making about RAM-only I/O. In this case, increasing the EBS volume's provisioned IOPS unexpectedly improved the result, suggesting that some part of the workflow was still storage-bound. Verify the actual mount locations, temporary-directory settings, and read/write system calls rather than relying only on the tmpfs configuration.
The important issue turned out to be the input layout: the pipeline was opening and processing 128 separate files and then concatenating them. That creates a lot of file-open, parsing, scheduling, and synchronization overhead, and the larger machine made that overhead more noticeable. Concatenating the files into one input before loading them into Polars substantially improved the runtime. Polars parallelizes columnar operations well, but that does not mean that hundreds of small file reads will scale with the CPU count.
A large instance is not automatically faster for one in-memory workload. NUMA layout, cache locality, memory bandwidth, CPU frequency, and thread-synchronization overhead can all hurt performance when work crosses sockets. Try inspecting the machine with numactl --hardware, then pin the process and memory to one NUMA node, for example: numactl --cpubindnode=0 --membind=0 -C 0-11 python script.py. Also compare performance with one thread and with several smaller thread counts rather than assuming more cores will help.
Profiling the lazy query and checking CPU utilization with tools such as Polars' query profiler, py-spy, or perf should show whether the bottleneck is computation, scheduling, memory access, or file handling.
It is also worth confirming that Python and Polars versions match between machines, but changing Python versions alone did not account for the slowdown here. CPU frequency differences and SIMD support can matter, though they were secondary to the many-file ingestion overhead.