I've recently undertaken a project to significantly reduce memory usage across my Python web applications. We have 23 containers running on a single server with 16GB of RAM, and the memory consumption was approaching 65%. By focusing on just two of our apps, I managed to cut down memory usage from approximately 2GB to just 472MB. Here's a quick overview of what I did to achieve these savings:
1. **Switched to a single async Granian worker**: I rewrote the app using Quart (an asynchronous Flask framework) and replaced the multi-worker setup with one fully async worker, which saved 542MB.
2. **Raw and DC database pattern**: I replaced MongoEngine with raw queries and slotted dataclasses, saving 100MB per worker and nearly doubling the requests per second.
3. **Subprocess isolation for the search indexer**: The indexing daemon was using 708MB due to import chains, so I isolated it in a subprocess. Now it only uses the necessary imports for about 30 seconds during re-indexing and reduced memory from 708MB to 22MB.
4. **Local imports for heavy libraries**: Instead of importing libraries like boto3 and pandas at the module level, I moved those imports into the functions that actually use them, which was a significant help to cut down on memory overhead.
5. **Switched to diskcache for small-to-medium caches**: I moved some of my in-memory caches to diskcache, which, although it provides modest savings, contributes to my overall memory reduction.
In total, I freed up 3.2GB across all our applications! A full write-up with detailed before and after comparisons is available on my website. I'd love to hear any feedback or tips from others who have tackled similar issues.
3 Answers
Your approach to subprocesses for indexing is a smart move! It reduces peak memory usage significantly. Just be aware that while you're saving memory, the overhead from starting processes can affect your response time. Have you benchmarked performance before and after this change?
I like your points about lazy imports! They can certainly help in managing memory if you're only calling those functions rarely. Just be cautious with that strategy—if a rare function ends up being called during a peak load, you could still run into high memory requirements. Have you kept track of how it’s performing during those peak times?
Absolutely, that’s a valid concern! Since the worker processes recycle every 6 hours and those functions are infrequently used, the overall memory stays lower most of the time. I tried to account for those peaks in my workflow.
It's impressive how you've managed to reduce the memory footprint! I must admit, though, I'm curious why 63% RAM usage was a problem for you—since it's there to be used. 10.5GB seemed like it could be fine, but I get that optimizing where you can is important! Moving stuff from RAM to disk can slow things down, so I'm interested to know if you've observed any performance hits from that decision?
Good question! I realize that moving to disk might seem counterintuitive, but using diskcache is actually faster than swapping to disk when memory fills up. It was about balancing performance and resource efficiency.
Yes, it has added some overhead when starting the subprocess, but the reduction in memory usage was worth the trade-off for us!