I'm working on a backend service that handles user-uploaded PDFs, and I need to extract each page from these PDFs as separate PNG files stored in Google Cloud Storage. For instance, a 7-page PDF should result in 7 individual PNGs. However, this process is really resource-intensive; I'm currently using pypdfium, which is the lightest option I've come across. Even for a basic 7-page PDF, it's using around 1GB of RAM, and larger files often cause the job to fail and trigger auto-scaling. I initially tried a virtual machine with 8GB RAM and 4 vCPUs, but I had to upgrade to a 16GB RAM instance due to failures.
How do others manage PDF page extraction in production environments without encountering out-of-memory errors? Here's a snippet of the code I'm utilizing for the extraction.
6 Answers
You might want to double-check if there are any resource leaks in your code. Although you're closing the `page` and `pdf`, make sure to close the `bitmap` and `pil_image` objects as well. Also, I noticed that your cleanup code isn't in a `finally` block—exceptions could prevent those resources from properly closing. Using a `with` statement might simplify your code and ensure everything gets cleaned up. Also, how large is the PDF file you're working with? You might be re-parsing the PDF each time for each page, but it could be more efficient to reuse the same PdfDocument object for multiple pages. Have you isolated your function to verify if it's indeed causing the high memory usage?
I think reusing the PdfDocument could really help, thanks for that tip!
I remember using a Java library for splitting large PDFs that worked really well. It might be worth exploring that option if memory issues persist.
I've managed to process PDFs with over 20 pages using mupdf, and it only required about 1GB of memory. Might be worth checking out if you're dealing with larger files—let me know if you want me to share some sample code!
You could try modifying your file handling by first writing the PDF bytes to a temporary file. For example, change the line where you create the PdfDocument to use a temp file instead. That should significantly cut down on RAM usage when you process the PDF, making it more efficient overall.
Hey, can the Gemini framework handle PDF page extraction too?
Another option is to run `gc.collect()` after processing each page to clean up any large objects left over from previous iterations. It’s not the cleanest method, but it requires minimal effort and might help reduce memory allocation.
That's great advice! I'll check the cleanup process to make sure there are no leaks.