I'm working on generating an array of unsigned 32-bit integers (u32) that I want to cache in SQLite as a blob. Right now, I'm using Brotli for compression, but I'm exploring other algorithms that are specifically designed for integers or may be more efficient for my needs.
Here's a sample of the kind of data I'm dealing with:
1979581047
2147354403
2143158563
1069350960
1056452628
1041771523
1041774594
1041783875
1057521728
1058570576
957973072
943429328
947566289
951760514
2109386370
2107289218
2140055426
2144252306
2127474834
2128524467
1996401905
1996400867
2004285670
1463179366
1461145143
The number of elements can range from a few thousand up to 40,000. My main priority is fast decompression and a good size ratio since I'm using this for local caching and need to read and decompress the data frequently. With Brotli, I managed to compress a sample of 14,518 u32 values from 153.7 KiB down to 54.4 KiB, but I'm open to suggestions for other algorithms that might work better for my case.
1 Answer
Have you considered using a database table instead of a blob? Using a blob might not be the best choice unless you really need all entries at once. If that is the case, okay, but you might introduce extra overhead.
Good point! I'm storing them for pattern matching between sets, so I need all of them available at once to compare. Thanks for the suggestion!