I'm curious if there are any reliable neural compression algorithms out there. I'm not too concerned about whether it's lossless or lossy at this point, although lossless would be preferred. What I'm really interested in is an algorithm that can analyze a given file, identify the patterns within it, and then when you want to access the file again, it can regenerate the content. I've seen something similar with images and diffusion models, but I'm looking for something with minimal randomness in the output. If you have any papers, code, or even basic videos to share, I'd really appreciate it!
3 Answers
A cool trick for compression is to look at the binary data and strip out all the zeros since they don’t carry any information. You can reduce the size significantly this way, sometimes cutting it in half! Then you can count the number of 1s and compress that even further. It’s a neat approach—though decompression relies on you to interpret the data correctly afterward!
You should check out Pied Piper's lossless 'middle out' compression platform—it's the best I’ve used so far! Really effective for pattern-based compression.
I recognized that Silicon Valley reference right away. Props for bringing it up!
If you're looking for something more advanced, there are some newer approaches in machine learning that focus on pattern extraction. I can look up some relevant papers if you'd like!
That’s a neat reference! Did you ever think about how they made that concept come to life?