I'm looking for articles, books, or research papers that explain how to design a file format from the ground up. I'm especially interested in defining the binary layout and bit patterns, organizing stored data, keeping file sizes manageable, and applying other useful optimization strategies. I'd also like to understand when compression should be part of the format itself versus handled as a separate layer.
4 Answers
Start by defining exactly what the format needs to store and how it will be used. A file format is essentially a specification for arranging bits and bytes: headers, fields, records, metadata, indexes, and so on. Compression is often layered on top rather than being the core of the format, so first design a clear uncompressed representation.
Look for introductions to binary serialization and digital-preservation file formats. Discussions of file-format building blocks and format design case studies can help you see how headers, primitive types, lengths, checksums, indexes, and version fields fit together. Writing a small specification before implementing anything is usually a good first step.
There isn’t one universal layout to copy—the right structure depends on your application. Think through things like portability, versioning, error detection, random access, extensibility, and whether files need to be streamed or edited in place. Looking at well-documented formats and learning basic C can also make byte layouts, binary serialization, and alignment easier to understand.
A few useful examples to study are Apache Avro and Apache Parquet, especially their schemas, metadata, encoding choices, and handling of compatibility. They may already solve your problem if you need structured or column-oriented data, although the best choice depends heavily on what you’re storing and how the files will be accessed.
That’s the part I’m trying to learn: how to turn the application’s data model into a concrete bit and byte layout.