I keep running into the same problem whenever I need to freeze a data format for an application, engine, or other project. I start worrying about every possibility I might need later, especially when the format is binary or otherwise rigid, and end up unable to make a decision. How do you draw the line and confidently say, "This is the format we'll use"? Are there practical questions, metrics, or design rules that help determine when a format is good enough?
5 Answers
Before committing to binary, compare alternatives. JSON or another structured format compressed with a standard tool can sometimes be close enough in size while remaining much easier to extend. If you truly need binary, consider an established extensible format such as Protocol Buffers, or even SQLite if your data fits a database model.
Design around today’s real requirements rather than trying to predict every future one. For rigid formats, version them immediately and plan migrations. A migration path is usually much easier to maintain than a format overloaded with speculative features.
Add a format version from the beginning. When the structure changes, increment the version and keep loaders for older versions when practical. New required fields are the main complication, but you can handle those case by case with defaults, migration logic, or an explicit “unknown because this came from an older version” value.
Try not to implement properties just because you might need them someday. You can often make a binary format extensible by storing fields with identifiers and lengths, allowing newer readers to skip fields they do not understand. That gives you room to evolve without designing every possible feature upfront.
For something like geometry, would you use that approach for optional data such as per-vertex colors, or separate the core vertex data from optional layers like colors?
A fixed header can reserve space for a format revision and metadata, with some unused bytes left for future expansion. The main structure should still support variable-length strings or sections where needed. The important part is deciding based on actual requirements: avoid speculative fields, but deliberately leave a migration or extension mechanism.
So for the current version, the idea is mostly to stop overthinking and define whatever the project actually needs now, while keeping the version number available for future changes?