I'm working on a Python program running in Ubuntu through WSL, and creating millions of nested empty folders currently seems like the most practical way I can think of to reduce redundancy in a large set of coordinate data. My in-memory list becomes too large, so I was considering using folder paths to encode the information leading to each datum. Could this approach make the system unstable or cause other serious problems?
4 Answers
This is probably an XY problem: the folder structure is an attempted solution, but the underlying data-storage problem may need a different approach. A database such as SQLite would likely be much more efficient and would let you query the data without keeping everything in a Python list or representing records as directories.
Directories still consume disk space through filesystem metadata such as inodes, even when they are empty. Creating millions of them can fill the disk, make directory operations and indexing painfully slow, and potentially leave the system in a difficult state if storage runs out.
Large collections of files and folders can make searching take a very long time, even if browsing the top-level directories seems fine.
For large coordinate sets, consider a compact representation, streaming the data instead of storing the whole list in memory, or using SQLite. A serialized structure such as a database table or partially loaded key-value store can avoid the redundancy without creating an enormous filesystem hierarchy.
A common filesystem pattern for content-addressed storage is to hash each item and use a few hash characters as shallow subdirectories, rather than making one extremely deep path. That only helps when files are actually the right storage mechanism, though.
Deep nesting also runs into pathname-length limits. Both Windows and Linux-related tools have limits on how long a complete path can be, so you would stop being able to create deeper folders long before reaching millions of levels. The exact limit depends on the filesystem and APIs involved.
Once you describe the actual data and queries you need, it should be easier to choose an appropriate structure instead of using the filesystem as a database.