I want to write a program that compares the filenames and subfolder names in two directories containing roughly 100,000 files and folders—about 650 GB of data stored on my phone. I would like to run the comparison directly on the phone and repeat it fairly often, since transferring everything to a computer takes too long.
I have programmed professionally since 1979, but I am new to Android, PCs, and Python. I have tried several Android apps, but they either lack the needed functionality or run into storage-permission restrictions.
Is this technically possible without rooting the phone? If so, are there existing scripts or recommended approaches for comparing the directory trees efficiently?
4 Answers
For copying files afterward, avoid transferring thousands of individual files through slow device protocols if possible. Put the selected files into one staging directory and create an archive, such as a tar file, before copying it. That reduces the overhead of handling each file separately, although you need enough free space and should verify the archive before deleting anything.
Yes, it should be possible, but Android's storage permissions are the main limitation. A terminal environment such as Termux can run Python and command-line tools. You may need to grant it access to shared storage first, and it still might not be allowed to inspect certain app-private directories. For ordinary user-accessible folders, a Python script using pathlib can recursively collect and compare relative paths without reading the file contents.
If you can open a shell with access to both directories, the standard recursive comparison command is `diff -qr dir1 dir2`. Be careful with paths that begin with a hyphen: the command may interpret part of the path as an option. Use `--` before the paths, or prefix them with `./`, for example: `diff -qr -- './-Thomas/-B/hist/hist0/- -/-Unsorted' './-Thomas/-B/hist div all'`. Quoting protects spaces, but it does not by itself prevent option parsing.
A Python approach would be to compare sets of relative paths rather than opening every file. In outline: use `Path(folder).rglob('*')`, keep the relative path for each item, and compare the two sets. This is generally practical even for a very large tree because you are comparing names and directory structure, not transferring or hashing 650 GB of content. If you also need to detect files with identical names but different contents, that is a separate and much slower hashing step.
The error mentioning an invalid option came from the leading hyphens in the directory name, not from the `o` in the command. `--` tells the program that everything following it is a filename rather than another option.