How Can I Compare Two Large MD5 Hash Files for Duplicate Emails?

From my experience, SQL Server should be capable of handling tables with 14 million records. You can set the MD5 hash as the index and perform a join on the equal indexes to find duplicates. It shouldn't take too long to run, especially if your query is optimized.

One way to tackle this is to concatenate the hash files into a single file, then clean up the hashes to ensure they're in a consistent format. After that, sort the lines and compare each one to the next; duplicates will show up as repeated lines. While I'm not sure Excel can handle 14 million rows, this is something you could also script. Just ensure your comparison doesn't miss any minor differences in format.

Considering your restrictions, I’d suggest using PowerShell if you have experience with it. If your organization has a legal or eDiscovery team, they might already have tools in place for comparing hashes, which could simplify things for you since they do this regularly.

You might want to break the files into smaller chunks and use Excel. Honestly, I can't think of any file comparison tools that can handle such large files, especially with the software restrictions you have. If Excel is unable to manage the file size, you could explore loading the data into a SQL database and running SQL queries to find duplicates. That's usually a solid option if you have access to a database system.