I need to hash roughly 10,000 keywords and would prefer an extremely low chance of collisions. I wrote a small custom implementation and tested it with a handful of short strings such as "a", "aa", "aaa", "b", "bb", "bbb", and "ab". They produced different results, but I'm not sure how to evaluate whether the approach is actually safe or whether XOR, multiplication, or another technique would be more efficient. Would I be better off using an established hash function or a lookup table?
2 Answers
For only 10,000 values, use a standard, well-tested hash implementation rather than judging a custom algorithm from a few toy inputs. If collisions absolutely cannot happen, hashing alone cannot guarantee that unless the output space and input set are constrained; use a lookup table, assign unique IDs, or detect collisions and resolve them. Also consider the birthday paradox: even a large hash can collide sooner than intuition suggests. Test against a large word list or generated dataset, but testing cannot prove that a general-purpose hash has no collisions.
A few short examples producing different outputs say almost nothing about a several-hundred-line implementation. For a 10,000-item collection, a normal hash table or dictionary is likely the simplest and most efficient solution. It can compare the original keys when two hashes match, so collisions are handled correctly. Unless you have a very specific constraint, there is little benefit to replacing a mature hash function with custom XOR or multiplication code.
That makes sense. I’ll test it against a much larger dataset and look into using a standard implementation instead.