I'm curious about the practice of incorporating randomness in unit tests. For instance, if I have a test that performs N operations (let's say N is 100) and checks if the resulting state is correct after those operations, I want to confirm if the operations are commutative (meaning the order of operations doesn't change the outcome). Considering that there are a huge number of permutations possible, testing every single one isn't feasible. Is it acceptable to sample a limited number, like 50 random permutations, to see if they all pass the test? If things start to go wrong, I could log the seed used to generate the random permutation for troubleshooting. Are there better testing strategies for this kind of scenario?
5 Answers
Definitely avoid randomness in unit tests. They're supposed to be clear and targeted, testing specific functions or modules. Instead, use those random values in functional or integration tests where the complexity of the interaction between components can be properly handled.
I try to steer clear of randomness as well. If your code needs it, then it might be overcomplicated. Tests should aim for predictability. If you must include randomness, at least capture the seed to reproduce failed tests—it helps with debugging.
Fuzz testing is a different beast since it aims to expose bugs through random input values. However, using randomness in your unit tests can complicate them and make them hard to interpret. Instead, keep unit tests isolated and focus on well-defined boundaries in your modules.
A good rule of thumb is to test specific cases thoroughly, including edge cases that may lead to unexpected behavior. For stochastic processes or comprehensive input coverage, separate tests into automated and manual—automated tests should be deterministic, while manual ones can explore randomness if needed.
In general, I think using randomness for unit tests isn't a good idea. It undermines the repeatability that unit tests are meant to provide. If a test fails and random values are involved, you may never reproduce that failure, especially if the environment changes. Have you considered property testing instead? It's more structured and eliminates the randomness issue.
That makes a lot of sense! I like the idea of property testing – I'll have to look into that.
True! It's essential to track those seeds to make sure we can backtrack any larger issues.