I have a set of behavioral tests for a CLI tool and want to run the same tests against several storage formats without duplicating them. I'm considering a parametrized fixture that an autouse fixture depends on. Is that a good fit, or are there clearer alternatives?
Is a parametrized fixture a good way to run a test suite against multiple storage formats?
Answers
That works and uses a built-in pytest feature. A fixture declared with params will run dependent tests once per parameter, so an autouse fixture depending on it can apply the matrix across tests in its scope without changing every test signature.
The tradeoff is visibility. If the autouse fixture lives in a separate conftest.py, a reader may not realize why a test runs multiple times or what setup it gets. Keep the fixture close to the relevant tests, and give the parameter values readable IDs so the test results show which format failed.
Reply to this answer
Take a look at pytest.mark.parametrize with indirect=True too. You can pass format values to a fixture indirectly, letting the fixture do the setup while the test or test class declares the cases.
That puts the matrix closer to the tests, but it also means declaring or sharing the parametrization where it's needed. It's a different tradeoff from applying it implicitly through autouse.
Reply to this answer
For a fixed set of storage implementations, fixture parameters are pretty straightforward and save a lot of repeated test declarations. I wouldn't add a custom mechanism just to build the matrix when pytest already supports this directly.
Reply to this answer
Property testing with Hypothesis is worth considering for a different part of the problem. It can generate varied inputs for the behavioral tests, while your fixture still selects the storage implementation. That's complementary to a format matrix, not necessarily a replacement for it.
Reply to this answer
If the matrix or the test case organization gets more complicated, pytest-cases is another option to explore. For a small list of formats, though, a built-in parametrized fixture may be enough without another dependency.
Reply to this answer
Add your answer
Answers are reviewed before publication.
n.560~fusekey:
How would a parametrized fixture apply to every test without being autouse?
Varel.8s:
It wouldn't apply automatically. Tests can request it explicitly, or another fixture they use can depend on it. Autouse is the option when you want implicit application within a scope.