I'm curious about how long your smoke tests typically take. I've noticed that as applications grow in complexity, it's really tempting to pack more into the smoke tests. For us, it currently takes about 15 minutes with our medium-sized codebase, and we're trying to figure out how to bring that down. We use these tests to decide if we need to roll back a deployment or not, so efficiency is key. What about you guys? How long do yours take and what's the complexity like?
4 Answers
What's the scope of your tests? The complexity really affects timings. Providing more details might help others give better feedback since it sounds like your tests are running too long.
I think smoke tests ideally should really take no more than a minute. If yours are taking 15, it sounds like they might be too lengthy for what they're meant to do!
Totally get that! I might need to reevaluate what we're including in them.
My regression tests are significantly quicker and don't take nearly that long. Smoke tests should ideally be pretty instantaneous. Are you sure you're running them efficiently?
When running smoke tests, it's best to spin up a fresh environment with the latest release. This way, you can deploy containers in a temporary setup just to confirm that everything initiates correctly. It shouldn’t take longer than 5 minutes if set up well. Also, preloading components could help cut down on network delays. If it's longer than that, it might be more like integration testing, which should be done separately.
Good point! Our product deals with big data analytics which adds some startup overhead.