I'm currently facing a challenge with automating tests for a SaaS product that we deploy around 15 times a day. It's really tough to maintain a solid testing strategy under this model. Right now, our setup mainly employs a basic smoke test suite in our CI pipeline, which only manages to catch the most obvious issues. Our more comprehensive tests unfortunately take too long and run overnight, meaning any potential problems don't show up until the next day.
I'm dreaming of a comprehensive automated testing strategy that can run quickly enough to allow us to gate each deployment. However, with our end-to-end tests taking around 45 minutes even when we run them in parallel, the feedback loop gets disrupted. So, I'm wondering how to strike a balance between quality testing coverage and deployment speed without bogging down our release pipeline.
3 Answers
Have you thought about why you need to deploy that frequently? Some teams see it as essential to have every PR trigger a deployment. It can be beneficial for different developers who deploy when they're ready. At my previous job, small changes could be self-validated and pushed once the pipeline was clear, but that also meant taking ownership of the ticket until it went live.
One common approach is to shift the majority of your more extensive tests to run post-deployment against a canary environment. This way, you can catch most issues early, typically within a few minutes, and still have the option to roll back if necessary.
This is so relatable! While it sounds amazing to ship 15 times a day, the testing part can quickly become overwhelming. Many teams find that they have to choose between speed and confidence in their releases. One strategy might be to keep E2E tests focused and rely on more detailed tests for components. But, honestly, it’s never a straightforward fix.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically