I'm curious about how other teams ensure their backend performs well before pushing updates. We've started running custom load tests on our backend with each code merge. If there are no existing tests, we generate them using OpenAPI and traffic logs from recent activity. Our pipeline tracks P95 latency and error rates, and it can pause rollouts for review if certain thresholds are not met. This approach has reduced our production rollout failures by 60%. How does your team handle holding back backend releases or creating traffic scenarios for new services?
3 Answers
I recommend checking out Argo rollouts with analyst templates; it's built for scenarios just like this! It could save you a lot of time and hassle when validating performance.
Wait, you actually validate performance before going to production? That's a novel approach! We only do performance tests on our critical services, and honestly, the times we faced issues were usually with the non-critical ones. For our tests, we use Gatling, and once we're live, we keep an eye on P95 with standard monitoring. We're also thinking of implementing Argos Rollouts to make the whole process even smoother.
We're using Argo rollouts with canary deployments. It’s a great way to compare response times and error rates between the canary and stable versions. If the metrics are within the acceptable range, we promote the canary; if not, we roll back. This method eliminates the need for generating load tests or setting up production-like environments, which can be really costly for larger systems.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures