Last week, our team wrote a postmortem about a database connection pool exhaustion issue we encountered on Tuesday. It took us three hours to fix, but while reviewing, we discovered that the same issue had occurred back in June. The recommended action items from that earlier postmortem were to increase the pool size and improve monitoring, but unfortunately, these were never addressed because we had to prioritize shipping features to stay competitive. Now, we have repeated this incident for the third time this year, and leadership is baffled by the recurring problems. It seems like postmortem action items just get buried behind ongoing feature work, leading us back to square one. I'm frustrated and wondering why we even do postmortems if nothing ever changes. How do I make sure action items are prioritized?
4 Answers
I suggest establishing a dedicated day, like a tech debt day, where your whole team focuses solely on clearing tech debt rather than deploying new features. You might also seek buy-in from someone in a management position to advocate for prioritizing these items during planning sessions. Having higher-ups support the need for addressing past issues can significantly increase their visibility and importance.
It's common to have features and ops improvements vying for the same prioritization in backlogs. What might help is to designate a certain percentage of your team's resources specifically for infra-ops and tech debt, so they don’t get neglected. For instance, consider setting aside 10-20% of your team's time for these improvements rather than letting feature work completely dominate the schedule.
It sounds like your previous postmortems didn’t fully address the core issues. Instead of just looking at the symptoms, you need to get to the root cause of why the connection pool got exhausted in the first place. Understanding the underlying technical reasons can help prevent these issues from escalating and ensure that the proposed fixes are adequate.
Make sure to assign monetary values to each incident when discussing prioritization. This helps clearly illustrate the costs associated with downtime and the potential business losses. You could analyze previous incidents for factors like projected business losses and operational costs to restore service. Create regular reports that highlight how often these issues occur and their financial impacts, ensuring this gets reviewed by senior leadership regularly.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures