I've been working as a DevOps engineer for a while, and honestly, the reporting part is what I dislike the most about my job. Each week, I manually sift through Grafana and CloudWatch, taking screenshots and deciding which ones are relevant. Then, I copy everything into a Confluence template to prepare the weekly infrastructure summary and any root cause analysis (RCA) documents. This process takes me around 4-5 hours for each client. I'm wondering how you all manage this—do you have tools or workflows that help streamline the process, or is everyone just doing this manually too?
3 Answers
I wrote an RCA last year when a majority of the organization was down for half a day. Honestly, I don't think anyone read it. Most people who care will just ask a few direct questions, but in general, there's minimal interest in the nitty-gritty technical details.
Spending 4-5 hours per client is pretty rough, but unfortunately, it's a common issue when the process is reliant on screenshots. Here are a few things that have helped us reduce this time:
- Ditch the default screenshotting. Instead, link to your dashboards (like Grafana and CloudWatch), and just embed the essential exceptions plus a couple of key panels for SLOs, latency, and capacity.
- Automate the process. For instance, Grafana has rendering/export endpoints (for image/PDF). Set up a job to fetch 5-10 "standard" panels and push them to Confluence via REST API.
- Agree on a simple one-page weekly template that covers uptime/SLOs, the three biggest incidents, top three risks, capacity trends, and planned changes.
- For RCAs, base the document on the incident timeline and automatically pull in metric graphs for the incident period while keeping raw graphs as links.
Even a small script to stamp the same panels onto a Confluence page can save you hours, bringing it down to about 30-60 minutes with just a quick edit for the narrative.
I honestly haven't had to deal with this kind of reporting. If we mess up badly, we might write an RCA, but it's usually a similar format for all customers and happens maybe every few months. Sounds more like a job that should fall under a manager's purview. We really need to change how we talk about manager roles—sometimes they are seen as low-paid, easily replaceable folks who mainly ensure that expense claims are correct and manage time off while also being responsible for customer reports and keeping all the graphs looking good, even if technical terms get whimsically autocorrected.
It's quite common to have reporting processes like this, especially in the managed service provider (MSP) space. It can actually be beneficial—it helps in preventing future issues!
Wow, I can do a screenshot and crop it in about 10 seconds! It blows my mind that it takes 4 hours for you. Are you sure you don’t have links for everyone to access Grafana and CloudWatch? Standardization and automation could really help you out, and having engineers input the details into incidents might ease the burden too.