For L1 and L2 engineering roles, roughly how much of a normal workday is spent testing, troubleshooting, performing root-cause analysis, and identifying and documenting issues? Do these engineers…
For an L1 or L2 support engineer, roughly how much of a typical workday is spent testing, troubleshooting, performing root-cause analysis, and identifying and documenting issues? Do these…
For a typical workday, how much time do L1 and L2 engineers usually spend testing, troubleshooting, identifying issues, performing root-cause analysis, and documenting their findings? Do they ever…
For an L1 or L2 engineer, roughly how much of a typical workday is spent testing, troubleshooting, performing root-cause analysis, and identifying and documenting issues? Do these engineers…
My manager removed developers' production access because they were making weak assumptions during incidents—for example, seeing high CPU usage and immediately suggesting a service restart. The change also…
My company removed developers' production access because they were suggesting quick fixes based on incomplete evidence—for example, seeing high CPU usage and recommending a service restart. The downside…
In a Kubernetes environment, a failing node, pod, or dependency can trigger a flood of alerts that appear to represent separate incidents. What process do you use to…
In a Kubernetes environment, a failing node, pod, dependency, or network segment can trigger many alerts that appear to represent separate incidents. What process do you use to…
Last week, our checkout service began returning 5xx errors shortly after a deployment. Monitoring worked as expected: Prometheus alerted within a minute. The difficult part was figuring out…
After a recent deployment, our checkout service began returning 5xx errors. Prometheus detected the problem within a minute, but identifying the cause took much longer. I had to…
When operating production systems, there is usually plenty of telemetry available: metrics, logs, traces, alerts, deployment records, and dependency information. The difficult part is often connecting those signals…
I support a department where a mid-level supervisor regularly emails me about IT problems using wording like "the application is down AGAIN" and copies managers and senior staff,…
When a production system suddenly becomes slow, how do experienced engineers or SREs reason through the investigation when the root cause is unknown? Suppose you have access to…
AWS says its generally available DevOps Agent can reduce mean time to resolution by up to 75%, speed investigations by 80%, and achieve 94% root-cause accuracy, supposedly enabling…
I'm building an agent that analyzes failed CI runs and tries to determine the underlying cause. Given the pipeline definition, changed files, test results, logs, and run history,…
I'm building an agent that investigates failed CI runs. Given the pipeline result, logs, test output, repository changes, and other run metadata, I want it to reason about…
I'm trying to understand what actually happens during a production incident. Suppose I'm on call at 2 AM and get paged because checkout is failing. We may already…
Our service recently started having problems at 2:15, and a deployment had completed at 2:07. Everyone immediately focused on the release, but an hour later we found that…
Our service starts showing problems at 2:15, and someone notices a deployment completed at 2:07. The release immediately becomes the main suspect, even though the real cause may…
Many production incidents eventually appear to line up with a deployment, configuration update, infrastructure change, or scaling event. Once you've identified a likely change, what do experienced teams…