Should DevOps troubleshoot business-logic problems in production?

0
0
Asked By MellowPine47 On

My manager removed developers' production access because they were making weak assumptions during incidents—for example, seeing high CPU usage and immediately suggesting a service restart. The change also prevents them from viewing live production logs, so when customers report problems, DevOps has to manually collect and forward the relevant evidence.

The proposed replacement is for DevOps to learn the application's business logic, investigate incidents independently, and provide developers with both the root-cause evidence and a proposed solution.

Is this a normal responsibility for DevOps or SRE? How much should DevOps understand about an application's internal logic, and who should own root-cause analysis for business-logic bugs?

5 Answers

Answered By NorthStarVex5 On

There are two separate issues here. It can be reasonable to prohibit direct production changes and require a break-glass process for restarts, deployments, or configuration changes. It is much less reasonable to prohibit developers from seeing the information needed to debug their own code.

Give developers audited, read-only access to application logs, infrastructure metrics, traces, dashboards, and perhaps carefully limited production queries. Also make sure the telemetry covers more than CPU: error rates, latency percentiles, database load, network behavior, saturation, and downstream dependencies are all useful. High CPU is a symptom or signal, not automatically the root cause.

Answered By CopperLark82 On

The usual middle ground is read-only access. Developers should be able to search production logs, dashboards, metrics, and traces, but not change or restart anything directly. Centralize the telemetry in tools such as a log aggregation system and dashboards, and use request or correlation IDs so a customer report can be traced through the system.

Removing access because someone made a bad guess just encourages more guessing from incomplete information. Improve the observability and the troubleshooting process instead. DevOps can own the platform and make the evidence available, while developers investigate bugs in their own application logic.

QuietHarbor6 -

Read-only access does not have to mean SSH access to production. A restricted observability portal is safer, easier to audit, and works across multiple servers or workers.

Answered By SilverMaple19 On

DevOps should understand the application well enough to operate it, recognize what normal behavior looks like, and narrow down whether an incident is infrastructure-related or application-related. That does not mean DevOps should become the replacement development team or own every business-rule defect.

A good model is shared ownership: developers are responsible for the code they build and help diagnose or fix its failures, while DevOps provides deployment, monitoring, tracing, safe access controls, and operational expertise. “You build it, you run it” does not mean every person must be an expert in every part of the system; it means the teams should not throw problems over a wall.

AmberNotebook31 -

The same applies in reverse. Developers should understand enough about infrastructure and data access to avoid writing code that creates obvious operational problems, such as inefficient queries or unbounded resource usage.

Answered By BriskWillow24 On

The exact split depends on the organization. In a small team, engineers may develop and operate the same services, so a broad understanding of the application is expected. In a large or regulated environment, development and operations may be more separated, with SRE or platform teams investigating incidents and developers handling code fixes.

Even in a regulated environment, separation should mean controlled and auditable access—not forcing one team to act as a courier for logs. DevOps can produce a clear incident report and narrow the problem, but the team that owns the business behavior should normally validate and implement the fix.

CloudyTamarin8 -

A useful test is whether the team can deploy and operate the service from a clean starting point and explain its important failure modes. They do not need to know every business rule, but they should not treat the application as a black box.

Answered By GoldenRook63 On

The deeper problem is the incentive and process, not the job title. If developers immediately recommend restarting or scaling whenever something looks wrong, provide training, run blameless incident reviews, and improve the dashboards and runbooks. If DevOps is expected to understand every application deeply while developers never participate in production support, you are creating a fragile single point of failure.

A healthy process lets DevOps make the system observable and safe, lets developers investigate and fix their code, and makes both teams accountable for the service’s overall reliability.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.