Yesterday I spent nearly five hours helping a developer troubleshoot a production outage. Our operations team currently has access to all production servers, while developers can access only development and staging environments. Since I wasn't familiar with the application, I mostly ran commands dictated by the developer and copied back the output. It was slow, frustrating, and not a great use of anyone's time.
I'm wondering whether application owners should receive temporary access during production incidents so they can investigate their own systems directly. The outage wasn't revenue-impacting, but the process exposed a clear gap. What does your incident and access process look like? Do developers get emergency production access, and if so, how do you keep it controlled and auditable?
5 Answers
The deployment pipeline can remain centrally managed while the developers retain responsibility for their applications. A practical model is: developers are paged for application incidents, they investigate with normal read access, and operations are called for infrastructure issues or approved privilege escalation. Any emergency access should be scoped to a specific incident, expire automatically, and leave an audit trail.
Access alone won’t solve every outage. Teams still need strong testing, observability, documented runbooks, and clear ownership. Even thoroughly tested releases can fail because of memory leaks, queues, dependencies, or unusual runtime behavior. If the same diagnostic commands are used repeatedly, turn them into automation or a runbook instead of manually copying them between teams.
A good compromise is read-only access to production logs, metrics, traces, and relevant dashboards for the developers. If they need to make a change or run a command, use a time-limited, approved, and fully audited break-glass process. That gives them the access needed for urgent debugging without making unrestricted shell access the default.
For containerized services, direct host or container shell access should usually be the exception. Developers can often diagnose issues through centralized logs and metrics, then change configuration or code and trigger a controlled redeployment. Elevated platform permissions can remain with the operations team, especially for cluster or infrastructure changes.
The main pushback will probably be that a redeployment takes time during an urgent incident. We need a safe emergency path for cases where a script or one-off command really is necessary.
The application team should own the service in production. They understand the code, logs, dependencies, and likely failure modes better than an infrastructure team does. Operations can support platform-level problems, but making someone unfamiliar with the application act as a human terminal for hours is a sign that ownership and incident response need to be redesigned.
That temporary escalation model is what I’m leaning toward. I’ll need to research how to implement it in a way that satisfies our compliance requirements.