Yesterday I spent nearly five hours helping a developer troubleshoot a production outage. Our operations team has access to all production servers, while developers can access only development and staging environments. Because I wasn't familiar with the application, the developer told me which commands to run, and I copied the output back to them. It was slow, frustrating, and not a great use of anyone's time.
The outage wasn't revenue-impacting, but it made me wonder whether application owners should receive temporary production access during incidents. We already have an automated deployment pipeline and a strict testing process, but applications can still fail because of memory leaks, queueing problems, or other runtime issues.
How do you handle this? Do developers receive controlled access during incidents, or does operations run every command on their behalf?
5 Answers
For containerized workloads, direct access should usually be limited even further. Developers can inspect shipped logs and metrics, update configuration, or trigger a controlled redeployment. Platform engineers can handle cluster or infrastructure changes. If a production script really must be run, make it an approved, repeatable operational action rather than relying on ad hoc shell access.
I wouldn’t make unrestricted shell access the default. A better model is read-only access to logs, metrics, traces, and dashboards, combined with a documented escalation path for anything more sensitive. If deeper access is needed, use a temporary, approved, time-limited session with full auditing. Useful recovery commands should then be turned into runbooks or automated diagnostics.
The application owners should be responsible for troubleshooting their service. Operations can help with platform-level problems, but having an infrastructure person blindly run commands for five hours is a sign that ownership and access boundaries need improvement. Developers should understand the systems they deploy and be involved when they are paged.
Access is only part of the issue. Teams also need enough observability and realistic testing in staging to diagnose failures quickly. Even a correctly deployed application can fail later because of leaks, queues, capacity, or unusual runtime behavior, so application owners should know how to read its logs and identify the likely failure modes.
A break-glass process is a good compromise. Developers can request elevated production privileges for a specific incident and purpose, with approval, automatic expiration, and command logging. This can satisfy compliance requirements while avoiding a situation where operations has to act as a human terminal for the application team.
That matches what I’m considering. Temporary audited access would let the application owner investigate without permanently expanding production permissions.