Wrong disks, neighboring switch ports, and the wrong deployment branch can all cause trouble surprisingly fast. What habits or safeguards help catch those mistakes before they become outages?
What checks help prevent infrastructure mistakes?
Asked by
~perhaps!in_transit
Answers
For physical work, point at the exact component and say what you are about to do before touching it. It sounds a little silly, but it forces a pause between deciding and pulling.
Pair that with an independent identity check. A clear label helps, but it can be wrong. Confirm the server and drive through management tools, and use an identification light where available. For a cable, verify the switch port and its destination rather than relying on the neighboring link lights.
Reply to this answer
Parallel patches are a trap when the branches look alike. Before deploying, check the branch and the actual changes included in the artifact, then make sure that exact version has been reviewed. A deployment pipeline that promotes the reviewed artifact is safer than relying on someone to pick the right patch by hand.
Reply to this answer
Don't treat diagnostics as automatically harmless. A debugger or a long trace can interrupt a live service, even when the machine was assumed to be a test box. Check the host's role and current activity first, and understand whether the diagnostic will pause anything before running it.
p28!tramline_631:
Traffic and disk activity can provide clues, but low activity alone doesn't prove a server is safe to interrupt.
Reply to this answer
If someone asks for remote hands, get the exact server and component confirmed before pulling anything. A vague instruction is a reason to stop and ask, not a reason to guess. Especially when the storage setup and its tolerance for a drive removal aren't clear.
Reply to this answer
Make recovery part of the plan too. Know what depends on the server or link, what the expected impact is, and how to restore service if the wrong thing gets touched. A drive swap can turn into a much longer recovery if the array or controller has another problem.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Tiff:
The distinction matters. Pointing at the wrong label doesn't make it the right device, so the confirmation needs to come from somewhere else.