There are plenty of tutorials for setting up infrastructure, but fewer chances to practice when things go wrong. Do you make broken environments, rely on real incidents, or use hands-on labs? What would make an incident simulation useful?
How do you practice troubleshooting DevOps incidents?
Answers
A team drill on a staging cluster is a good place to start. Have one person introduce a fault without telling the person investigating what it is. Try a bad image tag, a readiness probe pointed at the wrong port, or a node running out of disk.
Use the tools you would normally have during an incident, and include some irrelevant log lines. A timer can add pressure, but don't make finding the cleverest fix the goal. If a straightforward rollback restores service, that should count as a successful response.
Reply to this answer
The best scenarios aren't tidy puzzles where the error message gives the answer away. Make the system unfamiliar to the learner, give them realistic tools and output, and let them work through noisy evidence.
I'd also try a drill with limited telemetry. If metrics or application logs are missing, the investigation might still use audit logs, configuration history, and code changes. That teaches a different skill from following a perfect dashboard to the root cause.
Reply to this answer
Real incidents are hard to fully reproduce in a lab. Practice can teach useful moves, but working through an actual failure still brings uncertainty that a prepared scenario may not capture. I'd treat drills as preparation, not a replacement for that experience.
Reply to this answer
For Kubernetes practice, a small local cluster can be enough to get started. Have someone else introduce a taint or break a kubelet configuration, then investigate the affected resources and events before jumping straight into logs.
A platform that injects unknown faults could be useful for solo practice, especially if it offers hints without immediately revealing the cause.
Reply to this answer
There's also the AI angle. Automated tools can help investigate failures, so a lab could include practice using them. The useful test is whether someone can evaluate the suggestions and work out what to do when the answer is incomplete.
Reply to this answer
Add your answer
Answers are reviewed before publication.
2a~plaidkettle_846:
Stateful workloads are worth a separate drill. A node failure involving persistent storage can behave quite differently from stateless pods being rescheduled.