Feedback on My Disaster Recovery Plan and Runbook Structure

0
2
Asked By MellowCedar47 On

I'm a solo junior SRE responsible for a web application with IoT components. Our infrastructure includes Docker Swarm, Kafka, Redis, and several other clustered services. The servers are hosted remotely and managed mostly with Ansible and Terraform. Each major service has its own cluster, in addition to the application's Swarm cluster.

I've been building our disaster recovery documentation from scratch for nearly two years. My goals are to support ISO 27001 compliance and make sure coworkers can recover systems if I'm unavailable or eventually leave the company.

Our documentation is organized into four main areas:

- Disaster recovery plans: incident scenarios, decision-making, and links to technical procedures
- Runbooks: detailed operational procedures
- Incidents: postmortems and historical records
- Workshops: training sessions and records of who has practiced which procedures

The DRPs and runbooks are grouped by cluster and environment, with additional sections for security and server management.

A typical DRP covers scenarios such as a node failure, quorum loss, or Raft corruption. I intentionally keep commands and detailed technical steps out of these documents so they can be followed by someone who is less familiar with the operational side. Each DRP generally includes:

- Criticality, RTO, and RPO
- Initial assessment, including detection links, expected user impact, and precautions
- Analysis, including diagnostic references and trigger confirmation
- Recovery actions with different branches for possible causes
- Links to the relevant runbooks
- Post-recovery validation and cleanup steps
- Last-update information and links to related postmortems

The runbooks contain the technical material, including architecture information, infrastructure configuration, command references, diagnostic guides, and procedures such as adding or removing nodes or restoring cluster state. I try to keep them concise and task-focused.

I recently added separate diagnostic pages for checking cluster and service state. That has made me question the workflow: should it be diagnostics, then the DRP, then the runbook? Or should the DRP remain the main entry point and link to diagnostics as needed?

Is separating DRPs from runbooks a sensible approach? How do larger organizations structure these documents? For incidents that have never occurred, should I label the procedures as hypothetical, tabletop-tested, or something similar? I can sometimes reproduce part of the impact, but the symptoms and logs are not always identical to a real incident.

I've tested the full procedures at least once, and many runbooks were written while carrying out the actual work, but some documents using the newer template have not been fully tested yet. I'd appreciate feedback on the structure, workflow, missing information, and ways to make this more useful for coworkers.

4 Answers

Answered By CopperField56 On

I’d make the handoff between the document types especially clear. DIAG pages should contain inputs, checks, dashboards, and expected versus abnormal signals. DRPs should contain severity, impact, decision branches, communication requirements, approvals, and stop conditions. RUN pages should contain prerequisites, commands, expected output, verification, cleanup, rollback details, and an abort point.

For each procedure, it’s useful to record the expected duration, potential blast radius, owner, reviewer, and last successful test date. By blast radius, I mean the systems, users, data, or dependent services that could be affected if the procedure succeeds, fails, or is performed incorrectly.

The most valuable improvement may be regular rehearsal rather than more categorization. Pick the most important scenarios and run tabletop exercises with a non-SRE teammate driving from the DRP. Every question they ask is a sign that the documentation could be clearer.

Answered By QuietHarbor82 On

Keeping the DRP and runbooks separate is a good idea. They answer different questions: the DRP explains what is happening, how serious it is, who makes decisions, and which recovery path is appropriate. The runbook explains exactly how to perform one technical action.

I’d make the workflow explicit near the top of every DRP: start here, run the linked diagnostics, select the appropriate recovery branch, execute the linked runbook, then validate the result. The DRP should be the map, not a second copy of the commands.

Diagnostics can be treated as a shared reference library rather than a mandatory stage. Both DRPs and runbooks can link to the same diagnostic pages, depending on what needs to be confirmed.

For scenario maturity, add a clear status such as observed, rehearsed, tabletop-only, or hypothetical. Also track the owner, last review date, last rehearsal date, expected blast radius, rollback or stop conditions, and post-recovery checks. A stop condition is especially valuable because it prevents someone from continuing when the situation is becoming worse.

MellowCedar47 -

The idea that the DRP should be the map makes sense. I think that’s the distinction I was missing. I’ll also add clearer status and rehearsal information so people don’t assume an untested procedure is reliable.

Answered By SilverPine29 On

The most important test is whether somebody else can actually use the documents under pressure. A procedure that looks complete but has never been fire-drilled is still mostly an assumption. Run the whole chain with another coworker, measure where they get stuck, and update the DRP and runbooks based on what happens. The folder structure matters much less than whether the process works in practice.

Answered By AmberLynx31 On

Your overall structure is already more disciplined than what many larger teams maintain. I would keep the three document types separate:

- DIAG proves which state the system is in.
- DRP decides what recovery path to take.
- RUN executes one specific procedure.

The DRP can remain the human entry point. A typical flow would be: identify the symptom, use linked diagnostics to confirm the situation, choose a recovery case, run the relevant procedure, validate the result, and record the incident.

For scenarios that have never happened, label them clearly with something like tested, tabletop-tested, synthetic-tested, or theoretical, along with a last-reviewed date. A tabletop exercise is useful when reproducing the failure safely is impractical, and it also provides useful evidence that the process is being reviewed for compliance purposes.

In larger organizations, the concepts are usually similar but more formalized. They tend to define service tiers, assign owners, schedule recovery drills, record RTO and RPO targets, and automate failover where the cost is justified.

Your largest risk is probably the dependency on one person. The workshop and training area is therefore important. Have someone unfamiliar with the systems follow a DRP while you only observe. Record every question, hesitation, and point where they need help. Those observations usually reveal more than adding another page of documentation.

MellowCedar47 -

That makes sense, especially treating diagnostics as a shared library rather than a step that must always come first. I also agree that improving the workshop and reducing the single-person dependency should be a priority.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.