We're a small-to-medium-sized company that currently backs up our essential servers to the cloud. In a serious outage, we can spin those systems up in the cloud and access them over a VPN. We recently acquired another location across town and are planning to add infrastructure there, essentially creating a semi-warm or warm site. It's close enough that it wouldn't qualify as a geographically independent disaster recovery facility, but the two offices already communicate through dedicated connectivity.
I'm considering deploying another domain controller at the second site to provide local DNS and DHCP, along with replicating selected servers or virtual machines there so they could be brought online faster than cloud-based systems. We probably wouldn't make both sites fully hot, but the secondary location could help if the main building became unavailable and might reduce cloud recovery costs.
What would you prioritize in this setup? Are there particular services, systems, or operational procedures that are especially important to include?
4 Answers
You should have domain controllers and DNS available at both locations. Configure systems to use the local DNS server first and the DNS server at the other site as a secondary option. A local domain controller also gives the second office a better chance of continuing to authenticate if the main site is unavailable.
If you eventually want both locations to be active, you could design services to run at either site, but that can become complicated. Starting with replicated infrastructure and a documented failover process is a practical first step.
Since the sites are already connected with dedicated circuits, you have a good foundation for replication. I’d separate the plan into tiers: local DNS, DHCP, and authentication first; then critical applications and databases; and finally lower-priority systems.
Make sure failover includes more than starting a VM. You’ll need a plan for application dependencies, IP addressing, routing, firewall rules, backups, DNS changes, and returning services to the primary site. Run recovery exercises periodically so the procedure doesn’t exist only on paper.
Before buying hardware, define your recovery time objective, recovery point objective, and the number of virtual machines you actually need to restore. Those numbers should drive the design.
For a smaller environment, replicating only the most important workloads to the second site may be more cost-effective than building a complete duplicate data center. You’ll also need to account for storage, power protection, networking, firewalls, monitoring, and testing. A secondary site is only useful if you regularly verify that the replicated systems can actually start and operate there.
We already have a direct connection between the locations, so there’s no need to build a separate VPN for this. I need to inventory the workloads and decide which ones should be restored locally first rather than trying to duplicate everything.
Put DHCP at each site. DHCP depends on the network being available, so relying on a server across the link adds another failure point and makes troubleshooting harder. Local DHCP is inexpensive and lets each location keep assigning addresses even if the inter-site connection or the main office is down.
The same general principle applies to DNS and authentication: keep enough local infrastructure at each site for it to function independently during an outage. Avoid stretching a single broadcast-dependent service across the WAN when a local instance is easy to provide.
Even though the connection between the offices is a point-to-point dedicated circuit, I’ll still coordinate with the networking team about deploying DHCP locally. That removes one more dependency on the main building.
That makes sense. Our primary and secondary DNS are currently in the same building, so moving one to the other site seems like an easy improvement. We probably don’t need two fully hot sites, but having local copies available would be faster than starting everything in the cloud.