After hearing about the FAA outage, I wondered how network changes are tested in critical infrastructure. Do operators maintain a duplicate network for this, or use some other approach?
Do critical infrastructure operators test router changes on a replica first?
Answers
A separate test environment can catch a lot, but it doesn't have to be a complete copy to be useful. You can test the relevant configuration, routing behavior and failure cases on a smaller setup, then deploy in stages with checks between sites.
A change that works at one site can still fail at another if configurations or connections differ. Testing should go alongside a rollout plan, monitoring and a way to back out the change, not replace them.
Reply to this answer
Duplicating an old network can be a real problem. If routers are decades old and no longer supported, finding enough matching equipment may be difficult, never mind reproducing every remote connection and years of configuration changes.
That doesn't mean testing is pointless. It means a lab may only represent part of the system, and the gaps matter when deciding how cautiously to deploy.
Reply to this answer
Timing and recovery matter too. Avoid making a risky change during a busy period if it can wait. Make sure monitoring will show trouble quickly, and have a practical rollback plan with people available to carry it out.
A rollback plan isn't much comfort if nobody knows whether it can be executed within the time the service can tolerate being down.
Reply to this answer
Testing an internal router configuration may not cover everything at a remote site. A provider connection and its equipment can introduce behavior that isn't present in a lab, so the scope of a test matters as much as whether it passed.
That is a general limitation, not an explanation of what happened in the FAA outage. The discussion doesn't establish what was tested or what caused it.
Reply to this answer
Add your answer
Answers are reviewed before publication.
_y61+brindlevisor.8:
So a successful test is evidence that the approach works, but not proof that every site will behave the same way?
7x+socketpiston_641:
Right. The closer the test matches the actual site, the more useful it is, but differences still need checking before rollout.