Telstra's outage affected emergency services and was attributed to an obsolete time server and a GPS week rollover bug. How does that bug work, and what could prevent a similar failure?
How can an old time server cause a network outage?
Answers
Some GPS receivers use a 10-bit week number, which can represent 1,024 weeks before it wraps back to zero. That's just under 20 years. A receiver has to interpret the rollover correctly or it can report a date from the wrong cycle.
The reported device was a Symmetricom SyncServer S300, and its date reportedly reset to 2006. A bad time source can cause trouble for systems that depend on accurate time, though the supplied account doesn't explain the full chain from that date error to the emergency service disruption.
Reply to this answer
A time server isn't necessarily some irreplaceable custom box. The S300 was a commercial network time server, so checking replacement options and compatibility would be a sensible starting point.
Replacing the box still needs planning. You have to know which systems rely on it, how they consume time and how to change the source without creating another outage.
Reply to this answer
The maintenance lesson is bigger than the age of one server. Keep an inventory of time sources, check vendor support and known rollover behavior, and make sure someone owns the replacement plan. A risk written down with no funded action can sit there for years.
That doesn't establish why this particular device remained in service. The outage account alone doesn't tell us what decisions were made internally.
Reply to this answer
Redundancy deserves a look too. Multiple time sources can help, but not if they all share the same failure mode or clients blindly accept a wildly wrong date. Monitoring for large time changes and testing rollover behavior are worth considering alongside hardware replacement.
Vessby-7k:
That makes sense. A spare with the same bug wouldn't necessarily help when the rollover happens.
Reply to this answer
Critical services need scrutiny beyond this one failure. An emergency calling outage can also come from a change or upgrade, so replacing old equipment alone isn't a complete resilience plan. Testing changes and having a response plan matter as well.
Reply to this answer
Add your answer
Answers are reviewed before publication.
5v~notch_318:
So the rollover itself is predictable. The tricky part is whether a particular receiver or its software handles it correctly.