A timing glitch was behind Telstra’s nationwide outage. It points to a bigger vulnerability
Skip to content
George Chan/AAP
https://theconversation.com/a-timing-glitch-was-behind-telstras-nationwide-outage-it-points-to-a-bigger-vulnerability-287160
https://theconversation.com/a-timing-glitch-was-behind-telstras-nationwide-outage-it-points-to-a-bigger-vulnerability-287160
Link copied
Share article
Share article
Copy link
Bluesky
Messenger
X (Twitter)
Print article
Telstra experienced a second major network fault after yesterday’s nationwide outage, with the telco confirming late last night that some calls, including to Triple Zero, were not going through.
At a press conference this afternoon, Michael Ackland, Telstra’s Chief Financial Officer, apologised for the disruption. He said the company had completed 639 welfare checks on people who tried to call emergency services, and that seven people required assistance.
Yesterday’s outage crippled more than just people’s ability to make calls. It also brought down train services, payment systems, public transport ticketing systems and electric vehicle charging stations.
Telstra has blamed the outage on a computer timing failure in its network. But how can Telstra’s clocks have gone out of whack? Could this happen with other pieces of critical infrastructure that rely on accurate time? And what can be done to make timing systems more resilient and robust?
Read more:<br>How does a Telstra outage bring down trains? A telco expert explains
A clock that ticks with the universe
All computers incorporate what’s known as a software clock.
This clock serves several functions. It allows the computer itself, and any applications running on it, to tell the time. It also enables events to be timestamped and ordered, so we can ensure they are unfolding as they should and take timely action if needed, and enables time intervals to be calculated, so we know when something is taking too long or is happening too quickly.
If a software clock is inaccurate, many things can go wrong. For example, events may be accepted when they should be ignored, or flagged as dangerously late when actually they’re fine. Response times may be assessed as adequate when they’re anything but.
For a software clock to track real time, it needs to connect to timing hardware – something that actually “ticks” with the universe.
Typically, this is provided by an electronic circuit known as an oscillator that contains a thin piece of quartz crystal, relying on it to produce a regular “tick”. But these don’t tick perfectly. Left to themselves, a clock built on them will drift off quite quickly.
To ensure all software clocks agree, they must take input from a more accurate source aligned with a standard time reference, such as UTC (Coordinated Universal Time). They can get this standard time from satellites via a GPS signal, or access it over a data network. This is why your laptop clock never runs behind, while a simple battery-operated desk clock can run behind or ahead.
Most of the world’s computer population makes use of a hierarchy of time server computers. These communicate timestamp information (via the “Network Time Protocol”) over packets sent over the internet.
At the top of this hierarchy are so-called Stratum-1 servers. These are the only servers that actually connect to reference hardware sources, such as GPS. A single Stratum-1 server communicates time to a set of Stratum-2 servers over the network, each of which communicates to a set of Stratum-3 servers, and so on. The goal of such a system is to propagate the reference accuracy down the hierarchy inexpensively.
However, such a network brings vulnerabilities, as each Stratum-1 guides (or misleads) an entire server tree below it.
More accurate timing networks rely on additional hardware and dedicated links, and can be far more expensive.
So, what went wrong with Telstra?
The telco giant has said the outage was caused by nodes that managed time synchronisation within some of its network data centres.
The Sydney Morning Herald cited two internal sources who said a faulty update caused some of Telstra’s servers to reset their clocks by almost 20 years, making them believe it was November 2006.
We do not have any details beyond this at this point.
But it’s entirely possible these were Stratum-1 NTP servers, and that their issues misled the servers and computer lying below them in the hierarchy across the data centres and perhaps beyond.
The resulting errors could then have led to a cascade of secondary effects involving multiple interconnecting software systems.
Keeping track of time
Telecommunications is only one domain in which timing faults can have serious consequences.
In fact, because of society’s now enormous dependence on GPS as a source of time as well as position, the vulnerabilities are very broad indeed. A particularly important example is the electricity grid, which...