NTP server that traveled back in time caused massive Aussie mobile outage

Jump to main content

Search

REG AD

networks

NTP server that traveled back in time caused massive Aussie mobile outage

Telstra skipped a patch, didn’t record changes, had no idea it was an accident waiting to happen

Simon Sharwood

Simon<br>Sharwood

APAC Editor

Published<br>fri 17 Jul 2026 // 05:48 UTC

Australian telco Telstra has revealed the cause of the recent incident that caused widespread connectivity problems across its mobile networks, inculding outages to Australia's 000 emergency services line, plus outages to electronic payments services and transport networks.<br>The carrier explained itself in a submission [PDF] to a Senate inquiry into outages affecting Australia’s emergency services which initially investigated an outage at Telstra's main rival, Optus. The Optus incident is linked to multiple deaths after people could not reach emergency services.<br>Telstra's submission reveals that the carrier's attempt to address a known resilience problem was the instigating incident.

REG AD

That problem was a faulty backup power feed in the chassis used to house a network time protocol (NTP) server. A few minutes before midnight on July 7, a Telstra techie started work to replace that chassis.

REG AD

By 3:38 AM on July 8, the worker had finished the job and powered up the server.<br>The NTP server included a GPS card that Telstra says “did not operate as expected” when the server came back online.

MORE CONTEXT

Telstra outage: Failed emergency services calls, train chaos, payment systems down

Aussie telco Telstra says soz after accidentally diverting traffic meant for encrypted email biz through its servers

Telstra chairman: If those darn kids can earn $5m playing Fortnite, why can't execs?

Telstra told by regulator to keep copper for an extra five years

“We now believe this occurred because of an intentional design change that had previously been made to the equipment to fix an earlier fault [which] had not been properly documented,” the submission states. “This meant the maintenance team was not aware of the way the device would behave when restarted.”<br>Telstra also admitted it had not applied a software update to the device, despite knowing of its availability in early 2026.<br>“Had that software update been completed or had the design change been properly reviewed and documented post the earlier incident, and reflected in the maintenance procedure, the outage may not have occurred,” the submission states.<br>The outage did happen because once the NTP server came online it reset its clocks to the year 2006.<br>The server then did what NTP servers do: publish that time to myriad other machines across the network.<br>Once those machines received the incorrect time, other kit on the Telstra network compared digital certificates and decided something dodgy was going on – so denied connections.

REG AD

Just one of Telstra’s three NTP servers had this problem, but enough bad timing info made it onto the carrier’s network to cause chaos.<br>The telco isolated the bad box at 7:11 AM, and by 10:30 AM had identified all network components that used the bad time information the naughty NTP box broadcast.<br>That wasn’t the end of the matter, because as valid time information rippled across the network, some equipment didn’t close IP sessions. Customer devices therefore couldn’t reconnect to the network unless rebooted.<br>The carrier has described the outage as “clearly unacceptable.”<br>“If maintenance work can trigger this kind of outage, it suggests our controls were not good enough. We are accountable for that, and our investigation will address why that design change was not documented, why the software update was not completed, and what needs to change in our controls so known risks are captured, prioritised and closed before they can affect customers.”<br>The carrier has done the usual thing by promising to conduct deep and lengthy self-reflection, submit itself to further flagellation at more inquiries, co-operate with a probe by the relevant regulator, and to do what it takes to prevent similar incidents. ®

network time protocol<br>australia<br>telstra<br>ntp<br>networks<br>outage

REG AD

off-prem

Billing software error sends billion-dollar AWS estimates

Amazon asks users not to panic as it works to fix the bug

security

AI spam filters are getting suckered by old-school text salting

Turns out decades-old email tricks still work against some LLM-powered email filters

Gobi X: Creating more energy for AI, not taking it from society

PARTNER CONTENT: How Envision is reversing the datacenter playbook by making computing chase abundant desert power, not the other way around

SAAS

Microsoft gives admins Exchange Online breathing room

Retirement of PowerShell -Credential parameter pushed back to the end of 2026

PaaS + IaaS

Amazon Web Services' most vocal customer now runs EC2

Retail foundation leader Dave Treadwell takes over as senior leader and 19-year vet...