OT Manual Operations Plan: What Manual Operation Buys You
✨ NEW GUIDE -- CIP-015 Compliance Guide helps industrial operators prepare for INSM requirements View the Guide X
Search
Blog
Contact Us
FREE PCAP Analyzer
Company
About EmberOT
Leadership
Our Partners
Events
Product
EmberOT In-Depth
Asset Inventory & Insights
Vulnerability & Risk
Detection
PCAP Analyzer Free Tool
Firewatch Assessment
IgniteOnsite
Resources
Resources
Blog
Documents
Podcasts
Newsroom
ICS Vulnerability Report
CIP-015 Compliance Guide
Solutions
Solutions
Oil & Gas
Electric Utilities
Industrial IoT
Manufacturing
Rural Co-ops
Request a Demo
Company
About EmberOT
Leadership
Our Partners
Events
Product
EmberOT In-Depth
Asset Inventory & Insights
Vulnerability & Risk
Detection
PCAP Analyzer Free Tool
Firewatch Assessment
IgniteOnsite
Resources
Resources
Blog
Documents
Podcasts
Newsroom
ICS Vulnerability Report
CIP-015 Compliance Guide
Solutions
Solutions
Oil & Gas
Electric Utilities
Industrial IoT
Manufacturing
Rural Co-ops
Request a Demo
Home1 > Resources2 > Blog3 > OT Manual Operations Plan: What Manual Operation Buys You
Blog
OT Manual Operations Plan: What Manual Operation Buys You
Jori VanAntwerp
CEO and Founder at EmberOT || Web<br>For over two decades, Jori has enabled industrial and IT organizations to be successful in reducing risk, increasing compliance, and improving their overall security efforts. He has had the pleasure of working with companies such as Gravwell, Dragos, CrowdStrike, FireEye, McAfee, and is now CEO & Founder at EmberOT, a cybersecurity startup focused on making security a reality for critical infrastructure.
On the morning of Monday, July 27, the operating controls at the well and water treatment plant in Braham, Minnesota, went dark. Braham is a city of about 1,700 people. Public works crews had the plant running again in roughly two hours on manual operation.
Braham was one of more than 30 community water systems in the state hit in a coordinated cyberattack that weekend. On July 30, the FBI and EPA issued a joint public service announcement confirming that water and wastewater utilities in at least seven states had reported incidents, with actors remotely changing PLC IP addresses and passwords and operators losing monitoring and control.
CISA issued its own alert the same day, noting that across the wider campaign the activity had led to boil water notices and sustained manual operations, and that entities of all sizes were being targeted.
Nearly every piece written about it since lands in the same place: get your PLCs off the public internet. That’s correct, and I won’t argue with it.
The part I keep coming back to is what the operators did about it. What they had, whether formal or informal, was the foundation of an OT manual operations plan: people who knew the process, local controls that still worked, and enough procedure to keep water moving without supervisory control.
In Plymouth, communications dropped at two water towers and several lift stations, all of it cellular-connected equipment. The city pulled that gear off the network while crews kept the system running manually. South St. Paul switched to manual while its automated controls were restored. Maple Plain declared a local state of emergency to speed its response. State officials reported water quality was unaffected at every system hit, and no boil water advisories were issued anywhere in Minnesota.
Some of those utilities lost supervisory control. Plymouth gave it up on purpose, cutting off equipment it could no longer trust. Either way the water kept moving, because people who understood the process stepped in and worked it by hand.
In the incident write-ups, that shows up as a footnote. Manual operation gets logged as a degraded state, the sad middle of the timeline before automation comes back. It’s one of the strongest defenses available in an industrial environment, and OT is the only place you get it.
Why Manual Operation Works
Manual operation is the practice of running a physical process through local controls when supervisory systems are unavailable or can no longer be trusted. What makes it a defense comes down to what an attacker in an OT environment is usually after. The objective is a physical outcome: a pump that runs when it shouldn’t, a setpoint that drifts, a valve that moves, a plant that stops. Access to a controller is the road to that outcome.
Manual operation cuts the road.
When an operator takes local control of a pump station, the compromised path between the network and the physical process stops carrying anything that matters. Whatever the attacker still holds, and they may hold quite a lot, no longer reaches the thing they want to affect. The process is being driven by a person standing next to it, through a mechanism with no network interface to compromise.
That’s out of band in the most literal sense.
Hold that next to an IT...