More Than 30 Water Utilities Lost Control in a Weekend: Give Every Unpatchable Controller an Owner and an End Date

The views expressed are my own and do not represent those of my employer.

On Monday morning, July 27, city officials across Minnesota started finding that automated controls at their water utilities had failed. A coordinated attack on July 26 and 27 had hit operational technology at more than 30 community water utilities. In Braham, the attackers shut down the operating controls, and the city briefly took its water plant offline.

The damage stayed limited. Utilities kept water service going through manual operations and contingency plans. In all confirmed cases, officials said drinking water quality was not affected. That outcome depended on people who knew how to run the process by hand.

Attribution is unsettled. Officials have not publicly attributed the attack to any actor, although Tenable researchers suspect the Iran-linked group CyberAv3ngers. I will treat that as suspected and nothing more. The exposure pattern matters more than the name. Tenable ties the group's history to small utilities that expose PLC interfaces directly to the internet or run OT through consumer remote access tools. CISA urged utilities to remove internet exposed PLCs. The FBI and EPA guidance said to place any PLCs that stay reachable behind secure gateways and firewalls.

You probably do not run a water utility. You run a plant, and you own the same problem in a different building. This is part 3 of 5. It covers one thing: what to do with the controller that cannot be patched.

The Sentence That Ends the Conversation

Every manufacturing CISO has heard it. Someone from operations says the controller cannot be patched, and the meeting moves on. The finding stays open for three years. The risk never gets a name, a date, or a budget line.

That sentence is a fact about a device. It is not a risk decision. The decision is who carries the risk, with what protection, until when, and what replaces the device. Until someone writes those answers down, "cannot patch" works as a permanent excuse.

"Cannot patch" describes the controller. It says nothing about who owns the risk or when it ends.

My job as a CISO is to turn that excuse into a tracked exception that ends. The rest of this post shows how I would build it.

Why OT Is Different, and Why That Is Not an Excuse

I will not pretend patching a controller is like patching a laptop. It is not. Downtime on a production line costs real money every hour. Vendors certify specific firmware and software combinations, and a patch outside that certification can void support or change behavior. Older controllers have no patch at all.

NIST agrees. In SP 800-82 Rev 3, NIST notes that a patch can change how an operating system or application works with the control application, which can cost the control application some of its functionality. Sometimes the patch does not exist. Tenable's write up describes a vulnerability exploited in the broader campaign that has no available vendor patch. Tenable also says it has not been confirmed as the vector in Minnesota.

So the constraint is real. What changes is how you handle it. NIST does not tell you to ignore patching. It says the patch process should cover how to select compensating controls that limit exposure of the vulnerable system when patching is delayed. Delayed implies a date. Your program needs one.

One Row Per Asset: The Register That Replaces the Excuse

Start with a register. Every asset that cannot be patched safely, or at all, gets exactly one row. No row means no exception, and an asset with no exception is a finding. The row holds seven fields:

• Asset: the controller, HMI or engineering workstation, with location and process it supports.
• Exposure path: every route that reaches it, including remote access, vendor connections, cellular modems and shared engineering laptops.
• Named business owner: a person from the plant or the business, not a team and not security.
• Compensating controls: the specific protections in place today.
• Expiry date: the day this exception stops being valid unless someone renews it.
• Review triggers: the events that force a review before expiry.
• Replacement budget year: the fiscal year when money for the replacement is approved.

The exposure path comes second for a reason. It tells you how urgent the row is. A controller on an isolated cell network and a controller with a remote access path look identical in an asset list. They are not the same risk.

The owner field does the most work. I want a plant manager or operations director in that cell. When security owns the exception, the business has no reason to end it. When operations owns it, the person who benefits from the uptime also carries the risk.

One more point from NIST. SP 800-82 says compensating controls are not exceptions or waivers to the baseline controls. I read that as a design instruction. The controls are the work. The register row records what residual risk is left and who accepted it.

The End of Life Column You Fill In by Hand

Every register needs an end of life column. For IT software, tools can fill it in. I like endoflife.ai for that. It tracks lifecycle dates for 700+ products and reconciles them against vendor lifecycle pages and the endoflife.date community dataset. Its 0 to 100 EOL Risk Score weighs how long a version has been past end of life, attack surface, CISA KEV exposure, and whether extended support exists. The checker is free.

Look at what it covers. The published examples are products like Node, Python, Ubuntu and Postgres. I have not found PLC firmware or proprietary HMIs in its coverage, and you should check for your own models before you rely on it. It does earn a place on the IT side of the plant network. It also lists end of support status for edge devices, and the firewalls and gateways around your OT zones belong in that check.

For controllers and vendor HMIs, make the column manual. Record the vendor's last sale date, the last firmware release, any end of support notice, spare parts availability, and where you got each date. Name a person who rechecks it. A blank cell is useful information, because it means nobody knows how long the vendor will stand behind that device.

Fix the Exposure First, Then Stack the Controls

The compensating controls field is where people list a firewall and call it done. Fill it in a priority order instead. Exposure goes first because exposure decides how fast the clock runs.

An IANS faculty summary of the Q3 2026 Vulnerability and Breach Update, from Mike Saurbaugh, says to rank fixes by real exploitation and exposure, not CVSS. The full report is client only, so I will stay with that headline. It fits OT well. Rapid7 reports that exploitation happened on average just three days after public disclosure in Q3 2026. An unpatchable controller with a path to the internet has no time to spare.

1. Remove internet exposure. No OT device should answer from the public internet. Minnesota showed what exposed PLCs invite. This is the one control I will not negotiate.

2. Build conduits and virtual patching. Put the controller in a zone and allow only the specific flows its process needs. Then block the vulnerable service path at the network layer, so the exploit cannot reach the weakness even though the weakness remains. I covered how to harden the boundary in NCSC Secure Connectivity Principle 5: Harden Your OT Boundary.

3. Allowlist what runs. On HMIs and engineering workstations, allow only approved applications. A frozen system that runs only what it should is a strong control for an unpatchable host.

4. Monitor passively. Watch traffic with a passive sensor on a span port. Do not run active scans against controllers. NIST SP 800-82 favors passive methods for the same reason: an active probe can disturb a process.

5. Keep offline backups. This one gets its own section below.

Know your cutoff plan too. If the controls fail, you need a way to cut the plant zone off from everything else without stopping production. I wrote about that in NCSC Secure Connectivity Principle 8: Establish an Isolation Plan. For the broader practice of ranking by exposure, see Stop Scanning. Start Managing Exposure.

A compensating control that nobody reviews is a permanent exception with better paperwork.

Offline Backups Are the Control Nobody Funds Until Monday Morning

If you cannot patch, you must be able to recover. Tenable advises backing up all PLC logic and configurations offline on secured physical media, and testing restore procedures. The reason is documented. CISA's updated advisory documents attackers stealing PLC project files for the first time. The FBI also reported that at least one organization found modified project files after noticing ladder logic discrepancies across several sites.

Two practices matter here. First, compare running logic against your known good copy on a schedule. The FBI suggests using vendor integrity tools and visually comparing the running program to known good logic. Second, treat the backup itself with suspicion. If you restore from backup, verify the backup contains no malicious logic before deployment.

Add a restore test date to the register row. An untested backup is a hope. On controllers with a physical mode switch, also check that it sits in the Run position. Tenable says this blocks remote modification of PLC logic and configurations.

Risk Acceptance That Expires

The expiry date is what separates a managed exception from a standing excuse. Without it, an accepted risk renews itself by silence.

My recommendation is a ceiling of twelve months. Shorter for assets with an exposure path. The business owner signs, not the CISO. At expiry, one of three things happens: the owner renews with evidence that controls still work, the replacement moves forward, or the decision escalates to an executive. Silent renewal is not an option.

Controls also carry a time limit. Compensating controls cannot substitute for architectural decisions indefinitely, according to Anchor Defense's analysis of SP 800-82. That is a commentary, not NIST text. It matches my experience, and it supports putting an end date on every row.

Then add review triggers, the events that pull a review forward. I use these:

• A new CISA KEV listing that touches the asset or its platform.
• A change in the exposure path, such as a new vendor connection.
• A vendor end of support announcement.
• A failed restore test or a failed control check.
• A change to the process the asset supports.

A Replacement Roadmap That Survives Procurement

The replacement budget year is the field that turns a date into a commitment. It is also the field most likely to be fiction.

Test every replacement year against reality. Work backward from the year. How long does the vendor take to deliver a controller? When is the plant's next planned shutdown? Who writes and validates the new logic? Does the integrator have capacity? If the plant only shuts down once a year, a replacement that misses that shutdown slips twelve months.

I ask procurement and plant engineering to review the roadmap together before I take it anywhere. They find the problems I cannot see. Then the budget year is a plan someone can defend, and not a wish.

Agree the Emergency Change Path Before the Emergency

Sooner or later a vulnerability lands that your controls do not fully cover, on an asset in your register. That is a bad day to invent a process.

Write down the emergency path in advance. Who can authorize an outage window outside the normal calendar? Who calls the vendor? What is the tested rollback? What is the manual operations fallback? When do you isolate the zone instead of changing anything? Get the plant manager, operations lead and CISO to agree on those answers when nothing is on fire.

The best emergency change is the one the plant already approved on a quiet Tuesday.

Getting Organizational Support: Say It in Dollars and Downtime

Nobody funds a register because it is neat. They fund it because it answers questions they care about. Frame every conversation as business risk.

Plant leadership. They worry about uptime and safety. Tell plant leadership that the controller cannot be patched safely, so it sits behind named controls with an owner and an expiry date. Tell them you are not asking to stop the line. Tell them the register protects production because it plans the replacement for a shutdown they choose, and not for one an attacker chooses. Ask them to name the owner.

Operations and engineering. Treat them as co authors of the register. They know the exposure paths, the vendor limits and the real shutdown calendar. If the register is something done to them, they will route around it.

Finance and procurement. Bring the replacement budget years as a forward schedule. Finance would sooner see a planned replacement curve over several years than an emergency request. Procurement tells you what lead times really are.

Executives. Give them a one line risk statement per cluster of assets: what can stop, for how long, and what it costs in revenue per hour. Use their numbers, not mine.

The board. Boards want to know whether exceptions end. Report four numbers each quarter: unpatchable assets, how many have an exposure path, how many are past expiry, and replacement spend by year. A board can read that table in a minute. Item three tells them whether management follows its own rules.

Key Points

• "Cannot patch" describes a device. It is not a risk decision.
• One register row per unpatchable asset, with seven fields.
• The owner is a business leader, not security.
• Fill the end of life column by hand for controllers and vendor HMIs.
• Remove internet exposure first. Then conduits, allowlisting, passive monitoring and offline backups.
• Test restores and compare running logic against known good copies.
• Every acceptance expires. Set a twelve month ceiling.
• Check the replacement year against procurement lead times and shutdown calendars.
• Agree the emergency change path in advance.

Pro Tips

Sort the register by exposure, not by age. The oldest controller on an isolated cell is usually a lower risk than a newer one with a remote access path. Let exposure set the review order.

Walk the plant before you fill the exposure column. Network diagrams miss cellular modems, vendor laptops and the connection someone added during an outage. Walk it with a controls engineer.

Put the vendor's name next to the date. An end of support date with no source is a rumor. If the vendor will not give a date in writing, record that fact.

Run a restore on a bench unit. You can seldom test a restore on a live controller. Keep a spare of the same model and firmware, and use it for drills.

Make the register the audit evidence. Auditors ask how you manage unpatchable systems. Hand them the register, the sign offs and the control checks. That one artifact answers most of the questions.

Pitfalls to Avoid

Do not accept a risk without an expiry date. Without one, the exception becomes policy by default.

Do not let security own the exception. The owner has to be someone who can fund or schedule the fix.

Do not treat a firewall as the whole answer. Controls fail, and exposure paths change. You need monitoring, backups and review triggers around it.

Do not run active scans on controllers to build the register. Use passive discovery and the plant's own records, and test on a bench first.

Do not set a replacement year you cannot defend. A date that ignores lead times and shutdown windows loses credibility with finance and the plant at once.

Do not wait for a disclosure to decide who can approve an emergency outage. Settle it ahead of time.

Final Thought

Minnesota did not show that utilities lack good people. It showed what happens when a device that cannot change sits where anyone can reach it, and nobody has a date by which that stops being true. Your plant will always have a controller that cannot be patched. What you control is whether it has an owner, a date, and a year when it leaves.

If this was useful, subscribe to InfoSec Made Easy so you do not miss the rest of this series. Share it with the OT or plant engineering leader you work with. Tell me in the comments how you handle unpatchable controllers today, and what you would add to the register. I read every one.