Google has published its final incident report on the 20 October outage in europe-west2, the London region whose disruption hit a long list of UK services that morning. The headline cause is not a cyberattack, a software bug or an AI tool. It is one check in a programmable logic controller.
Here is the sequence, from Google’s incident report.
What happened
The facility hosting zone europe-west2-b normally runs on two independent utility feeds, A and B, with a generator bank behind them and five minutes of UPS battery to bridge the gap. After issues found in a quarterly generator load test on 14 October, feed B was opened and locked out for repairs. The site was running on feed A alone.
At 06:53 BST on 20 October, an underground cable fault outside the site cut feed A.
The generators should have started within the battery window. They did not. The PLC that commands a generator start was wired to sense voltage on the line side of the open B breaker. Upstream of that breaker, the utility was still live. So the PLC saw voltage, concluded a healthy source was available, and never sent the start command. Its logic did not also check the physical breaker position.
The batteries carried the load until they were empty. At 07:00 BST the data halls lost power. The site team tried a remote load transfer through the PLC, which inhibited it, and then moved the facility to a standby utility connection by hand. Power was back to the halls at 08:20 BST. Storage that had shut down uncleanly then needed hours of recovery before traffic could return.
The part that should worry operators
Every individual layer here was redundant. Two feeds. A generator bank. A dual redundant PLC. Batteries. Three zones in the region.
None of it helped, because the decision to use the backup was made by logic reading a signal that did not mean what the logic assumed. Two redundant PLCs reading the same misplaced input are one PLC.
Redundancy you have never failed over in your current configuration is a belief, not a capability.
The site was in a maintenance posture, running on one feed. That is exactly the moment the backup matters most and exactly the configuration least likely to have been tested end to end.
Zones did not contain it either
Google is candid that zones a and c were hit too. Failover traffic exhausted compute capacity in the surviving zones, some services hit failover issues as demand surged, and regional control planes kept timing out calling components in the dead zone. We made the same point the day it happened in zones are not a continuity plan: a cloud’s internal redundancy keeps the cloud up, on the cloud’s terms and timeline. It does not guarantee your workload a seat.
Why this matters for AI on critical sites
Turbines, compressors, substations and ground stations all run on the same pattern Google described: sensors, PLC logic, a backup that only engages if the logic believes it should. When AI is added to those sites, two things follow.
First, the model is only as good as what the sensors actually mean. A reading taken on the wrong side of a breaker is not wrong data. It is correct data answering a different question. Any system that monitors or reasons about plant state has to know where each signal is measured and cross check it against equipment state, the way Google’s fixed logic now checks breaker position as well as voltage.
Second, the decisions that matter during an incident cannot depend on a region that is having the incident. If your monitoring, alerting or AI assistance lives in the same cloud region as the failure, it goes quiet at the moment operators need it. That is the case for keeping critical inference on site and under the operator’s control, which we set out in what a cloud outage does to AI.
What to check this week
- For every automatic transfer or generator start, confirm exactly where each input signal is measured, and whether the logic also confirms breaker or switch position.
- List the maintenance postures your site runs in (single feed, bypass, bridged) and ask whether failover has ever been tested in each.
- Find out where your monitoring and any AI tooling actually execute during a site or region outage, and whether they keep working without the network.
Google says it will check PLC configurations at similar sites worldwide by 15 November. Every operator with a generator and a PLC can run the same check sooner.