On March 29th, Google experienced a six-hour outage in its us-east5-c cloud region, attributed to a failure in its uninterruptible power supplies (UPSes). The incident began with a loss of utility power, which left the UPSes incapacitated, preventing backup generators from activating. After engineers were alerted, generators came online nearly two hours later, allowing recovery of the majority of services. Google's commitment to prevent future incidents includes audits, better power recovery strategies, and collaboration with UPS vendors to enhance reliability.
Google's incident report states that the outage started with "loss of utility power in the affected zone." Hyperscalers build to survive that sort of thing with uninterruptible power supplies (UPSes) that are supposed to immediately provide power if the grid goes dead, and keep doing so for a few hours before diesel-powered generators kick in.
Engineers were alerted to the incident at 12:54 Pacific Time and their efforts saw generators come online at 14:49. "The majority of Google Cloud services recovered shortly thereafter," the incident report states.
Collection
[
|
...
]