<
https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-analytics-outage/>
"Beginning on Thursday, November 2, 2023, at 11:43 UTC Cloudflare's control
plane and analytics services experienced an outage. The control plane of
Cloudflare consists primarily of the customer-facing interface for all of our
services including our website and APIs. Our analytics services include logging
and analytics reporting.
The incident lasted from November 2 at 11:44 UTC until November 4 at 04:25 UTC.
We were able to restore most of our control plane at our disaster recovery
facility as of November 2 at 17:57 UTC. Many customers would not have
experienced issues with most of our products after the disaster recovery
facility came online. However, other services took longer to restore and
customers that used them may have seen issues until we fully resolved the
incident. Our raw log services were unavailable for most customers for the
duration of the incident.
Services have now been restored for all customers. Throughout the incident,
Cloudflare's network and security services continued to work as expected. While
there were periods where customers were unable to make changes to those
services, traffic through our network was not impacted.
This post outlines the events that caused this incident, the architecture we
had in place to prevent issues like this, what failed, what worked and why, and
the changes we're making based on what we've learned over the last 36 hours.
To start, this never should have happened. We believed that we had high
availability systems in place that should have stopped an outage like this,
even when one of our core data center providers failed catastrophically. And,
while many systems did remain online as designed, some critical systems had
non-obvious dependencies that made them unavailable. I am sorry and embarrassed
for this incident and the pain that it caused our customers and our team."
This is a good post mortem and apology. Via Christoph S.
Cheers,
*** Xanni ***
--
mailto:xanni@xanadu.net Andrew Pam
http://xanadu.com.au/ Chief Scientist, Xanadu
https://glasswings.com.au/ Partner, Glass Wings
https://sericyb.com.au/ Manager, Serious Cybernetics
