Cloudflare outages

Cloudflare proxies something in the region of a fifth of all websites. When it has a bad few minutes, a large and arbitrary-looking slice of the internet returns errors simultaneously, and a great many people conclude the internet itself is broken.

What makes these incidents worth studying is not that they are frequent - They are not - But that they illustrate a structural trade-off that applies to every edge network.

The structural problem

An edge network's value is that a change takes effect everywhere at once. A new firewall rule protects every customer in seconds. A routing change shifts traffic globally in moments. That speed is the product.

It is also the risk. A global deployment mechanism that can fix everything in thirty seconds can break everything in thirty seconds, and there is no meaningful difference between the two from the system's point of view.

Traditional software mitigates this with staged rollout: Deploy to 1%, watch, expand. An edge network can do that too, and Cloudflare does - But configuration and rules have historically been treated as a different class of change from code, distributed faster and with less staging, for the reasonable-sounding reason that a security rule is urgent.

That reasoning has produced more than one large outage, and not only at Cloudflare. The CrowdStrike incident had exactly the same shape: Content treated as lower-risk than code, pushed fast, everywhere.

The July 2019 regular expression

The cleanest illustration. A new rule was deployed to the web application firewall containing a regular expression that backtracked catastrophically - A pattern whose evaluation cost explodes on certain inputs.

The rule was distributed globally. CPU usage across Cloudflare's network went to 100%, and the network stopped being able to process traffic. A large portion of the web returned 502 errors for roughly half an hour.

The chain from one regular expression to a global outage is short: It was pushed everywhere at once, it consumed all available CPU, and CPU is what serves requests. There was no failing component to route around - Every location had the same problem simultaneously.

Cloudflare's post-mortem was unusually candid and included the actual pattern. The changes that followed were about process rather than code: Staged rollout for WAF rules, a global kill switch, and CPU-time limits on rule evaluation so no single rule can consume a whole core.

The July 2020 backbone incident

A router configuration change intended to alleviate congestion in one location. The change caused traffic to be withdrawn from a large part of the backbone in a way that sent far more traffic than intended to a single path, which promptly saturated.

Roughly half of Cloudflare's network was affected for around 27 minutes. The mechanism is the familiar one: A change intended to relieve a problem in one place redistributed load in a way that created a worse problem somewhere else, and the redistribution happened faster than anyone could observe it.

The 1.1.1.1 resolver

A separate and specific risk. Cloudflare operates the 1.1.1.1 public DNS resolver, which is configured on an enormous number of home routers and devices - Frequently by people who have no idea it is there.

In June 2024, a route leak combined with a BGP hijack made 1.1.1.1 unreachable for a portion of the internet. For affected users the experience was total: Nothing resolved, every site and app failed, and their connection was perfectly healthy throughout.

This is the most convincing impersonation of a global internet outage there is, and it produces a distinctive flood of reports from people whose internet is fine. The ten-second diagnostic is to open an IP address directly - If https://1.1.1.1 or another numeric address loads while names do not, the resolver is the problem.

It is also the argument for configuring two resolvers from different operators rather than one.

Telling a Cloudflare outage from a customer outage

Important, because most Cloudflare-branded error pages are not Cloudflare's fault.

Errors in the 520–527 range mean Cloudflare could not get a proper response from the customer's own origin server - Which requires Cloudflare to be working in order to tell you. Each of those numbers describes a specific origin failure.

A genuine Cloudflare incident looks different: Many unrelated sites failing at once, plain 502s or pages that do not load at all, and the Cloudflare status page reporting affected data centres. That page is unusually granular, listing incidents by individual point of presence, which makes it easy to confirm whether your nearest one is involved.

The transparency point

Cloudflare's incident reports are detailed, published quickly, and specific about what went wrong - Including the parts that are embarrassing. The 2019 report published the offending regular expression.

That is worth noting as a signal. The companies with the best reliability reputations are not the ones with the fewest incidents; they are the ones whose accounts of their incidents are detailed enough to be useful to everybody else. A public post-mortem is the strongest reliability signal a company can give, and it is voluntary.