The Fastly outage

On 8 June 2021, at 09:47 UTC, a very large part of the web stopped working at once. Amazon, Reddit, Twitch, PayPal, Spotify, the UK government's website, the New York Times, the Guardian, the BBC, Shopify stores and a long list of others all returned errors within the same minute.

The common factor was Fastly, a content delivery network most of the public had never heard of. The outage lasted about an hour, with the bulk of service restored within 49 minutes.

The mechanism

Fastly's account, published the same day, is admirably direct.

On 12 May 2021, Fastly deployed a software update containing a bug. The bug was not triggered by anything in normal operation and sat dormant for nearly four weeks.

On 8 June, a customer made a valid configuration change - An ordinary change, within the documented product, that any customer was entitled to make. That specific configuration triggered the dormant bug.

Within about a minute, 85% of Fastly's network was returning errors.

Why one customer could do that

This is the part worth dwelling on, because it is a property of shared infrastructure rather than a mistake anyone made.

A CDN runs the same software everywhere. Every point of presence executes the same code paths, and a customer's configuration is data fed into that shared software. When the software has a defect that a particular configuration triggers, every location processing that configuration hits the same defect at the same moment.

There was no failing region to route around, because there was no regional component to the failure. The bug was everywhere the software was.

The customer did nothing wrong. Fastly was explicit about this - The configuration was valid. The defect was Fastly's, and the customer was simply the first to exercise the path that reached it.

Latent bugs

The four-week gap between the deployment and the trigger is the most instructive detail.

Standard release safety practice assumes that a bad deploy shows up during or shortly after rollout: You watch error rates, you compare against a baseline, you roll back if something moves. That works for defects exercised by normal traffic.

It does nothing for a defect that requires a specific input nobody has provided yet. The deployment looks perfectly healthy because, for four weeks, it was. The failure is triggered later, by something entirely unrelated to the release, and the connection between the two is not obvious to anyone looking at it in real time.

The response

Fastly's handling of the incident is the reason it is remembered as a case study rather than as a scandal.

The problem was identified within about a minute of onset. The affected configuration was disabled, and the global network began recovering within 49 minutes. A permanent fix was rolled out later the same day.

A detailed public post-mortem was published on the day of the incident, describing the dormant bug, the trigger and the timeline honestly. That speed is unusual - Many companies take a week and say less.

What it showed about concentration

The outage was a widely cited demonstration of how few organisations the web actually depends on.

A handful of CDNs sit in front of a very large share of all websites. They exist because delivering content globally with low latency is genuinely hard and genuinely worth outsourcing, and the resulting concentration is an efficiency, not a mistake.

But it means the failure modes of the internet have changed shape. The network was designed to route around damage, and it still does - At the network layer. At the application layer, a great deal of the web now shares a small number of single points of failure that no routing protocol can help with.

This is the same observation as the us-east-1 problem and the Cloudflare incidents, arriving from a different direction each time. When several unrelated sites fail together, the explanation is almost always a shared layer.

What changed afterwards

Fastly committed to better testing of configuration paths and faster detection of the class of failure where a customer input triggers a code defect.

The broader industry response was more about architecture than about Fastly. Multi-CDN configurations - Where a site can fail over between two providers - Became noticeably more common among large sites afterwards, though they remain complicated and expensive, and most organisations looked at the cost and decided an hour was survivable.

Which is a reasonable conclusion, and worth being honest about. Most sites cannot economically be more reliable than the layer they sit on. The realistic mitigation is not redundancy at every level; it is knowing which shared layers you depend on, so that when several unrelated things break at once you can recognise what you are looking at.