The us-east-1 problem

There is a recognisable kind of internet outage. A smart doorbell stops answering. An airline's check-in fails. A project tracker will not load. A streaming service loses its home screen. A bank's app spins. All within about ten minutes, and none of these companies has anything to do with the others.

That pattern almost always means one thing: A problem in AWS us-east-1.

Why one region matters so much

us-east-1, in Northern Virginia, was AWS's first region and remains its largest. That history is the whole explanation.

It is the default. For most of AWS's existence it has been the region selected unless you chose otherwise. Countless deployments run there because nobody picked anything else, and moving later is expensive.

It gets features first. New services launch there, so anything built against a new capability starts there.

It is cheapest. Marginally, but at scale that matters.

And, critically, some global services have their control plane there. Certain AWS services that present as global are operated from us-east-1 - Historically including parts of IAM, the Route 53 control plane, CloudFront configuration and billing. A customer in Frankfurt running nothing in Virginia can still be affected when the mechanism that authenticates their API calls or updates their DNS is unavailable.

The control plane distinction

This is the concept that explains why a regional failure becomes a global one.

The data plane serves traffic: Instances run, requests are answered, objects are read. The control plane changes things: Launching an instance, updating a record, scaling a group, issuing a credential.

Well-designed systems keep the data plane working when the control plane is down. But recovery is a control plane operation. If you cannot launch instances, you cannot scale out of a problem. If you cannot update DNS, you cannot fail over. If you cannot issue credentials, nothing new can authenticate.

A recurring pattern

Several large incidents over the years have shared a shape, and the details differ less than you would expect.

The February 2017 S3 outage in us-east-1 is the classic. An engineer running a routine debugging playbook entered a command with a typo, removing far more capacity than intended. S3 was substantially unavailable for about four hours. Because an enormous number of services store assets and configuration in S3, the effects were everywhere - And, memorably, the AWS status dashboard itself could not update its own icons, because they were hosted on S3.

The November 2020 Kinesis outage began with a routine capacity addition that pushed a fleet past an operating system thread limit. Kinesis is a dependency of other AWS services, so the failure propagated into Cognito, CloudWatch and others, and from there into customer applications that had never knowingly used Kinesis at all.

The December 2021 incident involved an automated scaling activity triggering unexpected behaviour in an internal network, which then congested the network between the internal and main AWS networks. The recurring theme: An automated system, behaving as designed, at a scale nobody had tested.

Recognising it

From outside, the signature is breadth plus randomness.

A DNS problem breaks everything uniformly. An ISP problem is geographically clustered. A cloud region failure breaks a strange, arbitrary-looking subset: Some services completely dead, others perfectly fine, with no pattern a user can see - Because the pattern is which cloud region each company chose, which is invisible.

The other tell is the casualty list. When physical devices are involved - Doorbells, cameras, thermostats, robot vacuums - A cloud outage is very likely, because IoT products depend almost entirely on cloud services and have essentially no local fallback. A smart doorbell that will not answer is one of the most reliable indicators that something large is wrong in a data centre somewhere.

What AWS has changed

Cell-based architecture is now standard practice: Services are partitioned into independent cells so a failure affects a fraction of customers rather than all of them. Blast radius reduction is an explicit design goal, and post-incident reports frequently describe adding limits and circuit breakers to automated systems that previously had none.

AWS also moved to reduce global dependencies on us-east-1, though the migration of things built over nearly two decades is slow, and the region's gravitational pull - Default, cheapest, first - Has not changed.

What customers can do

Genuine multi-region is expensive and hard, and most organisations that believe they have it have not tested it under a control plane failure.

The realistic measures: Do not deploy to us-east-1 by default just because it is the default. Understand which of your dependencies have a global control plane there even when your workload does not. Have a static fallback - A maintenance page served from somewhere else entirely is worth more during an outage than a sophisticated failover that requires the control plane to execute.

And accept the honest position, which is that most services will be down when a major cloud region is down. The arithmetic of composite availability is unforgiving: You cannot be more reliable than the layer you sit on without genuinely leaving it, and almost nobody does.