The 2021 Roblox outage

On 28 October 2021, Roblox went offline. It stayed offline for about 73 hours - Three full days - Across a Halloween weekend, at a platform with tens of millions of daily players.

There was no crash, no attack, and no obvious broken component. That is exactly why it took so long.

The setup

Roblox runs its own data centres rather than using a public cloud, with thousands of servers coordinated through HashiCorp Consul, which handles service discovery and health checking. Consul is the layer that lets every service find every other service - A piece of infrastructure that is invisible when it works and total when it does not.

Roblox's own detailed account, published afterwards, describes the mechanism.

What happened

A new streaming feature had been enabled on the Consul cluster, intended to improve performance. Under the specific load pattern of Roblox's cluster, it did the opposite: It caused contention that increased latency in Consul's writes.

Nothing failed. Consul kept running, kept answering, and answered more slowly.

Slower service discovery meant every service that depended on it became slower. Those services then retried, adding load. The additional load slowed Consul further. The system entered a degradation spiral in which everything was running and nothing was usable.

There was a second layer. Consul's underlying storage engine, BoltDB, kept a log that grew during the incident, and its handling of freed pages made writes progressively more expensive as it grew - So the longer the problem continued, the worse it got, independent of the original trigger.

Why diagnosis took so long

This is the general lesson and it applies far beyond Roblox.

When something crashes, you get a signal: A process is gone, a log has a stack trace, a health check fails. The failure announces itself and points at itself.

When something merely becomes slow, there is no signal at all. Every component reports itself as healthy. Every dashboard is green. The system is doing exactly what it was designed to do, just less quickly, and the symptom - Everything is slow - Is the least specific symptom there is.

Worse, the spiral disguises the origin. By the time anyone is looking, hundreds of services are slow and retrying, and every one of them looks like a plausible cause. Distinguishing the thing that started it from the things responding to it is genuinely hard when they all look identical from a monitoring dashboard.

Why recovery took so long

Finding the cause was only part of the 73 hours. Bringing the platform back was the rest.

Restarting the cluster under normal load simply re-triggered the problem: Tens of millions of clients reconnecting created exactly the load pattern that caused the contention in the first place. The team had to rebuild the Consul cluster in an isolated environment, restore it to a known state, and then bring the platform back gradually rather than all at once.

This is the thundering herd problem in its most severe form. A platform with tens of millions of waiting users cannot simply be switched back on, because the act of switching it on generates a load spike larger than its normal peak. Every large recovery is staged for this reason, and staging takes time.

What Roblox changed

The published follow-up describes several changes, and they are a reasonable checklist for anyone running a coordination layer at scale.

Better observability of the infrastructure layer itself. Monitoring had been oriented around services, not around the substrate every service depended on.

Removing the circular dependency. The telemetry system used to diagnose problems partly depended on Consul, which meant the diagnostic tooling degraded along with the thing it was supposed to diagnose.

Staged recovery as a documented procedure rather than something improvised under pressure.

Working with HashiCorp on the underlying storage behaviour that made the problem worsen over time.

Why this one is worth knowing

Most famous outages have dramatic causes: A routing withdrawal, a bad update pushed to millions of machines, a typo that deleted capacity. They are memorable because the cause is a single identifiable event.

This one had no such moment. A feature was enabled for good reasons, it interacted badly with a specific load pattern, and a platform serving tens of millions of people degraded to unusability without anything breaking.

That is the failure mode most organisations are least prepared for, because every instinct and every tool is oriented towards things that stop. Monitoring that only alerts on failures will not see it. Alerting tuned to crashes will not fire. And by the time it is unmistakable, everything is implicated.