The CrowdStrike outage

On 19 July 2024, at 04:09 UTC, CrowdStrike published a routine content update for its Falcon security agent on Windows. It was the kind of update that goes out several times a day.

Machines that received it crashed. Not the application - The operating system, into a blue screen, and then into a boot loop where each restart crashed again before reaching a usable desktop.

Microsoft later estimated around 8.5 million Windows devices were affected. Because CrowdStrike Falcon is enterprise software deployed across airlines, hospitals, banks, broadcasters, retailers and government, the effect was immediate and physical: Flights grounded, surgeries postponed, payment terminals dead, television stations off air.

Why an update could crash the kernel

Endpoint detection software has to see what every process on a machine is doing, which means running in kernel mode. Code in kernel mode has no containment: A fault does not crash the program, it crashes the operating system.

The update was a "channel file" - Content rather than code, describing threat behaviours for the already-installed sensor to match against. That distinction mattered enormously for how it was distributed. Code updates went through staged rollout and customer-controlled deployment rings. Content updates were treated as lower risk and pushed rapidly to everyone.

The channel file contained a problem that the sensor's content interpreter did not handle. Executing in kernel mode, that fault took down Windows.

78 minutes, and then days

CrowdStrike identified the problem and reverted the update by 05:27 UTC - 78 minutes. By any normal standard that is a fast response.

It made almost no difference to the recovery timeline, and this is the part of the incident that matters most.

A machine that had already received the update and crashed could not receive the correction, because it could not boot far enough to reach the network. The fix had to be applied locally, per machine: Boot into Safe Mode, navigate to a specific directory, delete a specific file, reboot.

Multiply that by 8.5 million machines. Many were in locked server rooms, or at unstaffed sites, or were the laptops of employees working from home. BitLocker-encrypted machines needed a recovery key - Often stored in a system that was itself on an affected machine.

Why airlines took the longest

The aviation sector was the most visible casualty, and the differences between carriers were instructive.

Most airlines recovered within a day or two. Delta Air Lines took roughly five days and cancelled thousands of flights, well beyond its peers.

The difference was not how many machines were affected. It was crew tracking. Airline operations depend on software that knows where every crew member is, what they are qualified for, and how many hours they have worked under strict legal limits. When that system cannot be relied on, aircraft cannot legally depart even when the aircraft, the passengers and the crew are all physically present.

Delta's crew tracking system could not process the volume of changes the disruption generated, and the schedule came apart faster than it could be rebuilt. This is the same failure mode that produced Southwest Airlines' December 2022 meltdown, which had no IT outage at the start at all - Weather disrupted operations and the crew scheduling software could not re-solve the problem, producing ten days of cancellations.

In both cases the lesson is the same: The fragile system is not the customer-facing one. It is the operational optimiser nobody outside the company has heard of.

What changed

Staged rollout for everything. The core failure was treating content updates as low-risk. CrowdStrike committed to staged deployment, canary testing and customer control over content update timing - The same protections code updates already had.

Moving out of the kernel. Microsoft convened work on giving security vendors the visibility they need without kernel-mode execution, so that a vendor fault degrades a product rather than crashing the machine.

Remote recovery as a design requirement. Organisations discovered how many machines they could not reach without physical access. Out-of-band management stopped being a server-room luxury.

Recovery key storage. A lot of organisations learned that their BitLocker keys were stored somewhere that was also down.

What it was not

Two things worth saying clearly, because both were widely claimed at the time.

It was not a cyberattack. It was a software defect in an update.

And it was not really about security software. It was about update distribution. Any software with automatic updates, broad deployment and privileged execution has the same shape of risk - The reason this one was so severe is that all three were maximised at once.

The number that should be remembered is not 8.5 million. It is 78 minutes to fix, days to recover: The gap between those two is where the damage lived, and closing it is an engineering problem about distribution rather than about code quality.