Incident metrics that matter

Incident metrics exist to answer two questions: Are we getting better, and where should we spend effort. Most teams track the one metric that answers neither.

MTTR and its decomposition

Mean time to recovery is the average elapsed time from an incident starting to service being restored. It is the headline number, and on its own it tells you nothing actionable - Because it is a sum of four quite different phases.

The reason to split them is that teams consistently assume the problem is repair and it usually is not. A team with a four-hour MTTR that turns out to be three hours of detection has a monitoring problem, not an engineering one - And no amount of faster deploys will fix it.

Detection is also the cheapest phase to improve. A better health check costs an afternoon; halving repair time can be a quarter of work.

The mean is the wrong statistic

Incident durations are heavily skewed: A large number of short events and a small number of very long ones. A mean over that distribution is dominated by the tail.

Twenty incidents of five minutes and one of twelve hours gives a mean of about 39 minutes - A number that describes none of the twenty-one incidents. It is not the typical case and it is not the bad case.

Use the median for the typical incident and the 95th percentile for the bad one. Track both. A median that is stable while p95 grows means your worst incidents are getting worse, which is exactly the thing a mean would hide.

And count incidents by severity rather than aggregating everything. A metric that mixes a two-minute degradation with a four-hour outage is measuring two different phenomena.

MTBF is not useful here

Mean time between failures comes from hardware reliability, where components wear out on a distribution you can characterise and where the next failure is roughly independent of the last.

Software does not wear out. Failures are caused by changes, load and dependencies, which makes them correlated rather than independent: Incidents cluster after a big release, during a migration, or when a system is near capacity. A mean interval over that is not predictive of anything.

The useful replacement is change failure rate - The proportion of deploys that cause a degradation. That connects failures to their actual cause and is directly actionable: If a quarter of your deploys break something, the problem is the release process.

Error budgets

The most behaviour-changing metric on this page, because it turns reliability into a quantity you can spend.

If your objective is 99.9% availability, your error budget is 0.1% - About 43 minutes a month. Downtime consumes it. While budget remains, you ship features. When it is exhausted, you stop shipping and spend the time on reliability.

What makes this work is that it removes an argument. Instead of a recurring negotiation between product and engineering about whether to prioritise reliability, there is a number, and the number decides. It also legitimises spending the budget: If you have used none of it, you are being too conservative and could be moving faster.

Metrics that mislead

Number of incidents, on its own. A team that declares an incident for every degradation looks worse than one that only declares for total outages, while being better run. The definition drives the number.

Uptime percentage for a short period. "100% this month" is true of most services most months.

Anything used to compare teams. Metrics used for evaluation get optimised, and incident metrics are unusually easy to optimise the wrong way: Declare fewer incidents, close them earlier, classify them lower. Every one of those improves the number and makes the organisation worse.

A minimal set

For most teams, five numbers are enough:

1. Median and p95 incident duration, by severity.
2. Time to detect, tracked separately - Usually where the biggest win is.
3. Change failure rate.
4. Error budget consumed this period.
5. Percentage of incidents with a written post-mortem - A process metric, and the one that most predicts whether the others improve.

Review them monthly, look for trends rather than single points, and never use them to rank people.