You Have Too Many Metrics

What's in a measure?

Metrics can be incredibly powerful. But you have too many of them.

Let’s talk about how and when to use metrics.

The Golden Rule

The golden rule of metrics is this: any metric you maintain should directly drive action if outside expected bounds.

The reason this is an important rule is:

• Metrics are expensive to set up, so if you don’t use them, you’re wasting effort

• Metrics are expensive to maintain, so if you don’t use them, you’re wasting effort

• A metric that doesn’t drive action is a waste of time

A direct corollary - because the cost of setting up, maintaining, and actioning on metrics is high, you shouldn’t have that many metrics.

Let’s talk about setting up metrics and how to use them, as well as how to not use them.

Using Metrics

Setup

Setting up a metric takes time. You have to:

• Get the data

• Hook it up to a UI

• Monitor it to make sure it’s right and doesn’t fluctuate

Depending on your company and the metric, this could be a couple hours of work or a couple months of work via multiple tickets. Some tools give certain metrics by default, but that’s also not free (you’re paying for it).

The takeaway is that setting up metrics cost time and money.

Using Metrics

Good metrics:

• Have a regular review

• Have an expectation on the behavior of the metric

• Prioritize action if the metric isn’t what’s expected

Regular review

Regular review of metrics is critical, because any metric you don’t regularly review will eventually become inaccurate.

Many people come back to a metric a year after setting it up and realize it doesn’t look quite right. Upon investigation they realize a week ago someone changed the definition in middleware and it started double counting. This kind of thing happens all the time. The most pernicious version of this happens when the metric looks like things are working well, while actually it’s overreporting and things are having issues.

Regular review scrutinizes the metric value as well as matches it against other information to ensure it seems right. As a general milepost, expect to find something broken in a metric about once a year, needing repair.

Expectations

Metrics without expectations are just gossip prompts.

Some metrics only matter if they move > 10%. Some matter if they’re .1% off target.

State explicitly what your expectation is for a metric and the conditions of action in either direction. Otherwise, metric review will become an exercise in taking a really long time to realize people don’t know what matters.

Prioritization of action

If a metric is outside the bounds of certain expectations, you must take action. This is the biggest cost to maintaining metrics - you actually have to do something if the metrics indicate something you’ve said you don’t want to happen.

Too many people have 25 metrics in their dashboard and don’t do a damn thing about anything. They just go back to the same backlog they were working on and put away the shiny metrics for powerpoints when they need to convince someone of something.

Metrics should have expectations. If those expectations are not being met, action must be taken.

Examples

Let’s talk through a couple examples.

The Ambitious PM

An ambitious PM has a bunch of metrics in a dashboard. The PM uses them for things like: showing how well their team is doing, and asking for a raise, and asking for more resources.

The problem is that they only review those metrics in preparation for those activities. They actually have no regular review, and no owner, and no clear expectation of what it should or shouldn’t be.

Then one day the PM’s boss wakes up and says wait, this isn’t quite right. And that’s when you find out not only is the usage data wrong, but even if it was right, the whole concept of appropriate growth was not aligned upon. So in fact, all of these metrics were, for their entire lifetime, worse than useless.

Instead of showing the micro-cohort CSAT for 7 different customer profiles across 3 products, their manager should have asked to see one or two metrics, and should have scrutinized them regularly, with clear expectations on growth (not just that up and to the right is good).

The Cost Review

It’s common practice to regularly review infrastructure cost at companies with the finance team. Oftentimes these processes have two things correct: they review metrics and each piece of infrastructure has an owner. The common failure, however, is to not discuss what good and bad actually looks like.

When do we actually have to do something about a cost changing in a certain way?

This question is often never asked, and so people debate minor cost fluctuations and lack clarity on if any changes are needed. The simple answer is to set up a working agreement: we’ll review the cost to make sure it all makes sense, but we’re only actioning if it’s more than 2% above the quarterly forecast or 5% over the yearly forecast. And, if it’s above that, you must take effort to reduce it.

With this agreement, you gain efficiency when things don’t require action, and direct clarity when things do.

Summary

Having a few great metrics is much better than having a bunch of trash metrics. Metrics require intense focus to keep accurate and to have high integrity. Metrics require expectations and action to be worth spending all of the time it takes to make them accurate and have high integrity.

If someone says “I got metrics” ask them the last time they did something because of a metric. If they don’t immediately know, they don’t have metrics, they have a dashboard of graphs that they use to persuade people.

Share to Twitter , Hacker News , LinkedIn