Data: Use, With Caution

In the modern business environment everyone wants to be data-driven. You can hear the envy in a conference room when a good graph hits the audience right in their data-loving brains. Data is good. Data rules all.

What you see less are the glaring imperfections in most data. The show must go on - fine-grained analysis of the data used for strategic decisions holds up things up. Deep dives into data require ability to get that data, interpret that data, and understand that data. This creates a power imbalance where the data-creator is much better positioned to tell the story they want to tell; the data-consumer is hard-pressed to disagree with that story in real-time. Alas, decisions happen in conference rooms, data analysis happens at a desk.

These two realities are Saasy’s Axioms of Data:

• Data wins arguments. Imperfect data beats no data.

• All data is imperfect.

These realities have 3 major implications for how you should approach getting things done:

• Show up with data.

• Use data as a compass, not a map.

• Beware the abuse of data.

Show Up With Data

Most of the time some data is better than no data. You should strive to be data-driven. Our collective admiration of data comes from a reasonable instinct.

In a world where data is useful and people view it as a sign of mature, rational, and thorough work, you should show up with data. No matter your role, you will get more of the things you want and do a better job if you show up with data.

As many have pointed out, you can measure anything to get data for your case:

• Our team cycle time decreased by 20% last half on repos that were converted to using React and remained constant on ones that weren’t. We should convert the remainder of our repos to React.

• We left 290 style comments on pull requests in the last quarter. If each of those take 2 minutes to resolve on average, replacing that mechanism with an automatic enforced style-guide will save us a full day per quarter.

• Only 5% of IT requests take longer than 1 week, but those that do take an average 3 months. Those tickets also are responsible for the majority of internal-NPS for the IT team below a 4. We need to implement a process to get rid of these outliers.

All of these arguments are more compelling because they are data-supported. The data is imperfect, but it’s useful. It gives an overall, high-level assessment of the situation. It also allows for measurement of results, using the same data after the change.

If you’re not using data regularly you’re probably leaving a lot of impact on the floor, no matter if you’re a new IC right out of college or a seasoned executive. Show up with data.

Data Is A Compass, Not A Map

Data exists to help you understand, at a high level, what would be useful to do and how effective your actions have been. However, the more granular the decisions are, the less useful data becomes. Data’s utility is strongest when it tells a clear story, not when it whispers details. So you should use data more when the data is very clear and makes sense, and you should rely on it less when it’s more nuanced.

For example, employee satisfaction surveys provide useful data when the signal is clear and blunt: people are unhappy or people are happy. The surveys are less useful when it comes to specific feedback. A sole comment about the espresso machine might say that it’s too loud. If you then move the machine to the hallway, you might find that 90% of your team hates the new location because it’s farther away. You used the data like a set of instructions, not a broad picture.

Another example: I had a feeling awhile back that remote candidates were turning down our offers more than local candidates. I pulled the data and the story was clear - remote candidates actually closed at a significantly higher rate than local candidates. The data was imperfect, but it gave no credence to my assumption, and I had other ways I can improve recruiting so I moved on.

In the compass analogy: I thought I was heading South, but my compass told me I was heading North. I might have actually been going NorthWest, but I’m happy enough to know that I’m not going South.

You might think that the advocacy to use data when it’s bold and intuitive is a catch-22, that data is most useful when it’s counter to your intuition and changes your behavior. However, I disagree - data is most often important in two cases: 1) it provides direction when you have no prior intuition at all and 2) it sets you up to measure the success of your work from the get-go.

Beware The Abuse Of Data

Because data is powerful and imperfect, and because decisions are using presented data and not data-analysis, there are many who would wield its power without prudence. You can see these people show up with hand-crafted data any time they want to get their way. And because it can seem petty and defensive to question that data - after all, you’d be the amateur who didn’t show up with data - those people can often get their way, even when the data isn’t bold and the conclusions are debatable.

There’s no one-size-fits-all solution to these situations. High level though, I’d recommend the following:

• Work to ensure your company has a healthy data culture

• Show up with data for the things you really care about

First, a healthy data culture uses data but understands its limitations. A healthy data culture is a humble data culture. If data is moving you towards good decisions with clear signals and helps you measure success, that’s great. If data is wielded as a weapon or is paraded around as a sign of greatness, watch out.

For an example of a bad data culture, your no-assholes-allowed company might put up with a salesperson because “they put up crazy numbers”. Indeed, that person might wear their quarterly numbers like a badge that lets them behave on a different plane of existence. However, are you tracking how often they are making concessions with negative downstream impacts? Are you tracking how much time they take from the rest of the organization to support their sales? This is to say nothing of the culture impact of that person.

The nuance here is important. Their quarterly numbers tell a clear story - they sell a lot of product. But the question was more nuanced: should we put up with their behavior? In reality, you likely have almost no data that quantifies the totality of their impact on the company. Their sales are just one number in a complex ecosystem.

In this example, you used imperfect data to let you do something you wouldn’t otherwise. You made data a king and ended up being ruled by a tyrant.

Second, if you really care about something, considering showing up with data. Otherwise, you’ll be at a disadvantage if someone else does.

Summary

Data, like any power, should be wielded with respect, humility, and caution. However, unlike most power, you should be wielding it regularly.