Managing by metrics is a reality. One that comes with pitfalls. After 20 years navigating strategy, startups, and education I'm still surprised by how many people are not even aware of these pitfalls. If you're already aware of these ideas, that's awesome. Spread the word. If you're not, let's build a shared understanding.
It’s natural to want to understand things like developer productivity, student test scores, manufacturing scrap rates or whatever it is that may be core to your organization, so you measure them.
As soon as you measure something, even if you don't explicitly set a target, you set the stage for incentives.
If you're not a Charlie Munger disciple then the word “incentives” might not be clear. Here’s what I mean:
People pick up quickly on what is valued, both financially and culturally, within their organization. Whether they’re after variable comp or words of affirmation, they will see what’s measured, understand that’s what is valued, and work to maximize it.
Sounds good, right?
Well, hold on. Incentives can spiral into unintended and sometimes perverse consequences. This the crux of Goodhart's Law, Campbell's Law, and Wildavsky’s Law. The McNamara Fallacy is a close cousin. That's enough name dropping. Don't get hung up on the semantics of calling these "laws." The takeaway is that these are principles that people have encountered and found true across domains and eras. Coupled with what I've seen throughout my career, and it's enough for me to take these seriously.
The ideas mentioned above tell us that creating targets will alter behavior, but the following ideas show us how. Nearly 30 years ago Brian Joiner gave us three categories of expected “patterns of behavior” people will exhibit in a system especially when staring down the barrel of a numerical target tied to their comp or job:
I've added a fourth category to Joiner's list:
4. (Try to) Improve the system, which is all too often not possible or there are too many roadblocks thrown in someone's way to do this
A reason that small YC startup can seem unreasonably effective while a massive organization can't ship is reason #4. In a small startup, alignment and agency to improve the system are common. In Big Co. the system forces too much behavior into the first three options.
Picture an x-axis with 'startup' on the far left and 'big co' on the far right. In my experience, the further along an organization is towards 'big co' on the right, the more likely you are to encounter perverse incentives. I imagine that there's an inflection point when an org grows beyond 30 people.
For startup founders, everything is fine until one day it isn't. However, the seeds of whatever unintended consequences are rearing their head were likely planted long before it becomes apparent.
For a concrete example you can listen to Brian Chesky explain this phenomenon, breaking down how fast moving companies become slow-moving bureaucracies, how it happened at AirBNB, and what he is doing about it.
These four responses can both explain and predict behavior. When you know this, it's not a solution, but you're no longer walking in the dark.
One note I want to harp on: tunnel vision can set in about looking at financial incentives that can cause these behaviors. Meanwhile a team or company's culture can be just as powerful at causing some of these unwanted behaviors even when there is no money on the line for an individual.
I don't think it's particularly useful to try to create the "perfect" metric. Do understand that any measurement and especially any target will have these predictable impacts. As you manage by metrics, sniff out the unintended consequences. Keep your eyes peeled for perverse incentives at play. See and understand, then weigh, and measure the trade-offs that come from people trying to hit the metrics they are managed by.
If you're an executive, you are being paid for your judgment. After seeing the field clearly, you need to judge whether the juice of whatever metrics you are using is worth the squeeze.
Eli Goldratt famously said "tell me how you measure me, and I will tell you how I will behave."
Similiarly, W. Edwards Deming who never wore kid gloves stated, "If management sets quantitative targets and makes people’s jobs depend on meeting them, they will likely meet the targets–even if they have to destroy the enterprise to do it.”
Heed their warnings, then go and make hay.