Years ago, I worked a short while for a very dysfunctional organization. I was young and in the labor market for less than two years. In less than three months, the key performance indicators (KPIs) to which my performance would be judged grew from zero to over twenty-five. With no experience nor the authority to highlight the problems that come with working on 25 KPIs — at the same time — I quit the job prematurily.
According to professor in history, Jerry Z. Muller, I was a victim of metric fixation. In his book “The Tyranny of Metrics“, he wrote a concise piece on the havoc that metrics can cause. Many of you reading this are active in marketing. So if you need a reason to question what you do, let this line be your trigger to order this book: “Trends in contemporary advertising illustrate the phenomenon of ‘measurability bias‘, the tendency to prefer options, simply because they can be measured.“
Excessive measurement, inadequate measurement, not metrics, but metric fixation
In 1986, Management guru Tom Peters — although we share names we are not related — coined the phrase of “what gets measured, gets done.” In modern business, this oftentimes results in performance being equated with standardized measures. In essence it means that:
- Judgment can be replaced with standardized criteria;
- By making these criteria public, we ensure that institutions carry out their purposes;
- Bad results should be penalized, good results should be punished.
Of course, the #1 counterargument is Goodhart’s Law: “Any measure used for control is unriable.” Because any metric that becomes a target will be gamed. The sales manager, judged on conversion rate, will only work on the leads with the highest potential. The surgeon, judged on mortality rates, will only accept patients that have a reasonable chance of survival to maintain his streak. The CEO, judged on quarterly profits will avoid long-term investments.
Complicatedness: more reporting, less doing
How did we get here? As managers became trained for managing, instead of acquiring deep knowledge of the company and the market they worked in, they rely on standardized metrics for decision-making. Sadly, this often results in the ever-expansion of rules, procedures, coordination bodies, meetings and reports. It’s what Muller dubbed as rule cascades: rigidity and compliancy regarding metrics slows down an institution functioning and reduces its efficiency.
Citing the usual suspects, the book touches on the philosophical critique of metric fixation through Hayek’s pretense of knowledge, Knightian Uncertainty and Marx’ deskilling. But the best quote Muller used comes from Isaiah Berlin. Judgment is a sort of skill at grasping the unique particularities of a situation and it entails a talent for synthesis rather than analysis: “a capacity for taking in the total pattern of a human situation, of the way in which things hang together.”
For over two hundred years, capitalism created the context in which creativity can flourish. Through the market, it penalizes unwanted results and incentivizes innovation. However, metric fixation is the exact opposite. If Microsoft would just be in the business of selling operating systems, and the team would be judged on sales numbers only, we would still be working on Windows 95 or it would be outcompeted by Apple or Google. Muller makes a brilliant analogy with the Soviet production system: bureaucrats would set targets and factory managers would build shitty cars.
Spreadsheets, AI, and the illusion of the depth of analysis
But it’s not only a change of managerial culture, it’s also the introduction of IT on the work floor. In 1980, journalist Stephen Levy wrote that the spreadsheet is not only a tool: it is a worldview. Since it focuses on numbers, it only emphasizes the aspects that are easily embodied in numbers. Intangible factors aren’t so easily quantified. They create the illusion of depth of analysis.
I would personally like to build on Levy’s work to complete the argument. With the increasing capacity to capture, store and process data, his remarks have become even more urgent.
The omnipresence of data has created the illusion that anything can be quantified. And to a certain degree, it is. If a marketing manager in retail wanted to know how many of his customers have an affinity for consumer electronics and how that segment has been growing over the past months, that would have been impossible roughly 20 years ago. Now, data scientists can simply run a clustering algorithm to answer that question. Even better: CRMs offer this feature out of the box.
However, even algorithms can be fooled: playing around with the data that is fed to the algorithm, or simply widening confidence intervals can artificially (no pun intended) improve results. In brief: use data with care.
Want to know more?
The experienced reader will finish Muller’s book, “The Tyranny of Metrics” in under an hour. It’s concise, to the point, not preachy and properly structured. The first half of chapters are spent on the argument and the final half of chapters is spent on case studies in fields such as academics, medicine, the military and business.
Addendum. To my surprise, the company that I worked for years ago still exists. I don’t know if the 25 KPIs are still around and I am not certain that the guy who took my place spends days on gathering the required metrics. What I do know, is the following: on 28 August, together with my colleague Senne Vermassen, I will elaborate on our experience as data consultants: what are returning issues and how can they be offset, both in a context of business intelligence and artificial intelligence. You can register here.