“Quantitative measures of performance are tools, and are undoubtedly useful. But research indicates that indiscriminate use and undue confidence and reliance in them result from insufficient knowledge of the full effects and consequences. Judicious use of a tool requires awareness of possible side effects and reactions. Otherwise, indiscriminate use may result in side effects and reactions outweighing the benefits, as was the case when penicillin was first hailed as a wonder drug. The cure is sometimes worse than the disease.”

The passage above is an excerpt from a 1956 paper by American academic V.F. Ridgway titled “Dysfunctional consequences of performance measurements”. It doesn’t specifically relate to brand measurement, but it perfectly sums up my experience of how brand metrics tend to be used and – just as frequently – abused. Being able to navigate the vast minefield of available brand measures is a vital skill for brand strategists, but there’s precious little in the way of practical guidance for how to do so. This paper sums up some of what I’ve learned in 22-plus years of wading through quantitative brand research to develop confident brand strategy. What follows helps me to stay out of trouble.

A (very brief) history of brand measurement

1956 was a landmark year in brand measurement (and not just because of V.F. Ridgway). The concept of ‘brand loyalty’ was coined by Ross Cunningham in his 1956 Harvard Business Review article, “Brand Loyalty – what, where how much?”. His research concluded that consumers are brand loyal in more than 90% of household goods purchases. In a follow-up HBR article two years later, Pierre Martineau found that despite retail stores offering the same goods, at the same prices with equally good service, customers showed partiality because of their different ‘personalities’.

1956 also saw the popularisation of another revelatory marketing concept – segmentation ­– by Wendell R. Smith. The task of the marketing department at the time was to encourage consumers with diverse needs to converge on the specific characteristics of the product its company happened to sell: marketers were employed “to bend demand to the will of supply”. Smith’s revolutionary idea was that divergent needs could actually be seen as an opportunity rather than an inconvenience: bending supply to the will of demand. Smith proposed that marketers should break down heterogenous markets into smaller sets of homogenous markets, or ‘segments’. Smith concluded that the ability of businesses to market their products this way depended on “a flow of market information that can be provided by market research”. Smith’s theory was developed eight years later by Daniel Yankelovich, who proposed that markets could be segmented by buying behaviour, motivation, values and aesthetic preferences, as well as by more popular socio-demographic variables.

Brand loyalty, brand personality and segmentation are foundational concepts in brand measurement. Together with the brand funnel, they are the antecedents of many of the metrics we apply to brands today. Although technological advances mean we can now collect data at a faster rate and analyse it in more sophisticated ways, surprisingly little has changed in terms of the types of measure we use to guide decision-making. Given marketers’ current obsession with Net Promoter Scores (NPS), it’s very possible that today’s marketers actually consider a smaller set of measures than their mid-century predecessors.

And focusing on too few measures was a huge area of concern for V.F. Ridgway.

He used the example of an employment agency to illustrate the danger of focusing on a single performance metric. The agency’s task was twofold: to help workers find employment and to help employers fill vacant roles. Initially, the agency incentivised its staff based on the numbers of workers interviewed. In response, staff devoted themselves to finding as many candidates as possible to interview, but with the (unforeseen) consequence that they didn’t devote adequate time to finding jobs for those candidates. Ridgway concluded (from this and other examples) that focusing on a single measure always results in a dysfunctional outcome because success in any activity is rarely – if ever – achieved by focusing effort in a single direction.

This phenomenon is more commonly known amongst measurement geeks as Goodhart’s Law: named after economist Charles Goodhart who observed that once a measure becomes a target, it ceases to be a good measure. Any metric is bound to be abused if people are incentivised to improve it. For example, the 2016 Wells Fargo scandal has its roots in a seemingly innocent internal initiative called “Going for Gr-Eight”, which incentivised branch staff to cross-sell at least eight products to customers. Under pressure from their supervisors, staff fraudulently opened as many as 2 million accounts and credit cards without customers’ consent. The scandal ended up costing Wells Fargo nearly $3 billion.

This is an extreme example but it’s not exceptional.

Any marketing team (or business) that focuses on a single measure of brand health should be worried, including those that consider NPS (or brand value, or ‘Brand Power’) to be a silver bullet for measuring brand health.

Rule #1: There’s no such thing as a silver bullet: only silver buckshot.

Besides their simplicity, part of the allure of these single measures of brand performance is the argument that they correlate strongly with financial success:

  • According to Bain, Net Promoter Scores explain roughly 20% to 60% of the variation in organic growth rates among competitors in most industries
  • Between April 2006 and April 2021Kantar’s BrandZ portfolios of the world’s most “powerful” brands has outperformed the S&P500 by a factor greater than two
  • McKinsey’s portfolio of “powerful” brands has outperformed the MSCI World Index by 96% since 1999.

If you spend enough time looking at quantitative data across enough categories, the same pattern emerges: the biggest brands typically have better scores across ALL brand measures: awareness, salience, consideration, loyalty, advocacy, personality, image. This is why some researchers ‘rebase’ their data to identify where specific brands over- or under-perform relative to their market share. This means pretty much any brand measure – single or composite – will correlate with financial success.

Rule #2: Don’t be impressed by brand metrics that correlate with financial performance.

The above rule means you should be wary of anybody who claims that they have a ‘scientific’ or ‘proven’ measure that reliably and invariably predicts financial performance. I frequently work with global brand owners who have built their marketing approach around ‘scientific’ principles or laws. Usually, these are based on observed correlations between variables across markets, such as the relationship between share of voice and market share. The most popular of these relationships is between market penetration (which measures the percentage of customers in a market who purchase a particular brand within a given period) and frequency of purchase. Typically, brands with higher penetration are also bought more often. The people at the Ehrenberg-Bass Institute call this the ‘double jeopardy law’ because smaller brands get hit twice: their sales are lower because they have fewer buyers who buy the brand less often. The chart below is a reproduction of one of the many examples of the ‘double jeopardy law’ presented by the Institute:

The chart shows that there is, indeed, a correlation between penetration (on the x-axis) and loyalty (represented by repeat visits on the y-axis). I’ve seen plenty of data like this and I agree that in general loyalty and penetration are correlated. What it absolutely doesn’t demonstrate is that there is a scientific law at work, like the law of gravity. M&S, Lidl, Iceland, Aldi and Waitrose have similar penetration, but vastly different repeat visit rates. Waitrose customers are twice as loyal as M&S customers, which is why Waitrose has a market share around 4.9%, compared to M&S’ 3.2%. Not only is it disingenuous to dress up a correlation as a “law”, it’s also dangerous.

The ‘double jeopardy law’ suggests the only “scientific” way for M&S to grow its market share is to increase penetration, which means the marketing team should concentrate on stealing customers away from other brands. But Waitrose’s example shows that it’s possible to punch above your weight in this market by delivering a great experience that keeps customers coming back. Stealing share is expensive, whereas often small, thoughtful tweaks in a brand experience can create significant impact.

When you see a chart like the one above, don’t just look at the thick red line. Look for the outliers and try to understand what they are doing differently. Goodhart’s law exists because seemingly simple relationships between two variables are often more complex and nuanced than charts like the one above suggest, so there are usually plenty of ways to subvert the relationship.

It’s a brand strategist’s job to question, challenge and even subvert established trends in data: great brands should aspire to be outliers and the entire premise of behaviour-change models and transformative innovation is to break established relationships between how we act today and how we will act tomorrow. The happiness of future generations relies on our ability to do so.

Rule #3: There’s no such thing as a scientific law in marketing.

V. F. Ridgway was also sceptical about multiple measurements, although it’s not entirely clear why. His article returns to the case of the employment agency. Having realised the error of its single-metric approach, staff were encouraged to focus on a wider set of measures: not just quantity of interviews, but also successful placements and referrals (and the ratios between them). His principal issue with this situation is that it is left to the individual to decide which of these measures is most critical to overall performance.

Personally, I think this is an advantage.

Two years prior to V.F. Ridgway’s article, Peter Drucker had listed the following as important performance measures: market standing, innovation, productivity, physical and financial resources, profitability, management performance and development, worker performance and attitude, and public responsibility. Nearly seventy years later, Drucker’s list looks remarkably relevant. Drucker also stressed the importance of a “balanced” approach to objective-setting: his list contains variables that need to be traded-off against one another. It forces managers to consider whether they believe worker attitude or management performance are more important in improving financial performance. It forces them to think about how they intend to reconcile profitability with public responsibility. These aren’t pesky details! THIS IS STRATEGY!

This is one reason why I typically advise clients to avoid off-the-peg brand equity surveys. They seem impressive at first, typically because they correlate strongly with financial performance (see Rule #2) and because they allow you to benchmark your brand against a database of thousands. But the price for this is that you have to measure your brand using a set of measures that matter to everybody else, rather than to you. These surveys are structured based on a researcher’s idea of what a ‘great’ brand looks like, rather than your own. Fatally, they encourage marketers to look at the world through exactly the same lens. Great brands are supposed to have a unique point of view on the world: this should be reflected in the way they measure greatness and how close (or far) they are from it.

What you measure should reflect what you believe to be important, not what your research company believes to be important.

What are the decisions you want to be able to make based on the data you gather?

What are the challenges you need to address?

What are the opportunities you want to validate?

Are these REALLY going to be captured in an off-the-peg survey?

Rule #4: Tailor what you measure around the decisions you want to make.

Picking a balanced set of measures

There’s an old Indian parable about seven blind men and an elephant, which I believe is well-known in brand measurement circles. It relates to the difficulty in establishing truth when information is limited or difficult to obtain. Seven blind men decide that they are going to touch an elephant to learn what it is like. Each touches a different part of the elephant and when they compare notes they find that they are in complete disagreement. The man who touched the elephant’s leg believes an elephant is like a tree. The man who tugged the elephant’s tail considers an elephant to be like a rope. The man who felt the elephant’s side declares that an elephant is very much like a wall. And so on.

The parable is further reason to be suspicious of single-metric approaches to measuring brand health or equity. It also illustrates the importance of being able to step back from the data and think about how all the variables you’re looking at hang together. I really like Peter Drucker’s list. It contains a mix of lots of different types of measure: financial and non-financial, leading and lagging, process and people, internal and external, behavioural and attitudinal. But great brand measurement isn’t just about compiling a list of variables that matter to you: it involves thinking about the relationships between those variables. The Balanced Scorecard is a decent starting point for thinking about how to develop a hierarchy of brand measures:

  • Financial measures: that capture your beliefs about the contribution your brand makes to the top and bottom line (and access to capital if you’re feeling clever)
  • Strategic measures: that underlie these financial measures, such as market share, penetration, NPS, salience, power
  • Customer measures: that reflect what the customer wants to achieve, for example: convenience, time-saving, recognition
  • Image measures: that capture the associations you believe will drive improvement in these strategic measures (i.e. what will make you different and desirable in the minds of the audiences your brand interacts with)
  • Brand touchpoint measures: that capture the key interactions in your brand experience that you believe will have the biggest impact in improving your brand’s relationship with these audiences
  • Brand processes metrics: that capture the efficiency and effectiveness of your brand management processes and governance approach
  • Brand growth measures: that capture your pipeline of growth initiatives, insights and ideas

I should stress that never in my life have I seen such a comprehensive brand measurement system… Or anything that even comes close. But we’re not talking about a dashboard of 200 variables here. The important thing is to avoid the temptation to include variables in your brand measurement system simply because they are the same ones you’ve always tracked, or because they are the same variables that everyone in your sector or category measures. I frequently encounter brand measurement presentations (health trackers, equity studies, segmentation reports, shopper insight studies) that run as high as 300 slides. They are mind-numbingly dull and generally contain less insight than well-considered documents less than a tenth of their size.

Most of the clients I work with have shedloads of data, but the problem is that it’s often in the wrong place. Usually, the emphasis is on ‘hard’ behavioural data: often at the expense of ‘soft’ brand image and attitudinal data. This is a particular issue in data-rich categories such as FMCG, where there are reams of potential measures to focus on: AWOP, FOP, penetration, frequency, volume, value, loyalty, etc. etc. The temptation is to include them all in your strategy.

Resist it!

Ideally, you should be able to map out a brand strategy on a page that starts with your financial ambition and then works through the hierarchy down to the underlying growth initiatives that will help you to realise that ambition, with clearly signposted relationships between variables at each level in the hierarchy. Here’s an illustrative (incomplete) example I made earlier:

Rule #5: Map your strategy and the relationships between your metrics.

You can’t drive a car looking in the rear view mirror

I used to say this a lot. In a brand measurement context, it’s a stock phrase we rattle out when we want to point out the folly of obsessing over backward-looking data (which is, basically, all data that you’re not looking at in real-time). But the truth is that rear view mirrors exist for a good reason. It’s helpful to look backwards as well as forwards. And I’ve noticed that marketers seem to be particularly resistant to contemplating the past, particularly when it comes to strategic planning. The emphasis here is almost always on the future and rarely (if ever) is the following question asked:

How much of the stuff we predicted and planned a year ago actually happened?

I’ve spent the past fifteen(ish) years on management teams and NEVER ONCE asked this question. Partly, it’s because I’m more excited by future possibilities than what’s already done. But the main reason is because I know that many of last year’s predictions never happened and few of last year’s plans worked out. Brand consultancies aren’t the only culprits. Often when I’m asked to join in clients’ strategic planning processes, the previous plan is rarely shared with me. Understandably, as people we’re reluctant to admit how terrible we are at predicting things (even with the wealth of data at our disposal) and how few of our plans come to fruition. And on the occasions where the previous plan is shared with me, usually this is for “context” rather than for serious scrutiny.

I feel a great deal of shame in admitting all of this, because I know it’s deeply wrong and I’m part of the problem. I think a large part of the issue comes with the formulation of strategy in the first place: understandably, we invest in it. Not just financially but emotionally. We paint a vivid picture of a desirable future. We desperately want it to materialise. So it’s crushing to admit when it doesn’t. But if we fail to acknowledge past failures, we are doomed to repeat them.

On the surface, the process of developing a system for measuring your brand sounds like a dry, dispassionate process. But the better you are at it, the more emotional it gets. A strategy map lays your thoughts and beliefs out there for all to see. And then adding KPIs (or even CPIs) to the strategy gives you nowhere to hide: like it or not, you’re going to find out where and how you’re underperforming. This is something as a community we could be better at embracing: understanding what worked last year gives us one less thing to fix this year. And finding out what’s not working (as soon as possible) means we know where to focus future efforts. And for the record, I’ve seen this work positively, too: success sometimes flourishes in unexpected places and it’s also important to learn from these positive failures of prediction.

Rule #6: Be prepared to challenge your own beliefs… And find out you were wrong!

Don’t be the drunk under the streetlight

You’ve probably heard this one:

A policeman on night patrol sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, “no”, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, “this is where the light is”.

This is a significant issue in brand measurement. There are plenty of attributes relevant to brands that we can measure easily enough: awareness, advocacy, familiarity, usage. But there are also plenty of unmeasurables: for example, their propensity to inspire creativity, enterprise and adaptability within an organisation. The problem for brand measurement is that it’s easy to set out seeking to measure what’s important, simply to end up attaching importance only to what can be readily measured. Amongst measurement geeks this is known as the McNamara fallacy, coined by Daniel Yankelovich (mentioned earlier) after Robert McNamara, who was the Defence Secretary who presided over the USA’s disastrous war in Vietnam.

McNamara had a love of measurement and statistics, which he had applied to key decisions about how much his country should invest in the war. With the benefit of hindsight, many of these data-led decisions proved to be wrong. McNamara’s legacy is a sober warning to anybody who wants to use measurement to make decisions about brands.

It’s important to be realistic about the limits of what measurement can achieve. Being clear about all the aspects of a brand you can’t measure – and acknowledging the importance of these – is vital to the responsible and prudent application of a brand measurement system. Not everything that can be counted counts. And not everything that counts can be counted.

Rule #7: List the important unmeasurables and make sure these are included in your strategy.

For anybody who works with quantitative segmentation, there are two further watchouts. The first relates to how segments are often presented as pen portraits in PowerPoint presentations. What follows is a simplified example of something I see all the time, based on real data. Here’s a quick pen portrait I put together based on some old data on attitudes and behaviours to money management:

The bar chart to the right of the pen portrait contains indexed data, which compares the segment’s attitudes to the total sample average. An index over 100 means the segment is more likely than the average person to agree with a statement. An index over 200 means that the segment is more than twice as likely as the average person to agree with a statement. And I’ve based the pen portrait on the highest indices for this segment. The picture it paints is of someone who loves spending money on new things but absolves themselves of any responsibility for managing their own finances. Although this is a simplified example, I see this sort of thing a lot: pen portraits based purely on indexed data.

Here’s a more complete picture of the data I based this on:

What the data shows is that although the indices are high, fewer than half of the people in this segment agreed with any of the above statements, which means that the pen portrait is monstrously misleading. Here’s the rest of the data:

The data table shows that the statements that people in this segment are most likely to agree with tend to be broadly in line with the average person’s views. As a rule of thumb, I generally look for an index around 125 or over (or 80 and below) as a sign that the segment’s data is materially different from the sample average. People in this segment actually consider themselves to be very good at managing their money: they plan their weekly shop and look out for special offers. But then again, so does the average person.

I’ve highlighted in green the statements I consider to be genuinely representative of what makes this segment interesting. These are the statements that a clear majority of people in the segment agree with and have an index higher than 125. They reveal that this segment is largely driven by convenience: they are busy people who tend to focus on the here and now and are willing to pay more for things that make life easier. They also like to use cash. Imagine you worked in the marketing department of a bank. The (misleading) pen portrait suggests this is a group of people who may want a credit card or personal loan. A closer look at the data reveals people who may benefit from a simple way to save for the long-term.

If you’re ever presented with a pen portrait (or any data) that shows indices without the underlying averages, then always ask to see the data in spreadsheet form. It’s your job to understand the data you use to base your strategy on.

Rule #8: Always ask to see the spreadsheet.

The previous segmentation watchout should make you suspicious of indexed data. But you should be suspicious about averaged data, too. There’s a beautiful, life-affirming article about the perils of averages, which was written in 2013 by Stephen Jay Gould, an evolutionary biologist who taught at Harvard University. It’s an account of Gould’s experience of being diagnosed with a particularly rare and deadly form of cancer, with a median mortality of only eight months after discovery. Confronted with the same diagnosis, many of us would draw the reasonable conclusion that we probably have eight months to live.

But Stephen Jay Gould had a firm grasp of statistics, so he understood that if the median life expectancy was 8 months, then half of people with the same cancer must have lived for longer than 8 months after diagnosis. Rather than obsessing over a single, abstract average, he looked at the underlying data to see how it was distributed. He found a long tail that extended for years after the eight month median. So, he did everything he could to make sure he ended up in that tail. And he lived for another two decades.

This is an important thing to remember when you’re looking at averages in segmentations: it’s possible that the variation of attitudes and behaviours within a segment is greater than the variation between segments. The closer an index is to 100 the more likely this gets, which is why I typically don’t pay much attention to anything that indexes between 80 and 125. This is also worth bearing in mind next time anybody tries to sell you on attitudinal differences between Baby Boomers, Gen X, Millennials and Gen Z. A lot of this stuff is nonsense: most of the attitudinal segmentations I’ve seen show very little correlation with age. I find this tremendously reassuring: ageing won’t turn me into a fossil fuel-loving bigot.

Rule #9: Every average hides a distribution.

My final rule is stolen. It’s the ‘Golden Rule’ developed by Tim Harford in his absolutely brilliant book, How to Make the World Add Up. The book is an antidote to the hard-nosed scepticism that leads people to assume all data and statistics are bullshit. And his Golden Rule is very simple:

Rule #10: Be curious.

Brand strategy inevitably involves wrestling with data. It’s a fantastic sport. Hidden in that data are opportunities and answers. Sometimes there are insights and epiphanies. Often there are also pitfalls and red herrings. Curiosity is the best tool you have to uncover these. If there’s a data point you don’t understand, then ask about it. If you’re unclear about a definition or how something was calculated, then ask the people who carried out the research to clarify. This sounds obvious but it’s really easy to make assumptions, particularly with data that’s most familiar to us.

A long time ago, I worked with a gravy brand to understand how they could encourage people to consume more gravy with their roast dinners. A recently commissioned quantitative study had revealed that only half of roast meals were consumed with gravy, despite a significantly higher proportion of people claiming to love gravy. It was my job to understand how to unlock this huge growth potential. I spent time in people’s homes watching them prepare their roast meals. I went shopping with them. I asked them to record everything they ate and drank during the week. I probed into their family and work lives.

But the reality just didn’t fit the data.

People who like gravy pretty much always consume it with a roast dinner. So, I asked to see the questionnaire that the quantitative survey was based on. It turns out that the definition of a ‘roast meal’ included any meal that featured roast meat: not just hot roast dinners, but also the meals that are invariably made using leftovers, such as chicken salads and roast beef sandwiches. And who would want to put gravy on a salad or in a sandwich? I wasn’t curious enough about the data. It’s one of the most valuable lessons I’ve learned in over 20 years of working and I no longer feel embarrassed about asking stupid or basic questions about clients’ data. It’s also one of the most enjoyable mistakes I’ve made at work. I love gravy.

If you’ve enjoyed this article and would like to download a PDF version, just click on the button below:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s