The Missing Metric in Measurement: Context
Bringing intuition and holistic thinking back to the decision-making table
Three researchers walk into a bar… well, that is almost how this story started: conversations among friends, collaborators, and mentors on the state of research in media and technology. For this month’s Beyond the Algorithm I am bringing together three voices of consumer insight leaders at the intersection of tech and media to share perspectives on our favorite thing and most-used crutch: MEASUREMENT!
Summarizing conversations over dinner, Zoom meetings, and emails, we grappled with fundamental changes in our industry and discussed the importance of the broader context of data, the dangers of relying solely on single sources of data in isolation, and the need for more holistic approaches to measurement.
In the media and technology categories, we measure things six ways til Sunday and it might be fair to add that we often do so with a great sense of confidence, even when our sources are known to be inherently imperfect. Fragmentation in the TV landscape has made measurement harder than ever. Where relying on a single syndicated source was once the industry standard, now that has quickly become a fool’s errand.
Sidebar: don’t even get me going on the impact that the pandemic has had on the consistency of our longitudinal data sources. Next month’s soapbox?
Data is alluring. Addicting. We love it, need it, crave it.
We’re held accountable by data; it is the proof-point behind meeting our goals. It makes us look smart, checks the box that we have done our homework, helps us navigate and question our personal and organizational assumptions, and it can help support very powerful, strategic business decisions. But the reality is that not everyone knows how to wield the sword in our current complex data landscape. Worse yet, organizations are not doing enough to fill skill gaps and train teams to navigate the nuances of the current state of our data sources. So many organizations have become too focused on the easy answer of statistical significance alone, losing sight of the bigger picture. When these things happen, data can lead us to make decisions that may not make sense when translated to the real world.
David Feick, Sr. Director, Brand & Consumer Insights at Windstream, has always been ahead of this problem in how he approaches research programs and growing insights functions. “In the era of AI, ‘big data’ and the democratization of analytical tools that empower even non-experts to discern patterns and conduct experiments, I see a growing disconnect between organizations and their customers. In the pursuit of quantitative truth, consumers are unwittingly distilled into mere data points through what's often an endless chain of A/B testing, leading companies to overlook the broader, more meaningful narrative,” says Feick.
Blind faith in statistical significance without considering the broader context begs the question: are we placing too much weight in reporting “stat sig” and shifts in numbers without grounding ourselves in the broader context?
Is this a habit we can work together to help break? Taking action requires retraining our minds to think differently about mixed methods, our research roadmaps, how and where we source our data, and how we interpret it.
While tools like AI promise easy solutions, it seems that the easy button days are over in complex categories. Researchers have to do a lot more work to identify the strengths, weaknesses, and gaps in each data source they are pulling from, be great interpreters of what means, be able to design custom studies to fill the gaps, and be prepared to knit together the pieces into a more holistic picture.
Dan Castro, Director of Data & Measurement at NBCUniversal contributed an artful (pun-intended) metaphor to describe this shift in mindset that we need to make when we think about data as a color palette.
“The change the measurement industry is going through right now is that previously, we just had Nielsen, which I would liken to an ink painting, just black and white. Now, you still need that ink base, but the landscape is more like a Bob Ross painting where you need to add your cobalt blues and rosy reds with your VideoAmps, iSpots, Samba TVs. You have to paint the entire picture using multiple media and not just a single one. You need the proper people in place to know which colors to use at which time based on the needs of the business.”
- Dan Castro
Researchers today need to be prepared to be architects rather than just data miners.
Building a new color palette requires interpretation, creativity, holistic thinking and a lot more customization than many teams have been historically trained for. This requires more first-party data, where you know your source inside and out and what you are controlling for, mixed methods mastery, and effective triangulation skills. Teams who take a more contextual, mixed methods and multiple data sources approach will have greater confidence in the whole because they will know all of the parts and the why behind areas of convergence and divergence, reducing the risk of biased or incomplete data sets.
Alongside these measurement woes, we also need to be smarter about the KPIs and north-star metrics that we hold as gospel for our teams. Teams tend to find themselves chasing single data points as heroes of their KPI narrative, but as Kirsten Lewis points out, assigning too much value to sole metrics in isolation can be fruitless.
Composite scores, on the other hand, layer in multiple facets of an engagement to understand both the PARTS and the WHOLE that are driving satisfaction, engagement, or use. These are always more powerful for an organization, as they can serve the same role that a North Star Metric does, but with the ability to drill down one click further to understand strengths and weaknesses within. It can also be leveraged as a diagnostic tool to set standards, and to target or segment users with different profiles within the composite score. Put simply: this approach has so much more utility and staying power than a single metric on its own… something Kirsten and I can say works with confidence since we built one many years ago together at Sonos which is still in use today!
As we move forward in this crazy tangled web that is the media category, let’s share in a commitment to bring our investigation, interpretation, intuition, and observation skills to the table in our work every day.
GRATITUDE for the contributions from a brilliant cast of collaborators:
David Feick: Sr. Director, Brand & Consumer Insights at Windstream. Previously, David has served in Insights & Analytics leadership roles at Sonos, T-Mobile, and Yahoo, and has been at the forefront of two decades of change in brand management, analytics, and consumer insights.
Dan Castro: formerly Disney/ESPN, now joining forces with Peacock as Director of Data & Measurement at NBCUniversal, and an expert propagator of Happy Little Trees and Bob Ross joy.
Kirsten Lewis: an innovation and user research leader with recent posts at Google, Sonos, and Bose, Kirsten brings motivational and future-forward leadership that inspires design and product teams to stay curious.