Beware Confirmation Bias in Surveys; Get Better Feedback Through Conversation Analytics
Beware Confirmation Bias in Surveys; Get Better Feedback Through Conversation Analytics
The best marketers are empathetic with their customers. To accomplish this requires technology with enough emotional intelligence to pick up on customer cues. These cues reflect sensitivities outside company pre-conceptions about brand (the pride of the marketer) and more objectively, to seek to understand the customer’s actual experience with the brand.
Traditionally, marketers use surveys to capture and benchmark customer opinion and sentiment about a brand. Surveys remain intrusive nature and continue to suffer declining response rates. In addition, they are subject to pre-conceived summaries and categorization which feed into a marketer’s own confirmation bias and the customer’s positive or negative perceptions about the brand.
For example, a sample survey may ask the respondent to rate its experience from 1 to 10 in satisfaction about a product. Dependent on the product, the survey measures anything from ease-of-use, product quality, style preference, suggestions, overall rating as to whether or not they would recommend the product to a friend, and how well the brand performed relative to the customer’s expectations. The net output of these survey results generate a product scorecard in the form of a Net Promoter Score with intent to assist the marketer with a summary of customer sentiment and create informed decisions regarding how to improve the product and/or its perception.
Confirmation bias, popularized by Nobel prize winners Kahneman and Tversky, explains the brain’s biases towards interpreting information to confirm one’s own preconceptions. By asking the marketer to pre-define which responses customers can answer and rate, surveys by default are already biased towards a marketer’s pre-conceptions of how a customer may interpret categorically how the customer thinks about a product. Often, a marketer strongly tied to its brand can have pretty strong opinions of what is driving positive or negative sentiment, and the results from the survey can feed into their own preconceived biases about how customers relate to the product which in turn is used to reinforce the game plan of next steps they may have in had in their own minds. Ideas they may have had before even conducting the survey.
Herein lies the problem. Yes, a good marketer’s intuition and “gut instinct” can sometimes merit a pretty high level and somewhat accurate picture of a product’s sentiment and the reasons that may be driving it. However, a survey’s simple ratings mute the emotions, understandings, and perceptions of the customer that generated the rating and feed the marketer only with what they want to hear by the categories they present to the customer to answer - and too often critical pieces of the puzzle goes missing of what actually drove customer opinion.
INSIDE SURVEY ASSUMPTIONS
A marketer of cosmetic products released an eye make-up product through DRTV television marketing. The surveys asked customers who purchased this product how to rate it in categories such as ease of use, time to apply, comfort, aesthetic appeal, quality of product and quality of packaging. Overall, the product launched with a low NPS rating, with many customers averaging out their negative response the same across every category in the survey - all “1s” for examples. For those who rated the product separately in the pre-defined categories, the categories of “ease of use” and “quality of product” drove the lowest sentiment while “quality of packaging” drove the highest.
The marketer’s intuition was pre-disposed to ideas they wanted to confirm from the onset - the product’s Chinese manufacturer’s credibility was in question, they knew the instructions inside the package pointed to a web address which added an additional layer of friction to get the customer the information they needed, and the marketer took particular time and effort to perfect the package to get it moving off retail shelves.
From survey response to marketer intuition … it seems the case was closed; the marketer’s preconceptions were true and the marketer had a path to a better action plan in place. But was it?
ENTER CONVERSATIONAL ANALYTICS
Across every touch point, customers are telling brands exactly what they want and how they feel. They are proud promoters or detractors. They have opinions and ideas. They have a need they want to fulfill or a curiosity they want to discover with the product they chose. They will interact with your brand by picking up a phone and asking a question, posting on social media, or voicing an online review. And inside every conversation are a wealth of ideas, concerns or opinions invaluable to every marketer.
Reviewing these conversations manually for critical customer feedback would require high overhead and prove way too costly. Hours of labor spent collectiong all the audio and text from each consumer is a timley process. “Conversational analytics” solves this by aggregating conversations across communication channels into a single platform view. This includes telephone calls which are converted from unstructured voice files into structured text data using automatic speech recognition (ASR) and natural language understanding (NLU) technology.
Focus on the consumer side of all exchanges creates the view from all their points of contact. Only collecting 100% of their comments can everything be mined for valuable emotional clues about their feelings for the brand. Aggregating these conversations into a selectable dashboard, marketers can quickly discover conversation points by keywords ranging from product quality (“broken”, “flimsy”), sentiment (“angry”, “love it”), confusion (“I don’t understand”), logistics (“missing”) and countless others. Next, the marketer can take a keyword group and drill down, with empathy, to the part of the customer conversation where the keyword appeared to understand it in its complete context. A product detractor, for example, may have combined keywords of “angry” with “missing” inside a phone call to highlight an issue in production causing missing items to ship. Equally, ideas currently sitting outside a marketer’s priority list may exist such as “do you have this for kids” may appear enough to highlight a new version in demand. The possibilities are endless.
The key is to have best in class audio and text recognition. One of the highlights of conversational analytics is that customer feedback is gathered passively, without need to ask customers again for an opinion they are already giving. This results in automatic participation by anyone giving feedback to the brand.
STEPPING OUTSIDE OF INTUITIONS AND LISTENING TO CUSTOMERS
In our case with the cosmetic product manufacturer, we employed conversational analytics to help the marketer get to the root cause of the low sentiment for its eye make-up product. By importing feedback from inbound sales calls, social media posts and online reviews, we found information sourced directly from buyers that completely slipped the marketers assumptions for its low reviews.
First, the television commercial referenced features about the package that was different than what was listed on the company’s website. And further, the description given by the inbound sales representatives misrepresented what was being shipped. The representatives were not aware of a feature advertised in the commercial, creating confusion and distrust for the customers trying to buy the product.
Second, buyers of the product found the process to apply the eye make-up “tedious” - something which aligned with the two most negative survey categories in “ease of use” and “quality of product”. It turns out, customers didn’t mind going to a web address to learn how to apply the make-up - they just found doing it themselves to be a lot more difficult than they expected and the materials a bit too “heavy” to manage properly.
With this new information afforded through conversational analytics, several of the marketers preconceptions and assumptions from the survey were broken down. In-package paper-based instructions or switching manufacturers, as they had originally believed was the best strategy for improvement, was not the actual problem at hand. They needed to find better materials to make application of the product easier. Additionally, they found interdepartmental communication black holes between the advertising team, contact center team, and web team causing confusion and distrust. It was a learning lesson to break down the silos and Individual preconceptions, driven upwards from the customer base.
CONCLUSION
A marketer must rely on its intuition; however, one must always be aware of confirmation bias leading to wrong decisions. Surveys can often lead to facilitate a marketer’s preconceptions, and therefore must be used in correlation with conversational analytics to dive deeper into the root cause of customer opinions. By having empathy with one’s customers, which simply means trying to listen and understand, one may find incredibly useful data outside our own intuitions to move our brands in better directions we didn’t know.
