The Quantum Mousetrap

Mark Eduljee's blog about Social Media Insights Intelligence and his FlightSim Movies

Posts Tagged ‘Social Media Listening’

The siren call of Sentiment Analysis

Sunday, June 1st, 2014

In days of yore, it was said that mariners seeking direction and safe passage, would be lured to all manner of disaster by a song of promise which was wonderful to listen to, captivating, beautiful, simple, and appealed to their most basic human emotions.

The siren song of sentiment analysis and data

In today’s modern business voyage, where there is a similar need and search for safe passage, direction and smooth sailing, corporate captains can be observed urging their organizations to tune into the siren song of sentiment measurements to act as their compass, motivator, resource allocator and catalyst for decision making.

Why is this? It’s because the basic expressions inherent to the Sentiment analytics output models are easy to understand and frame in a conversation. Sentiment measurements generally roll up to 3 simple states: Positive, Neutral, and Negative. In more layman’s terms: Happy, Sad, Meh! Everyone gets it; they are emotional states that are easy to relate to. It’s Simple Elegant Measurable. It calls.

And from there its but a small leap towards group-think, sound-bite solutions: How often have you thought of, or heard some version of this get-quickly-to-solution refrain?…“If only I had a scorecard that showed me my “sad” number, their regions, and themes, then, all I need to do is to figure out what went wrong, address, and that would convert them to.. !! Happy!!”

That sounds like a reasonable plan and process, and seems logical. You get the cookie for “connecting the dots”. Kudos for strategic thinking! This will help the company and others around you get better. Direction and safe passage! Smooth sailing!! Promotion here I come!!!!

Umm…“Captain, my captain…cover ye ears and drown yon evil siren sentiment song, for there be all manner of foul, confidence-busting rocks and shoals of confusion-seaweed out tharr!

But before we go further, this is NOT about ignoring or dismissing sentiment data. For any business decision maker, the need to understand customer emotion about the brand, its experiences, engagement, or communications, IS important. Like any data, sentiment is but one arrow in a quiver of analysis options that can be useful when used with intention and a full understanding of what it should or not be used for, its limitations, and of how it can easily be misinterpreted or misused (and so become dangerous for their ship and crew).

The simplicity of its 3-state data is in fact the root-cause and reason for Sentiment data overuse and abuse and these are sure signs to look for that point to Sentiment data not being used appropriately: …when sentiment measures becomes the preferred basis for decision making
…when it is adopted as a key business and performance success measurement
…when its numbers are used to determine budget allocations
…when it is the dominant trigger for marketing, communications, or customer engagements
…when service contracts are awarded based on how well, easily and/or quickly sentiment can be measured by those services
these all sure signs that there be rocks and propeller-fouling seaweed ahead.

Look around your business or where you work. Do you feel there is a disproportionate reliance on this simple measurement?  If yes, then this follow up question may also be well worth asking: Fundamentally, has this sentiment measurement strategy helped the business over the LONG term? Or, is it resulting in scorecards causing short term and reactive operational, communications and engagement activity (and churn)?

If sentiment were to be used as a long term value-add for the good ship Enterprise, it MUST meet the following data requirements: (actually this is true for most data). Any BI or Insights (what’s the difference?) measurements must be:
1) Meaningful – provides a level of detail enough that it can be assigned to someone who is holding themselves accountable to address
2) Relevant – information that has some bearing on what drives the business, its priorities, and direction
3) Contextual – information that can be used to influence and inform choices and outcomes by being repeatable, reproducible and with samples and examples that can be tested and documented
4) Timely – occurring when the opportunity for maximum gain is evident

Now I know what you are thinking…“Wait! What about the requirement that the data and insights be ACTIONABLE? Data needs to be actionable otherwise its useless! Why is that missing from your list of requirements?

You are right in one respect: Often the term Actionable is listed as a key data requirement.

But, in practical terms however, actionability is really a function of an enterprise’s ability and desire to take action, and that is defined and enabled by how it is led, managed, funded and organized, which, then deterimes whether taking action is an embeded culture, and how costly that process will be…in other words actionability a is a desired business state, or customer RESULT, not a data requirement. In truth taking action is a business objective whose probability of happening is based on, and is directly correlated with:
a) the methodology, framework, and processes in place and used to acquire and process source data
b) data confidence and quality generated by BI, Insights, Monitoring, or Listening programs and systems (read about the difference between Listening and Monitoring, and BI and Insights)

Therefore, once the 4 requirements and 2 objectives mentioned above are met, then the data becomes actionable.
Get them right, and the data becomes, by virtue of those foundational conditions being true, “actionable”.

Meeting the requirements and objectives listed above to make Sentiment data actionable is not a trivial exercise.

While there are numerous tools and services who say they meet the requirements, it’s all a bit of smoke and mirrors. Admittedly, while some (and these tend to be the more expensive options) have fairly sophisticated entity, semantic, and language processing science and IP underlying their charts, graphs and other eye-candy sentiment analysis delivery, the majority of these services provide questionable sentiment numbers, at best. Caveat emptor!

But even with caveats and context, no matter how great the tools say they are, there remain the following fundamental truisms about sentiment that the business must own and manage…because, at its very core, at its very essence, “sentiment”, as a long-term business value-add measurement suffers from the following:

Like the Sirens, Sentiment is beautifully ethereal. In fact it’s a reflection of a current state of mind, and does not reflect or have ANY connection to future potential or status. For this reason, it should only be used to reactively address very short term situations or very tightly scoped scenarios.

Like the complex emotions it is meant to measure, Sentiment analysis is not simple. Yet, in an attempt to make it so to drive wider adoption it has been reduced to a 3 point measurement that cleverly includes the catchall: Neutral. That is why the neutral bucket tends to dominate most sentiment results. And it is also why Neutral is generally treated with respect, but also ignored. “If they are not against us, then they must be sort-of with us…Good to go!!” Do you know of anyone who has a performance commitment that requires their work to reduce Neutral and shift those numbers towards Positive? Hmm…

Like the richly varied stories about Sirens told by sailor to sailor, Sentiment measurements are based on storied standards that vary from vendor to vendor…each claiming that their product offers the most accurate results. But where are the industry-accepted sentiment standards? There are none. Every company uses their own model. Generate sentiment scores on the same data from 5 different social media tools and you’ll get 5 different results — all claiming to be right.

Like Sirens who bend and influence minds to follow their call, so to can those providing Sentiment ratings be biased towards a mindset that effects ratings. There are any numbers of ways to provoke specific emotions that would influence sentiment scores for specific data sets. And because Sentiment is based on, and is a measurement of a shifting emotional state of the mind (that is in turn influenced and effected by millions of unknown, interconnected variables like culture, age, expertise, maturity…), it is relatively easy to then use tactics (often deployed unintentionally) to influenced the framework and mechanisms which records the data (or choose the source) to be used in sentiment analysis into getting what is wanted (as opposed to what’s really needed).  As examples…“provide feedback” sentiment scoring by its very nature will generally be neutral to negatively framed because it’s about improving what perceived to be broken (but paradoxically, it’s provided in the hope for a positive desired outcome – so where does that factor into the equation?), or, if the same feedback ask was framed as being about “opportunities to improve” will shift towards using more positive language. Also the timing of when a sentiment gathering instrument is presented to gather data, or its labeling can and will influence the sentiment a person feels as they contribute data …“send a smile”, “show how much you care”, “talk to us – you are important!”…are all examples of ways sentiment can be influenced to drive results wanted (not needed)

Like the Sirens of yore, Sentiment is an emotive phenomenon — there, but not there: not much different from rainbows or mirages that seem real under the right conditions and are characterized by shifting states with no anchors. At its core, Sentiment measurements are an interpretive science and not based on scientific modeling. Don’t agree? Here’s a simple test: Give me a sentiment score and its driver weight that, if acted on would move the sentiment/emotion being measured by that driver weight.  You could do this through with Satisfaction, or affinity or loyalty measurements using driver regression and dependent variable driver modelling and analysis. Not with Sentiment. So what this means is that in the end sentiment is a number that gets pulled out of a hat, and is determined by which hat it’s pulled from.

Having a measurement that shifts and changes to reflect real time market or competitive-driven emotion can be enervating and frustrating, and demoralizing for the enterprise struggling to do the right thing with finite resources. Picture facing a bewildered VP who has just been told that the 3 things he (or she) is spending millions on to change based on last week’s sentiment analysis has had no effect on negative sentiment scores because customers have shifted the focus of their emotions this week to some other shiny object. It’s the surest way to lose credibility and a seat at the table.

Of all the issues with Sentiment, the lack of standards stands out as THE most troubling, and a prime reason to go to it as a last resort. Unfortunately it’s this lack of accountability to any standard that is most manipulated to achieve self-serving goals both within the enterprise and by vendors and consultants.

Having BI and Insights systems (different) which deliver high confidence information, that result in actionable data, derived from predictable and repeatable methodology which can be explained and justified, will drive moral, a sense of accomplishment and eventually, organizational and business health. Sentiment is a siren’s call in that regard.

This is why I steer my stakeholders away from Sentiment as a success measure, and instead work closely with them to seek more stable, higher confidence data in its place that is more suited for the business need rather than the knee-jerk emotional want. (More about the differences between needs vs wants here) 

So what’s the alternative??

First: Establish WHY Sentiment analysis is being asked for. Sentiment is a useful analytics option for any data science team to use IF the business need is short-term and reactionary… efforts like PR, monitoring or reacting to an event, campaign, opening, or an evolving/dramatic human-interest story, or, keeping tabs on services operations and availability. Sentiment monitoring (that’s all it really is) has its place when used with intention and forethought about its scope and limitations. Use it wisely.

Second: By all means, feel free to be obsessed with the latest shiny sentiment trend visualization, but balance that with responsible, longer term, analytics investing for more stable, actionable Insights and BI. Invest in the people, tools, and process to build frameworks that are necessary to provide standards-driven results that return meaningful, relevant, contextual, and timely data. Only then will action-taking for permanent customer experience changes become real.

Third: Embrace and accept the idea that Big data and Insights intelligence analytics is neither simple, quick, cheap, or easy. If you absolutely must have sentiment analysis, set expectations and clearly come to terms with its scope and limitations. Identifying, reporting, and using high confidence data and insights that drive action and change requires investment, specialized skills, a long term horizon, sponsorship, focus, discipline, time, and leadership. 

In other words: snap out of it!! ...and watch out for the songs from the Sentiment sirens! If you do, your business, customers and people will thank you.

 

 

 

Image credit: Businessweek.com

 

The difference between Monitoring and Listening to Social Media

Saturday, November 27th, 2010

True story: I received a rather irate email a couple of weeks ago from a fairly senior Marketing Manager who had been forwarded a copy of one of the Voice of Community Social Media Listening Intelligence reports that my team publishes inside Microsoft, asking me (or was it, demanding?) to explain why I was providing this social media listening, because my team is already monitoring and measuring social media.

While I was surprised that this person had incorrectly jumped to conclusions and made some incorrect assumptions about what I did, I was not surprised (mildly amused actually) by how loosely this person had scrambled “Listening” and “Monitoring” together into the same sentence, interchanged and interwoven the two concepts, and then served it up on burnt toast.

But it’s no wonder…even Bing and the other who-shall-not-be-named search both serve essentially the same slices of over-toasted results if you search for the terms Social Media Listening or Social Media Monitoring.

There is however a big difference between the two, especially when viewed through the lens of, and within the context of, customer insights, marketing research, tools, analysis, and tracking.

Here’s how the two differ:


I hope you will agree that while both functions use social media as a data source, their metrics, nature, framework and goals are both are very different.

Now I know what you are thinking… “Wait! Can’t I get actionable insights by Listening to my View, Sentiment and other metrics?!!?”

Nnnno.

I’ll maintain that you are not listening. You are analyzing numbers for trends and patterns which will probably cause you to drive some sort of activity to improve a process or become more efficient from a production pov to move those numbers towards some set goals. And that’s not a bad thing. But it’s not listening.

Listening is about hearing the voice of your customer. Really good, unbiased Listening is about tuning into the conversation which is being organically DRIVEN BY THE CUSTOMER… not by you, on their own timeframe, at the locations of their choosing, using the words they wish to use to express their experience. That gives you listening intelligence. It can either confirm what you already may know, or, it will lead to new insights and discoveries (that’s the really exciting and VALUABLE part of Listening). That’s why if you bias your social media intelligence systems with specific keywords (things that I think are important to my business and that I’m interested in tracking), or locations (places where I know my key customers have told me they visit), or if you scope it to people or locations (I only want to hear from select, important customers), you are then monitoring social media. You have defined what you want. You are not listening to what you need to hear.  (See related post about Needs vs Wants: I want a Lollipop! …NOW! )

Is one “better” than the other? No. Each has its place. Each has its uses. Each has its benefits, and challenges.

But to confuse the two will cause team and business decisions that rely on their outcomes to shift and shuffle, incorrect assumptions to be made, and worse, will cause decisions to be made on the wrong insights and metrics.

As an example:

Let’s take a simple case: You may have heard or been witness to something like this… “Our social media listening tells us that our content views and our reach (followers, click through) is declining. To turn these measurements around, and to demonstrates that we are listening to our customers, we will launch a new ad campaign expanding it to 5 more languages, simultaneously updating our Online Help documentation, and adding 5 new sales incentives on our site”.  (Sounds like a great plan!)

Now let’s suppose that in a parallel universe (same people, same set of circumstances — this often happens in the Quantum Mousetrap), the Company has a distinct listening framework: It is not goaled with monitoring, but with the gathering and analysis of social conversations to discover voice of community insights. These insights then lead to the realization that Customers are blasting the Company (and spreading the word) in social media for an objectionable post it made to its Blog. This is hurting the company’s reputation, causing customers to desert the brand. (Less views, followers etc.) Knowing this, the decision is made to post an apology to the blog with steps being taken for corrective action, and to run an ad campaign focused on rebuilding company and brand trust.

Same metrics… but two different sets of decisions based on being clear about whether the discovery causes changes in production and campaign activity, or if it is insights based on listening to the customer voice to change strategy.

This difference was further highlighted at a conference I recently attended, The Market Research Event (TMRE) in San Diego. There I presented “Social Media Listening: How to drive big ROI”, (nice/kind review in SurveyAnalyticsBlog). At the event, I mentioned to the Marketing Research pro’s in the room that Social changes everything…everything about the way customers and businesses have traditionally interacted, exchanged, engaged, communicated, shared, recommended and opined. And since this is Market research’s (MR) traditional playground, I summarized that MR would need to stretch and evolve to account for this new reality too.

There were a few who seemed mildly threatened by this picture, but generally, the reaction I got was more along the lines of curiosity and excitement for what the change could mean. A few even privately admitted to me that the shakeup would be good for an industry that is top heavy with traditionalists who insist on maintaining fixed/established industry process. But in its defense, MR probably plays the traditionalist card I bit more than it would like to so as to keep the trust of those who rely on its data. (“They are solid and dependable!”)

Consider these amazing metrics about online social trending and participation globally:

  • Internet users up 13% Y/Y1
  • Twitter up 75% Y/Y2
  • FB up 51%Y/Y3
  • Search up 11% Y/Y4
  • Mobile international up 37% Y/Y but with only 14% penetration5

How will this trend towards social “change everything” for marketing research in the coming years?

  • The Social conversation is always on, and growing: Unlike traditional Marketing Research projects which are switched on and off, Social Media conversations never sleep. Someone somewhere is always creating content, expressing an opinion, or building relationships which will have a direct, collective effect on your product or service. There will no longer be a need
  • It’s always being updated: Social Media technology provides customers worldwide to provide updates to reflect their current experience.  No longer is the data out of date the moment the research project concluded.
  • It’s always relevant: This is because the conversations change with the experience relevant to the product lifecycle. If it’s a to-be released product, the conversations will reflect that reality. If the product has just launched, guess what the conversation will be about. And as it sunsets, conversations will trend towards the next version or evolution.
  • It has a historical “memory”: The conversation only evolves…its never deleted. It’s always there to be listened to by anyone who is willing to hear it.

All this means that Marketing Research will face a growing need to evolve away from its reliance  on traditional “offline” methods it uses to gather customer insights… surveys, panels, interviews, observations etc. True, surveys and other MR instrument can be shifted to run online, but that’s not the point.

The question that needs to be asked is this: Why not Monitor activity and Listen for insights in conversations that your customers are already having? Why spend the time and resources to build MR instruments and projects to create and collect customer insight data that already exists in Social?  Oh – you don’t agree that it already exists?? ..Then take this simple test…Think of something — Anything.  And Search for it. Chances are you are not going to draw a blank search result.  Admittedly, the result you see may not be exactly aligned to your topic, or what you expected, but that’s a function of the quick-and-dirty search method you just used, not that there is no data about your subject across the spectrum of Social. And that’s the point: The data is there. The challenge is to develop clearly differentiated Monitoring and/or Listening systems which can effectively, efficiently, reliably, and predictably tap into the always on, updated, relevant and historical social universe. (How to do that was the topic of my presentation at TMRE. I’ll post a link to it here in the near future… as soon as TMRE posts the recording)

THE BOTTOM LINE:

  • Clearly differentiate what you are doing/trying to do – Social Media Monitoring, or Social Media Listening. One is not worse or less desirable than the other. Each has its place. Gaining this clarity will help to scope effort, focus goals, set assumptions and expectations so your business gets what it needs, at the time it is most useful.
  • Social changes everything. Evolve the Market Research discipline to meet this new reality. Companies who evolve their near-term, on again/off again project-based marketing research efforts away from easy measurement and Monitoring of social media campaign trends and engagement (reach, followers, sentiment, keyword tracking etc.), and, instead, invest in a longer term social media research, analysis, and Listening  strategy based with an investment in a framework of people, process, and technology goaled with listening for revenue and/or efficiency intelligence (the right information, at the right time of the lifecycle, from the right community source/authority) will have both a competitive and a customer perception-influencing advantage.

 

Closing thoughts… the function of Listening to the voice of Social for the purposes of identifying actionable business insights to improve the customer experience is an emerging discipline. It’s harder than simply subscribing to social media monitoring tools. Listening takes more time. It takes added investment. It needs a long-term commitment. The payoff however is timelier, more relevant, justifiable, actionable, customer insight intelligence.

Are you monitoring your Marketing or Support social media activity, or are you listening to the voice of your customer for actionable intelligence and insights?

Credits: Data from a Morgan Stanley presentation by Mary Meeker at the Web2.0 summit in SFrisco 2010

1) Internet user stats per International Telecommunications Union
2) Twitter user figure reflects global unique visitors to Twitter.com in 9/10, per TwitterCounter.com, comScore
3, 4) comScore (global unique visitors for Facebook), PC World, comScore (global user data for Facebook, Google as of  9/10),
5) Informa WCIS