The Quantum Mousetrap

Mark Eduljee's blog about Social Media Insights Intelligence and his FlightSim Movies

Posts Tagged ‘BI’

The siren call of Sentiment Analysis

Sunday, June 1st, 2014

In days of yore, it was said that mariners seeking direction and safe passage, would be lured to all manner of disaster by a song of promise which was wonderful to listen to, captivating, beautiful, simple, and appealed to their most basic human emotions.

The siren song of sentiment analysis and data

In today’s modern business voyage, where there is a similar need and search for safe passage, direction and smooth sailing, corporate captains can be observed urging their organizations to tune into the siren song of sentiment measurements to act as their compass, motivator, resource allocator and catalyst for decision making.

Why is this? It’s because the basic expressions inherent to the Sentiment analytics output models are easy to understand and frame in a conversation. Sentiment measurements generally roll up to 3 simple states: Positive, Neutral, and Negative. In more layman’s terms: Happy, Sad, Meh! Everyone gets it; they are emotional states that are easy to relate to. It’s Simple Elegant Measurable. It calls.

And from there its but a small leap towards group-think, sound-bite solutions: How often have you thought of, or heard some version of this get-quickly-to-solution refrain?…“If only I had a scorecard that showed me my “sad” number, their regions, and themes, then, all I need to do is to figure out what went wrong, address, and that would convert them to.. !! Happy!!”

That sounds like a reasonable plan and process, and seems logical. You get the cookie for “connecting the dots”. Kudos for strategic thinking! This will help the company and others around you get better. Direction and safe passage! Smooth sailing!! Promotion here I come!!!!

Umm…“Captain, my captain…cover ye ears and drown yon evil siren sentiment song, for there be all manner of foul, confidence-busting rocks and shoals of confusion-seaweed out tharr!

But before we go further, this is NOT about ignoring or dismissing sentiment data. For any business decision maker, the need to understand customer emotion about the brand, its experiences, engagement, or communications, IS important. Like any data, sentiment is but one arrow in a quiver of analysis options that can be useful when used with intention and a full understanding of what it should or not be used for, its limitations, and of how it can easily be misinterpreted or misused (and so become dangerous for their ship and crew).

The simplicity of its 3-state data is in fact the root-cause and reason for Sentiment data overuse and abuse and these are sure signs to look for that point to Sentiment data not being used appropriately: …when sentiment measures becomes the preferred basis for decision making
…when it is adopted as a key business and performance success measurement
…when its numbers are used to determine budget allocations
…when it is the dominant trigger for marketing, communications, or customer engagements
…when service contracts are awarded based on how well, easily and/or quickly sentiment can be measured by those services
these all sure signs that there be rocks and propeller-fouling seaweed ahead.

Look around your business or where you work. Do you feel there is a disproportionate reliance on this simple measurement?  If yes, then this follow up question may also be well worth asking: Fundamentally, has this sentiment measurement strategy helped the business over the LONG term? Or, is it resulting in scorecards causing short term and reactive operational, communications and engagement activity (and churn)?

If sentiment were to be used as a long term value-add for the good ship Enterprise, it MUST meet the following data requirements: (actually this is true for most data). Any BI or Insights (what’s the difference?) measurements must be:
1) Meaningful – provides a level of detail enough that it can be assigned to someone who is holding themselves accountable to address
2) Relevant – information that has some bearing on what drives the business, its priorities, and direction
3) Contextual – information that can be used to influence and inform choices and outcomes by being repeatable, reproducible and with samples and examples that can be tested and documented
4) Timely – occurring when the opportunity for maximum gain is evident

Now I know what you are thinking…“Wait! What about the requirement that the data and insights be ACTIONABLE? Data needs to be actionable otherwise its useless! Why is that missing from your list of requirements?

You are right in one respect: Often the term Actionable is listed as a key data requirement.

But, in practical terms however, actionability is really a function of an enterprise’s ability and desire to take action, and that is defined and enabled by how it is led, managed, funded and organized, which, then deterimes whether taking action is an embeded culture, and how costly that process will be…in other words actionability a is a desired business state, or customer RESULT, not a data requirement. In truth taking action is a business objective whose probability of happening is based on, and is directly correlated with:
a) the methodology, framework, and processes in place and used to acquire and process source data
b) data confidence and quality generated by BI, Insights, Monitoring, or Listening programs and systems (read about the difference between Listening and Monitoring, and BI and Insights)

Therefore, once the 4 requirements and 2 objectives mentioned above are met, then the data becomes actionable.
Get them right, and the data becomes, by virtue of those foundational conditions being true, “actionable”.

Meeting the requirements and objectives listed above to make Sentiment data actionable is not a trivial exercise.

While there are numerous tools and services who say they meet the requirements, it’s all a bit of smoke and mirrors. Admittedly, while some (and these tend to be the more expensive options) have fairly sophisticated entity, semantic, and language processing science and IP underlying their charts, graphs and other eye-candy sentiment analysis delivery, the majority of these services provide questionable sentiment numbers, at best. Caveat emptor!

But even with caveats and context, no matter how great the tools say they are, there remain the following fundamental truisms about sentiment that the business must own and manage…because, at its very core, at its very essence, “sentiment”, as a long-term business value-add measurement suffers from the following:

Like the Sirens, Sentiment is beautifully ethereal. In fact it’s a reflection of a current state of mind, and does not reflect or have ANY connection to future potential or status. For this reason, it should only be used to reactively address very short term situations or very tightly scoped scenarios.

Like the complex emotions it is meant to measure, Sentiment analysis is not simple. Yet, in an attempt to make it so to drive wider adoption it has been reduced to a 3 point measurement that cleverly includes the catchall: Neutral. That is why the neutral bucket tends to dominate most sentiment results. And it is also why Neutral is generally treated with respect, but also ignored. “If they are not against us, then they must be sort-of with us…Good to go!!” Do you know of anyone who has a performance commitment that requires their work to reduce Neutral and shift those numbers towards Positive? Hmm…

Like the richly varied stories about Sirens told by sailor to sailor, Sentiment measurements are based on storied standards that vary from vendor to vendor…each claiming that their product offers the most accurate results. But where are the industry-accepted sentiment standards? There are none. Every company uses their own model. Generate sentiment scores on the same data from 5 different social media tools and you’ll get 5 different results — all claiming to be right.

Like Sirens who bend and influence minds to follow their call, so to can those providing Sentiment ratings be biased towards a mindset that effects ratings. There are any numbers of ways to provoke specific emotions that would influence sentiment scores for specific data sets. And because Sentiment is based on, and is a measurement of a shifting emotional state of the mind (that is in turn influenced and effected by millions of unknown, interconnected variables like culture, age, expertise, maturity…), it is relatively easy to then use tactics (often deployed unintentionally) to influenced the framework and mechanisms which records the data (or choose the source) to be used in sentiment analysis into getting what is wanted (as opposed to what’s really needed).  As examples…“provide feedback” sentiment scoring by its very nature will generally be neutral to negatively framed because it’s about improving what perceived to be broken (but paradoxically, it’s provided in the hope for a positive desired outcome – so where does that factor into the equation?), or, if the same feedback ask was framed as being about “opportunities to improve” will shift towards using more positive language. Also the timing of when a sentiment gathering instrument is presented to gather data, or its labeling can and will influence the sentiment a person feels as they contribute data …“send a smile”, “show how much you care”, “talk to us – you are important!”…are all examples of ways sentiment can be influenced to drive results wanted (not needed)

Like the Sirens of yore, Sentiment is an emotive phenomenon — there, but not there: not much different from rainbows or mirages that seem real under the right conditions and are characterized by shifting states with no anchors. At its core, Sentiment measurements are an interpretive science and not based on scientific modeling. Don’t agree? Here’s a simple test: Give me a sentiment score and its driver weight that, if acted on would move the sentiment/emotion being measured by that driver weight.  You could do this through with Satisfaction, or affinity or loyalty measurements using driver regression and dependent variable driver modelling and analysis. Not with Sentiment. So what this means is that in the end sentiment is a number that gets pulled out of a hat, and is determined by which hat it’s pulled from.

Having a measurement that shifts and changes to reflect real time market or competitive-driven emotion can be enervating and frustrating, and demoralizing for the enterprise struggling to do the right thing with finite resources. Picture facing a bewildered VP who has just been told that the 3 things he (or she) is spending millions on to change based on last week’s sentiment analysis has had no effect on negative sentiment scores because customers have shifted the focus of their emotions this week to some other shiny object. It’s the surest way to lose credibility and a seat at the table.

Of all the issues with Sentiment, the lack of standards stands out as THE most troubling, and a prime reason to go to it as a last resort. Unfortunately it’s this lack of accountability to any standard that is most manipulated to achieve self-serving goals both within the enterprise and by vendors and consultants.

Having BI and Insights systems (different) which deliver high confidence information, that result in actionable data, derived from predictable and repeatable methodology which can be explained and justified, will drive moral, a sense of accomplishment and eventually, organizational and business health. Sentiment is a siren’s call in that regard.

This is why I steer my stakeholders away from Sentiment as a success measure, and instead work closely with them to seek more stable, higher confidence data in its place that is more suited for the business need rather than the knee-jerk emotional want. (More about the differences between needs vs wants here) 

So what’s the alternative??

First: Establish WHY Sentiment analysis is being asked for. Sentiment is a useful analytics option for any data science team to use IF the business need is short-term and reactionary… efforts like PR, monitoring or reacting to an event, campaign, opening, or an evolving/dramatic human-interest story, or, keeping tabs on services operations and availability. Sentiment monitoring (that’s all it really is) has its place when used with intention and forethought about its scope and limitations. Use it wisely.

Second: By all means, feel free to be obsessed with the latest shiny sentiment trend visualization, but balance that with responsible, longer term, analytics investing for more stable, actionable Insights and BI. Invest in the people, tools, and process to build frameworks that are necessary to provide standards-driven results that return meaningful, relevant, contextual, and timely data. Only then will action-taking for permanent customer experience changes become real.

Third: Embrace and accept the idea that Big data and Insights intelligence analytics is neither simple, quick, cheap, or easy. If you absolutely must have sentiment analysis, set expectations and clearly come to terms with its scope and limitations. Identifying, reporting, and using high confidence data and insights that drive action and change requires investment, specialized skills, a long term horizon, sponsorship, focus, discipline, time, and leadership. 

In other words: snap out of it!! ...and watch out for the songs from the Sentiment sirens! If you do, your business, customers and people will thank you.




Image credit: