.

Web Business by Ken Burbary

Digital Marketing, Social Media, Web Technology

Web Business by Ken Burbary header image 2

The Dirty Little Secret of Social Media Monitoring

July 13th, 2009 · Comments · Social Media

Social Media Monitoring (sometimes called social media listening or conversation monitoring) is widely recognized as one of the first social media best practices. It is the first step in getting to know and understand the behavior and needs of an online audience. Most companies that are engaged in this activity have an internal team or outsource the effort to their agency.

There is no shortage of tools/platforms available to monitor your key terms. Companies like Radian6, Techrigy, Nielsen and Cymphony are quite busy these days rolling out new partnerships and product features, all aimed at providing more meaningful data and better ways to use it. It’s the no-brainer call right? Fire up a tool and plug in some terms, and voila, instant answers.

However, I’ve begun to notice inconsistencies in the data that different social media monitoring tools produce. The dirty little secret or so it seems, is they aren’t all working with the same data sources. For this post, I want to discuss some differences between Radian6 and Techrigy SM2. This is not meant as a criticism of either tool, because I think they are both fantastic at what they do, but rather an attempt to highlight an issue, generate a public conversation around it so that everyone using Radian6 or SM2 can benefit.

Some background: While doing a recent set of searches for a brand using the EXACT same keywords and phrases, without using source filters, the results look different. Take a moment to view the charts below for R6 and SM2 results.

Radian6 Results

SM2 Results

Now, ignoring the fact that both tools have some differences in categories, focus on the main sources of online conversation. Blogs, forums and micromedia (Twitter).

You’ll see a  dramatically higher number of blog results for Radian6 compared to SM2 (266 to 91). And for forums, vice versa (SM2 has 396 to Radian6 200).

While I expect to find subtle variations in the results between tools, I DO NOT expect to be put into a position to question which tool is “right” and which is “wrong”. Perhaps that isn’t the way to look at it though. Is the real answer an aggregate of both data sets? If so, how does one easily filter out the unique data versus duplicates?

This may raise additional questions that you need to ask yourself before selecting a monitoring solution.

Which subset, or channel, of social media does a particular tool specialize in?

Does this data mean if you know your audience is spending time on forums, SM2 may be better solution? Perhaps. Perhaps not. I suspect there is more to this than it appears on the surface. Hopefully someone else that is also monitoring can shed some light. With any luck, we can get some folks from both companies to chime in (David Alston, Amber Naslund, Jimmy Rey, Connie Bensen – any takers?)

UPDATED: I’ve been contacted by both Radian6 and Techrigy representatives, and am working with them to identify the source of the problems. After we have resolved the issue and feel confident about the source, I will update again.

Share and Enjoy:
  • Facebook
  • del.icio.us
  • TwitThis
  • Digg
  • StumbleUpon
  • Sphinn
  • Mixx
  • LinkedIn
  • Google Bookmarks
  • email

Tags: ··········

  • Ken,

    Did you ever resolve the issues with the vendors?
  • I think the fundamental answer here is that hooking a vacuum cleaner up to the Internet is very difficult (even for Google). Any social media monitoring software is going to have to make choices, which impact categories of coverage. You're likely seeing some of those programmatic choices in your data.
  • Randall Garcia
    Ken,

    Interesting article. Looking forward to hearing the updates, since my agency went through the same experience of inconsistent results. We had been using Radian6, but after seeing several presentations, including TNS Cymfony and WoolLabs Web Dig, we settled with the latter. I won't go into why we didn't choose the others (frankly, the differences made my head spin), but we did go with Wool's product because of the sheer depth of data. Real data. It isn't the prettiest GUI, but they said they are working on it. If you haven't heard of them, it is worth checking it out. We started with them for a 3 month trial, and have now signed on for an on-going engagement.

    Good luck!

    Randy
  • NewMediAroused
    Glad to hear R6 andSM2 have contacted you to work through things. I too use both of these tools, in tandem, for the same monitoring on a regular basis. You will find that many of the differences are a necessary by-product of how each service filters for spam and identifies "valid" posts.

    More importantly, both services allow you to import RSS feeds, so it is possible to combine the two into a de-duped, aggregate report (with some fussing with it). Personally, I like to have the final product come through Techrigy because they allow me to download the full content of all posts into a database that I can analyze on my own machine.

    Plug R6 into SM2, then add in a well developed Yahoo! Pipes feed as well, set your Filters wisely, and you've got the industry's best listening platform....

    Paul Dyer
    http://www.dyersituations.com
  • You're right Paul, the best platform right now would be a mix of the best features & coverage of many different platforms. While it's possible to do this now, the barrier for entry is higher. Not all organizations or individuals have the knowledge, resources and time to implement such a solution. However, for DIYers, there is light at the end of the tunnel. Thanks for the comment/visit.
  • Great post Ken, equally so the discussion you've sparked here!

    Finding the most relevant and interesting "chatter" whilst weeding out the spam will always be different one provider to the next, which solution(s) can provide the information you need in a timely and clean fashion?

    There is so much information available to us right now that filtering out the noise and finding the juicy bits is as big a challenge as getting complete coverage.

    Zak
    www.moreover.com
  • UPDATE: I've spent the last several days since this post was published working closely with the helpful and friendly folks at both Radian6 and Techrigy. We investigated the results, made some changes to configurations and re-run the searches. While this closed the gap between tools, there are still differences that remain, and are worth discussing.

    I'll be publishing the results, observations and thoughts about both tools in a new post soon.
  • Ken, you started a terrific discussion here that is all so timely with the growing urgency of social media monitoring for any type of online marketing nowadays. In our agency's experience with these tools (esp. Radian6 vis-a-vis Alterian/Techrigy SM2), we have come across both quantitative and qualitative differences, especially as these tools attempt to gauge more advanced metrics like conversation sentiments, engagement levels, influencer ratings, and so on.

    Frankly, I doubt you will be able to get a bottom line answer as to what these differences are attributed to. As noted by a number of participants in this discussion, the same issue applies to web analytics tools and search engines. So, what we usually do is rely on not just one, but a number of (paid and free) tools for social media monitoring and each time we go over the results we are getting with respect to the specific goal at hand. The social media monitoring tools are of great help (and will be of even more help as these technologies mature), but their use still requires significant oversight to ensure useful analysis and effective action taking.
  • Alexandros,

    You make great points, that I concur with. As much as I would like to see less complexity in choosing/operationalizing a monitoring tool, the reality is that we need to rely on many sources. I'd enjoy a conversation with you to learn a bit more about the differences you've observed.
  • Well, I've been watching all of these comments and found a couple of key points very interesting.

    1. The focus is on two of the many, many monitoring / analytic products and services out there. They all do things differently just like every responder to this post does things differently from their competitors. If each of you were given the same question by a client, you'd all likely have a different answer because you all do things differently to remain competitive.

    2. No search engine (Social Media Monitoring or otherwise) returns the same responses nor holds the same data. Why didn't you also ask Microsoft, Yahoo and Google to respond?

    The real issue is to know enough about your chosen tool provider(s) to ask the right questions to ensure you get the informationyou needed for your specific project. This is one key reason why the KDPaine's, Nielsen's, Cymfony's etc of the world attack the monitoring packages on a regular basis. In my opinion, too many people rely on the output of these systems without truly understanding their customer requirements, the system capabilties and to a certain extent, its shortcomings.
  • Steve - In principle I agree with the argument that "many people rely on the output of these systems without truly understanding their customer requirements". However, how does a customer get that level of understanding about the system capabilities/shortcomings before selecting one? That information isn't exactly the type you'll find in marketing material.

    Trial and error? Research on this is limited at best right now, given how young the space is. We're all learning what works/what doesn't together.
  • At first glance I'd guess these variations are in how the media are classified- as the total number of results is not that far apart, 688 for R6 vs. 822 for SM2. The two system work very differently with SM2 having a longer historical database (about 2.5 years for some sources) that it pulls from. I would compare the new results for each daily as they come in or set a date range- SM2 allows this, not sure about R6 (I am former marketing director for Techrigy but otherwise not affiliated with them).
    The other comparison that is far more important is the total amount of meta-data collected for each result. This meta data enables the wide range of analysis that the tools offer. Sheer results are meaningless without the ability to discover demographic, sentiment, emotive, popularity/influence, etc. of those results.
  • Ken, I like the premise of this post.

    I think it's important for any marketer looking at purchasing any type of marketing systems -- whether monitoring, marketing automation, content management, customer relationship management, etc. -- to keep in mind.

    I know as marketers we tend to want to focus on the bottom line, but the details matter. And this means we need to understand 1.) how different platforms approach the task they are focused on managing/automating/etc. and 2.) what the underlying data is that they are using.

    We need to be smarter buyers of marketing technology ... and we need to think more strategically about leveraging and purchasing marketing technology.

    This -- in my mind -- is as much about a 'call to arms' for marketers as it is about vendor capabilities. We need to balance our strategic and operating mindsets as marketers with a system mindset as we build out our systems for complex, integrated marketing management.

    I know this is a topic that Scott Brinker has touched on via his Chief Marketing Technologist blog (http://www.chiefmartec.com/), and I think it's a great call-out here.

    Great dialogue -- and great start at raising the tech IQ among us marketers.
  • jimschwab
    Hi Ken, great discussion here, as former SVP at Harris Interactive a market research firm and current VP at Techrigy a couple comments...

    - Your application should determine how important content integrity, accuracy or representativeness are.... our clients use SM2 for everything from sales lead gen to qualitative market research, and many others
    - All data/content have caveats, you need to understand your business intelligence sources caveats so you can make the most informed decision
    - I have seen the best and most scientific data gathering conducted only to be twisted into whatever conclusion the analyst wants, it happens....
    - If your business decision is a big one, use multiple sources of intell mixed in with your gut and experience

    Techrigy has spent 3+ years understanding how to collect, store and help users understand SM content, it is a very complex technology challenge and it's still an early market. We have documentation, training vidoes and other tools to help our users best understand the content they gather. Happy to share these with you all.
    Jim Schwab
    VP Sales & Marketing
  • terryfoster
    Ken

    This has long been a problem in the media monitoring industry in general as no two companies approach it in the same way and it is nearly impossible for them to replicate each others data sets exactly. For the social media monitoring companies, they often struggle to identify where the influential and relevant "conversations" are taking place. There is also the issue with how much of any given site is spidered and as well as how often. These factors, amongst other things, can affect the end results in a profound way.
  • leonchaddock
    This is very interesting. As others have said the technology varies so much from provider to provider. Taking this away from social media monitoring try comparing the results in Google to those in Yahoo, or google blog search to technorati. Completely different results in terms of volume.

    As others have pointed out spam filters can play a major role in changing these numbers aswell.

    What is key if you are looking at using a vendor for social media monitoring that you compare the results across several at a time to see what works best for you.

    We would recommend you add us to the list of vendors to compare!

    Thanks for this great subject matter!

    Leon
    www.sentimentmetrics.com
  • While I do think it is prudent to understand the data sources that each of these companies use to compile their solutions this is not just an issue in social media measurement. If you look at the broader more mature Web analytics market the same issue exists. Google Analytics, Sitemeter, Alexa, and Compete to name a few off the top of my head all produce different results. Measuring activity on the social web is much more challenging and getting matching data sets across a number of different providers is not going to happen in the near term.
  • There's also the issue of how Twitter "sentiment analysis" is done. It looks like many of the Twitter sentiment analysis tools simply count the number of happy-face emoticons as "positive", frowny-face emoticons as "negative", and emoticon-free tweets as "neutral". This is what Twitter search apparently does, and these tools simply copy the Twitter search "algorithm".

    While there's no "perfect" sentiment analysis algorithm yet, there are some that are *much* better than counting emoticons! If you're going to market a tool with a claim of "Twitter sentiment analysis", I think you should disclose the algorithm.
  • I am definitely interested in what both sides have to say about this. What happens if you are a small business and can not afford to have both but both are valid in your business? How do you decide which is best? I think this is where they brains of both should explain why they monitor their stuff one way and why the other does it another. I realize there aren't standards but how do you choose the best thing for your strategy?
  • jenniferzeszut
    Ken, at least in the case of Scout Labs (another provider to add to your list), we have robust spam / porn detection filters, which is so critical for our many big corporate clients. That's another thing to ask your provider: what are they doing to filter the unwashed social media masses? That may account for some count differences in counts. Of course, we allow you to show or hide the spam we find, so its no mystery. Thanks for going one level deeper!
    Jenny
  • Jennifer - Thanks for tossing another hat into the ring here. I will take a look at Scott Labs for comparison.
  • I echo Josh's comments. No monitoring service is full proof.

    I'm definitely checking back to see how each side responds. :)
  • You're right, no one tool is enough/ fool proof. Thus, it is increasingly important for marketers to understand what each tools limitations are, and what combination of tools is needed to effectively provide coverage.
  • Twitter Comment






    Why it's important to check how well your Social Media Monitoring tool is tracking different sources [link to post] (via @miketrap)

    - Posted using Chat Catcher
  • Twitter Comment






    Interesting data RT @dariasteigman: Terrific disc. by @kenburbary on different results from various SM monitoring tools: [link to post]

    - Posted using Chat Catcher
  • Twitter Comment






    Terrific discussion started by @kenburbary on differences among results from various SM monitoring tools: [link to post]

    - Posted using Chat Catcher
  • Twitter Comment






    but what does it all mean?! @bivings RT @kenburbary: The Dirty Little Secret of Social Media Monitoring - [link to post]

    - Posted using Chat Catcher
  • Twitter Comment






    @kenburbary We'll definitely drop by and add a comment. Stay tuned...

    - Posted using Chat Catcher
  • Wow, that's kind of crazy. I'll be watching to see if you get this all sorted out for the rest of us! Thanks for shedding some light, Ken and those who've already answered.
  • Twitter Comment






    The Dirty Little Secret of Social Media Monitoring - [link to post] (via @kenburbary)

    - Posted using Chat Catcher
  • Twitter Comment






    RT @adamcohen: I'd like to see @AmberCadabra and @cbensen arm wrassle over @kenburbary's latest post [link to post]. In good fun.

    - Posted using Chat Catcher
  • Twitter Comment






    I'd like to see @AmberCadabra and @cbensen arm wrassle over @kenburbary's latest post [link to post]. In good fun. Yes I said wrassle.

    - Posted using Chat Catcher
  • connie_techrigy
    I think that there are many factors to be considered.
    You could export the results for the blog posts from ea tool & compare them (if you have some time)

    We have been asked this question & here are my answers:

    1. our spam filter is different than other products. I doubt that there will ever be a 'standardized' filter set across tools
    (when I was in library world there were various tools that we could use to meet the CIPA filtering requirements, but again, each accomplished the goal in a different way)

    2. there isn't a standardized set of 'sources' either. Our method of collection is different from other tools. Our customers can add specific sources and use tools like Yahoo pipes to add the add'l sources that they require.

    3. specific use of the tool - what filters are being applied in SM2? that may affect the data; what URL's are marked as spam by the user?

    We devote a specific amt of dev't time to finding new sources. And we're continually evaluating what is spam or not. The social web is everchanging.
  • Connie, Thank you for taking the time to respond.

    This helps better understand some of the specifics of how SM2 is handling the searches. That, coupled with the support I'm getting from @jimmyrey are painting a better picture that I will elaborate on in an update to this post after the dust is settled. Oh, and for the purposes of our search, we were not filtering any sources.
  • I'm digging in now :)

    Jim Reynolds AKA Jimmyrey
    Director Of Sales
    Techrigy
  • Thanks for all your help/support on this issue today Jim, greatly appreciated.
  • Twitter Comment






    Interested to see the conversation that comes from this (from @KenBurbary): [link to post] (sketchy shortened URL, I know)

    - Posted using Chat Catcher
  • Twitter Comment






    RT @adamcohen: I'd like to see @AmberCadabra and @cbensen arm wrassle over @kenburbary's latest post [link to post]. In good fun | Me2!

    - Posted using Chat Catcher
  • Twitter Comment






    Most measurement comps use the same data, Radian6 and Techrigy SM2 different [link to post] (via @kenburbary)

    - Posted using Chat Catcher
  • I'm just learning about the various listening tools so this data is greek to me however I find it interesting that there is such a variation between the two you compared. I look forward to hearing from others, especially those involved so that I can better understand this topic. Thank you for sheding the light. We always talk about, What's next? To me, the data being created and monitored, analytics...that's what is next for serious sm users.
  • Twitter Comment






    RT @kenburbary: New Blog Post: The Dirty Little Secret of Social Media Monitoring - [link to post]

    - Posted using Chat Catcher
  • Hi Ken,

    Great post. I think your experiment solidifies how important it is to not rely solely on only one form of social media monitoring. There are definitely differences between the main market leaders right now. The space is huge, and the technology, channels and discussion is constantly changing. To expect that one tool can get everything you need would be underestimating the space. When I structure a social media monitoring program, I couple Radian6 with several outside search tools (Google, BlogPulse and others) to be sure we’re picking up everything we possibly can and giving the client the most accurate, relevant information.

    -Josh
  • Good points Josh. Exactly what I'm trying to bring to the surface with this post. As the social marketing activities mature, I think we'll have more established toolsets, a combination of search tools, that get deployed as a standard practice to give brands the best view of the landscape as possible. I don't see any single tool being able to either.
  • Twitter Comment






    RT @kenburbary: New Blog Post: The Dirty Little Secret of Social Media Monitoring - [link to post] [they spider different sites]

    - Posted using Chat Catcher
  • Twitter Comment






    @davidalston @ambercadabra @jimmyrey @cbensen - Can you solve this monitoring riddle? [link to post]

    - Posted using Chat Catcher
  • Twitter Comment






    RT @kenburbary: New Blog Post: The Dirty Little Secret of Social Media Monitoring - [link to post] [David]

    - Posted using Chat Catcher
  • Ken - hard to explain the individual site differences so i'll leave that to the respective providers. What I will say is that they both are likely to be crawling different sites.

    Further, I'd argue there is always an inherent margin of error in monitoring. It almost doesn't matter what tool and terms you utilize. Google News, Factiva, Lexis-Nexis, Radian6, Techrigy, whatever. There are always posts/articles/tweets that will elude the spiders. Perhaps they will come in a day or two, but expecting 100% capture from any service is just unrealistic in my view.
  • Chuck, thanks for stopping by. That's the obvious, and probably correct, answer. However, if so, it has important implications to brands/companies trying to decide which monitoring solution to use.

    I can't help but feel like I'm picking a cell phone provider/plan. Who has the best coverage for blogs? forums? twitter? etc... These could be very important details due to the strategies/actionable outcomes that are derived from the insights monitoring reveals.
  • Hey Ken - To your second paragraph here, the real danger is that the chances of us knowing who has the best coverage for blogs, forums or twitter beforehand is probably pretty small. That's what I'd be most concerned with.

    Anyway, I look forward to the update from R6 and Techrigy.
  • Twitter Comment






    RT @kenburbary The Dirty Little Secret of Social Media Monitoring | Web Business by Ken Burbary [link to post]

    - Posted using Chat Catcher
  • Twitter Comment






    RT @kenburbary: The Dirty Little Secret of Social Media Monitoring: [link to post] #socialmedia #PR

    - Posted using Chat Catcher
  • Twitter Comment






    New Blog Post: The Dirty Little Secret of Social Media Monitoring - [link to post]

    - Posted using Chat Catcher
blog comments powered by Disqus