A tool for evaluating Sentiment Analysis REST APIs

There is a growing number of Sentiment Analysis REST APIs out there, and the potential user is faced with a lot of choice. Accuracy of analysis is the most important factor, and the best way to see if an analyzer will perform well in the intended task, is to run different analyzers on a sample of your data, and compare their output with manually assigned sentiment labels.

To make it easier for potential users to run such experiments, we’ve released a small open-source project. The project implements clients to several Sentiment Analyzers: Alchemy, Bitext, Chatterbox, Datumbox, Repustate, Semantria, Skyttle, and Viralheat. As input, it takes a text file with short texts, each annotated as positive, negative or neutral, and outputs a spreadsheet where responses of each API are recorded, as well as an accuracy rate and an error rate calculated against the manual labels.

The project is available on github: https://github.com/skyttle/sentiment-evaluation. Once you clone/unpack it, you will need to install requirements:

pip install -r requirements.txt
pip install -r requirements-testing.txt (optionally, for testing the code)

Next, obtain access keys for each API you’d like to evaluate, and put them into the config.txt file that can be found in the root folder. The file is a two-column text file, where the first column is the name of the API provider and the second is the key.

After that you need to create “gold standard” data, annotating a set of short documents, and saving them to a text file. The text file is a two-column tab-separated file, the first column containing the document and the second the sentiment label, which can be one of “+” (positive), “-” (negative), or “0″ (neutral). The root folder has a example – evaluation_data.txt.

Optionally, comment out APIs should not be included into the comparison in compare.py, ANALYZERS_TO_USE.

To run the library:

compare.py <the name of the file with annotated texts>

The outputs are as follows. Labels assigned to each test document by each API are written in the CSV file called results.cvs.

output-example

To the stdout, the script prints the accuracy rate and the error rate achieved by each API.

Accuracy rate is the proportion of hits (cases when the automatically assigned label is the same the manually assigned one) to the total number of test documents.

Error rate is calculated taking into account whether a neutral label was confused with a positive or negative one (the error has the weight of 1), or a positive label was confused with a negative one (the error has the weight of 2). Error rate is the proportion of the sum of observed weighted errors to the maximum possible sum of weighted errors.

In addition, responses from all APIs are logged to a file.

You are welcome to fork the project and add more analyzers.

If you’ve used the tool in your experiments, please share the results with us!

Finding sentiment associated with entities and keywords

In many application scenarios, knowing that a document expresses positive or negative sentiment is not enough. Much more valuable insights can be obtained if one knows what entities or notions are being evaluated – which product features the consumer liked and which disliked, which are problem areas of a business and which are its strengths, or which aspects of their job the employee is happy or unhappy about. This is especially important in longer texts, such as product reviews, where different aspects of a product can be evaluated differently – some positively and some negatively. If only an overall sentiment score of a comment is calculated, positive and negative comments may cancel each other out, failing to explain the reasons for customer (dis)satisfaction. That’s why phrase-level sentiment analysis will be more helpful here.

There are sentiment analysis systems that require the user to define, in advance, words or names for which sentiment should be tracked. Skyttle is different in that it finds “unknown” keywords – keywords that have not been previously manually defined and input into the system, and recognizes if they are associated with any sentiment. Below we will describe how this can be achieved.

As the first step, you will need to subscribe to one of Skyttle’s subscription plans. Please follow this post on how to do that.

In the POST request sent to the API, you need to set the keywords, sentiment, and annotate parameters to 1 (note that since 3 different services of the API are used in one request, they will be counted as 3 separated calls to the API for Mashape billing purposes):


import sys
import json
import urllib
import urllib2
from pprint import pprint

URL = "https://sentinelprojects-skyttle20.p.mashape.com/"
MASHAPE_AUTH = '<YOUR MASHAPE KEY>'

def main():

    # request parameters
    lang = 'en'
    keywords = 1
    sentiment = 1
    annotate = 1
    domain = None
    text = """We have visited this restaurant a few times in the past, and the 
    meals have been ok, but this time we were deeply disappointed."""

    opener = urllib2.build_opener(urllib2.HTTPHandler)
    params = {'text': text, 'lang': lang, 'keywords': keywords,
              'sentiment': sentiment, 'annotate': annotate, 'domain': domain}
    headers = {'X-Mashape-Authorization': MASHAPE_AUTH}

    request = urllib2.Request(URL, urllib.urlencode(params), headers=headers)
    response = opener.open(request)
    opener.close()
    data = json.loads(response.read())

    pprint(data)

main()

For each document in the request, the response will contain the doc_text field, where the value is the input document, annotated with XML tags for sentiment and keywords:

<doc>
  <s id="0">
    We have visited this
    <k id="84f4966b50704820129d35025588feac9a9c1485" class="term">
      restaurant
    </k>
    a few times in the past,
    <c class="pos" id="1">
      and the
      <k id="af74d302b526b714ceb624cacd0bb9670b0462d0" class="term">
        meals
      </k>
      have been ok
    </c>
    ,
    <c class="neg" id="2">
      but this time we were deeply disappointed.
    </c>
  </s>
</doc>

Here, the c tag annotates stretches of text that express either positive or negative sentiment and the k tag annotates keywords. The annotation now can be used to recognize what sentiment is associated with which keyword. This can be achieved with an XML library, such as lxml for Python:

from lxml import etree
from StringIO import StringIO

parser = etree.XMLParser(recover=True)
root = etree.parse(StringIO(xml_string), parser)
for sentiment_snippet in root.xpath("//c"):
    sentiment_type = sentiment_snippet.attrib['class']
    print "Text:", ''.join(sentiment_snippet.xpath("/text()"))
    for term in sentiment_snippet.xpath('k'):
        term_id = term.attrib['id']
        term_text = ''.join(term.xpath("text()"))
        print 'Sentiment: %s' % sentiment_type
        print 'Keyword: %s (id: %s)' % (term_text, term_id)

Given the XML string above as input, the code above will print this output:

Text: and the meals have been ok
Sentiment: pos
Keyword: meals (id: af74d302b526b714ceb624cacd0bb9670b0462d0)

In other words, the code extracts the association between the keyword “meals” and positive sentiment. The id attribute on the keyword is a unique id that the keyword will have in different documents, even if it has different surface appearance in them, and based on which sentiment towards that keyword can be summarised over many documents.

Here is the full script that reads multiple texts from a file and outputs keywords, ordered by the sentiment they are associated with.

Annotating texts for sentiment and keywords

Skyttle lets you optionally annotate your input text for sentiment and keywords with XML tags.

(Unless you’ve done already, subscribe to Skyttle API on Mashape)

In the request parameters, set annotate to 1. Now if keywords, sentiment, or both, are requested, the response will contain the original text with XML annotations of keywords and/or sentiment. Here is an example request:


import sys
import json
import urllib
import urllib2
from pprint import pprint

URL = "https://sentinelprojects-skyttle20.p.mashape.com/"
MASHAPE_AUTH = '<YOUR MASHAPE KEY>'

def main():

    # request parameters
    lang = 'en'
    keywords = 1
    sentiment = 1
    annotate = 1
    domain = None
    text = "We have visited this restaurant a few times in the past, and the meals have been ok, but this time we were deeply disappointed."

    opener = urllib2.build_opener(urllib2.HTTPHandler)
    params = {'text': text, 'lang': lang, 'keywords': keywords,
              'sentiment': sentiment, 'annotate': annotate, 'domain': domain}
    headers = {'X-Mashape-Authorization': MASHAPE_AUTH}

    request = urllib2.Request(URL, urllib.urlencode(params), headers=headers)
    response = opener.open(request)
    opener.close()
    data = json.loads(response.read())

    pprint(data)

main()

Before running the script, you set your Mashape key in MASHAPE_AUTH.

For each document in the request, the response will contain the doc_text field:

<s id="0">
  We have visited this
  <k id="84f4966b50704820129d35025588feac9a9c1485" class="term">
    restaurant
  </k>
  a few times in the past,
  <c class="pos" id="1">
    and the
    <k id="af74d302b526b714ceb624cacd0bb9670b0462d0" class="term">
      meals
    </k>
    have been ok
  </c>
  ,
  <c class="neg" id="2">
    but this time we were deeply disappointed.
  </c>
</s>

The s tag annotates sentences, the k tag keywords and the c sentiment.

The k tag has two attributes:

  • id, the unique id of the keyword that will be the same for different morphological forms of the keyword (restaurant, restaurants) in the document or across documents, and
  • class, whose value can be “term” or “entity”.

The c tag also has two attributes:

  • id, the unique id of the stretch of text in the document,
  • class, the polarity of the sentiment, can be “pos” or “neg”.

Sentiment Analysis with Skyttle: a walkthrough example

Sentiment Analysis

Sentiment analysis is the task of detecting if a given piece of text conveys positive or negative attitude to an entity or fact it mentions.

There are two major flavors of sentiment analysis – document-level and phrase-level. Document-level analysis assigns an overall sentiment score for the whole document (for example, a score on a scale between -100 and +100). Phrase-level analysis identifies positive or negative emotions expressed in smaller chunks of text – sentences or parts of sentences, first finding such stretches of text and then assigning a score to each. There are strengths and weaknesses to both approaches, and their choice very much depends on the intended application.

Sentiment analysis in Skyttle API was designed to detect sentiment on the fine-grained phrase level. This gives a greater insight into which specific entities, facts or events are viewed with which sentiment. This is important in applications involving longer documents, such as the analysis of customer feedback or complaints, expert reviews, or blog posts. However, the API also determines the overall document scores by summarizing the results of phrase-level analysis.

Subscribe to Skyttle

The steps to perform sentiment analysis using the Skyttle API are:

  1. Create an account on Mashape, their infrastructure will keep track of the number of requests made and deal with subscriptions and billing.
  2. Subscribe to Skyttle, and choose the subscription plan suitable for your needs, selecting from Freemium (free allowance, which you can use for evaluation or small projects), Basic, Premium, or Ultra. If you need a custom pricing plan, get in touch with us. You will need to provide credit card details even if you subscribe to the free plan – you will be charged if you exceed the free allowance.
  3. You can test the API on the Mashape testing console.
  4. To access the API from your code, you will only need the URL (https://sentinelprojects-skyttle20.p.mashape.com/) and a Mashape key, available in the “Keystore” section on the dashboard.

Request Parameters

Here is an example Python script for analysing sentiment in a document.


import sys
import json
import urllib
import urllib2
from pprint import pprint

URL = "https://sentinelprojects-skyttle20.p.mashape.com/"
MASHAPE_AUTH = '<your mashape="" key="">'

def main():

    # request parameters
    lang = 'en'
    keywords = 0
    sentiment = 1
    annotate = 0
    domain = None
    text = """We have visited this restaurant a few times in the past, and the
    meals have been ok, but this time we were deeply disappointed."""

    opener = urllib2.build_opener(urllib2.HTTPHandler)
    params = {'text': text, 'lang': lang, 'keywords': keywords,
              'sentiment': sentiment, 'annotate': annotate, 'domain': domain}
    headers = {'X-Mashape-Authorization': MASHAPE_AUTH}

    request = urllib2.Request(URL, urllib.urlencode(params), headers=headers)
    response = opener.open(request)
    opener.close()
    data = json.loads(response.read())

    pprint(data)


main()


Before running the script, you need to set your Mashape key and the request parameters:

  • lang: the language of the text; ‘en’, ‘fr’, ‘de’, or ‘ru’ (default: ‘en’)
  • sentiment: 1
  • text: the text to analyse

The text field is limited to 10,000 characters. You can send several documents in one call to the API, enclosing them in a doc tag and providing an id attribute for each document:

<doc id="0">This is the first document.</doc><doc id="1">This is the second document.</doc>

Response Fields

The script prints the analysis of the text in the JSON format, to STDOUT:

{u'docs': [{u'doc_id': u'0',
            u'language': u'en',
            u'sentiment': [{u'polarity': u'pos',
                            u'text': u'and the meals have been ok'},
                           {u'polarity': u'neg',
                            u'text': u'but this time we were deeply disappointed.'}],
            u'sentiment_scores': {u'neg': 34.8, u'neu': 42.8, u'pos': 22.4}}],
 u'warnings': []}

The reponse contains the field docs, where the value is a list of documents analysed. For each document, the analysis includes:

  • doc_id: the id of the document,
  • language: the language of the document, and
  • sentiment: a list of sentiment-bearing snippets of text found in the document.
  • sentiment_scores: the sentiment scores of the whole document, the dictionary contains the percentage of positive, negative, and neutral content in the document

For each sentiment snippet, there are two fields:

  • text: the text of the snippet
  • polarity: the polarity of the snippet, positive or negative.

Any warning messages are added to the warnings field of the response.In the next post we will show how one can recognize sentiment associated with particular keywords or entities.

Keyword extraction with Skyttle: a walkthrough example

Keyword Extraction

Keyword extraction, also sometimes referred to as term extraction, terminology extraction, term recognition, keyword analysis, keyphrase extraction, among others, finds single words or multi-word expressions that describe the main notions and entities mentioned in the text.

For example, given this text:

United Nations forces provided attack helicopters and sniper fire to support Congolese troops advancing on rebel positions near the strategic mineral-trading hub of Goma, as the widening conflict continued to inflame tensions with neighboring Rwanda.

Keywords may be: United Nations forces, Congolese troops, rebel, attack helicopters, sniper fire, mineral-trading hub, tensions with Rwanda.

Subscribe to Skyttle

The steps to perform keyword extraction using the Skyttle API are:

  1. Create an account on Mashape, their infrastructure will keep track of the number of requests made and deal with subscriptions and billing.
  2. Subscribe to Skyttle, and choose the subscription plan suitable for your needs, selecting from Freemium (free allowance, which you can use for evaluation or small projects), Basic, Premium, or Ultra. If you need a custom pricing plan, get in touch with us. You will need to provide credit card details even if you subscribe to the free plan – you will be charged if you exceed the free allowance.
  3. You can test the API on the Mashape testing console.
  4. To access the API from your code, you will only need the URL (https://sentinelprojects-skyttle20.p.mashape.com/) and a Mashape key, available in the “Keystore” section on the dashboard.

Request Parameters

Here is an example Python script for extracting keywords from a document.


import sys
import json
import urllib
import urllib2
from pprint import pprint

URL = "https://sentinelprojects-skyttle20.p.mashape.com/"
MASHAPE_AUTH = '<YOUR MASHAPE KEY>'

def main():

    # request parameters
    lang = 'en'
    keywords = 1
    sentiment = 0
    annotate = 0
    domain = None
    text = "We have visited this restaurant a few times in the past, and the meals have been ok, but this time we were deeply disappointed."

    opener = urllib2.build_opener(urllib2.HTTPHandler)
    params = {'text': text, 'lang': lang, 'keywords': keywords,
              'sentiment': sentiment, 'annotate': annotate, 'domain': domain}
    headers = {'X-Mashape-Authorization': MASHAPE_AUTH}

    request = urllib2.Request(URL, urllib.urlencode(params), headers=headers)
    response = opener.open(request)
    opener.close()
    data = json.loads(response.read())

    pprint(data)


main()


Before running the script, you need to set your Mashape key and the request parameters:

  • lang: the language of the text; ‘en’, ‘fr’, ‘de’, or ‘ru’ (default: ‘en’)
  • keywords: 1
  • text: the text to analyse

The text field is limited to 10,000 characters. You can send several documents in one call to the API, enclosing them in a doc tag and providing an id attribute for each document:

<doc id="0">This is the first document.</doc><doc id="1">This is the second document.</doc>

Response Fields

The script prints the analysis of the text in the JSON format, to STDOUT:

{u'docs': [{u'doc_id': u'0',
            u'language': u'en',
            u'terms': [{u'count': 1,
                        u'id': u'af74d302b526b714ceb624cacd0bb9670b0462d0',
                        u'term': u'meals'},
                       {u'count': 1,
                        u'id': u'84f4966b50704820129d35025588feac9a9c1485',
                        u'term': u'restaurant'}]}],
 u'warnings': []}

The reponse contains the field docs, where the value is a list of documents analysed. For each document, the analysis includes:

  • doc_id: the id of the document,
  • language: the language of the document, and
  • terms : a list of extracted keywords.

For each keyword, there are three fields:

  • term: the textual form of the keyword
  • id: a unique id of the keyword that can be used to match identical terms across documents.
  • count: the number of occurrences of the keyword in the document.

Any warning messages are added to the warnings field of the response.