Multilingual Twitter Sentiment Classification: The Role of Human Annotators

Mozeti{\v{c}}, Igor; Gr{\v{c}}ar, Miha; Smailovi{\', Jasmina; {c}},
What are the limits of automated Twitter sentiment classification? We analyze a large set of manually labeled tweets in different languages, use them as training data, and construct automated classification models. It turns out that the quality of classification models depends much more on the quality and size of training data than on the type of the model trained. Experimental results indicate that there is no statistically significant difference between the performance of the top classification models. We quantify the quality of training data by applying various annotator agreement measures, and identify the weakest points of different datasets. We show that the model performance approaches the inter-annotator agreement when the size of the training set is sufficiently large. However, it is crucial to regularly monitor the self-and inter-annotator agreements since this improves the training datasets and con-sequently the model performance. Finally, we show that there is strong evidence that humans perceive the sentiment classes (negative, neutral, and positive) as ordered.
Research areas:
Year:
2016
Type of Publication:
Article
Journal:
PLoS ONE
Volume:
11
Number:
5
Pages:
e0155036
DOI:
10.1371/journal.pone.0155036
Hits: 1188

We use cookies to improve our website and your experience when using it. Cookies used for the essential operation of this site have already been set. To find out more about the cookies we use and how to delete them, see our privacy policy.

  I accept cookies from this site.
EU Cookie Directive Module Information