Investigating label suggestions for opinion mining in German Covid-19 social media

05/27/2021
by   Tilman Beck, et al.
0

This work investigates the use of interactively updated label suggestions to improve upon the efficiency of gathering annotations on the task of opinion mining in German Covid-19 social media data. We develop guidelines to conduct a controlled annotation study with social science students and find that suggestions from a model trained on a small, expert-annotated dataset already lead to a substantial improvement - in terms of inter-annotator agreement(+.14 Fleiss' κ) and annotation quality - compared to students that do not receive any label suggestions. We further find that label suggestions from interactively trained models do not lead to an improvement over suggestions from a static model. Nonetheless, our analysis of suggestion bias shows that annotators remain capable of reflecting upon the suggested label in general. Finally, we confirm the quality of the annotated data in transfer learning experiments between different annotator groups. To facilitate further research in opinion mining on social media data, we release our collected data consisting of 200 expert and 2,785 student annotations.

READ FULL TEXT

Authors

page 6

page 8

page 13

12/04/2020

FinnSentiment – A Finnish Social Media Corpus for Sentiment Polarity Annotation

Sentiment analysis and opinion mining is an important task with obvious ...
06/05/2020

Stance Detection on Social Media: State of the Art and Trends

Stance detection on social media is an emerging opinion mining paradigm ...
08/23/2018

Guidelines and Annotation Framework for Arabic Author Profiling

In this paper, we present the annotation pipeline and the guidelines we ...
09/03/2019

Annotation and Classification of Sentence-level Revision Improvement

Studies of writing revisions rarely focus on revision quality. To addres...
06/04/2021

Annotation Curricula to Implicitly Train Non-Expert Annotators

Annotation studies often require annotators to familiarize themselves wi...
07/24/2019

Investigating Correlations of Inter-coder Agreement and Machine Annotation Performance for Historical Video Data

Video indexing approaches such as visual concept classification and pers...
03/18/2020

How social feedback processing in the brain shapes collective opinion processes in the era of social media

What are the mechanisms by which groups with certain opinions gain publi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.