Investigating label suggestions for opinion mining in German Covid-19 social media

by   Tilman Beck, et al.

This work investigates the use of interactively updated label suggestions to improve upon the efficiency of gathering annotations on the task of opinion mining in German Covid-19 social media data. We develop guidelines to conduct a controlled annotation study with social science students and find that suggestions from a model trained on a small, expert-annotated dataset already lead to a substantial improvement - in terms of inter-annotator agreement(+.14 Fleiss' κ) and annotation quality - compared to students that do not receive any label suggestions. We further find that label suggestions from interactively trained models do not lead to an improvement over suggestions from a static model. Nonetheless, our analysis of suggestion bias shows that annotators remain capable of reflecting upon the suggested label in general. Finally, we confirm the quality of the annotated data in transfer learning experiments between different annotator groups. To facilitate further research in opinion mining on social media data, we release our collected data consisting of 200 expert and 2,785 student annotations.



page 6

page 8

page 13


FinnSentiment – A Finnish Social Media Corpus for Sentiment Polarity Annotation

Sentiment analysis and opinion mining is an important task with obvious ...

Stance Detection on Social Media: State of the Art and Trends

Stance detection on social media is an emerging opinion mining paradigm ...

Guidelines and Annotation Framework for Arabic Author Profiling

In this paper, we present the annotation pipeline and the guidelines we ...

Annotation and Classification of Sentence-level Revision Improvement

Studies of writing revisions rarely focus on revision quality. To addres...

Annotation Curricula to Implicitly Train Non-Expert Annotators

Annotation studies often require annotators to familiarize themselves wi...

Investigating Correlations of Inter-coder Agreement and Machine Annotation Performance for Historical Video Data

Video indexing approaches such as visual concept classification and pers...

How social feedback processing in the brain shapes collective opinion processes in the era of social media

What are the mechanisms by which groups with certain opinions gain publi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.