Small data problems in political research: a critical replication study

09/27/2021
by   Hugo de Vos, et al.
0

In an often-cited 2019 paper on the use of machine learning in political research, Anastasopoulos Whitford (A W) propose a text classification method for tweets related to organizational reputation. The aim of their paper was to provide a 'guide to practice' for public administration scholars and practitioners on the use of machine learning. In the current paper we follow up on that work with a replication of A W's experiments and additional analyses on model stability and the effects of preprocessing, both in relation to the small data size. We show that (1) the small data causes the classification model to be highly sensitive to variations in the random train-test split, and that (2) the applied preprocessing causes the data to be extremely sparse, with the majority of items in the data having at most two non-zero lexical features. With additional experiments in which we vary the steps of the preprocessing pipeline, we show that the small data size keeps causing problems, irrespective of the preprocessing choices. Based on our findings, we argue that A W's conclusions regarding the automated classification of organizational reputation tweets – either substantive or methodological – can not be maintained and require a larger data set for training and more careful validation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/11/2022

Top Gear or Black Mirror: Inferring Political Leaning From Non-Political Content

Polarization and echo chambers are often studied in the context of expli...
research
07/01/2022

Panning for gold: Lessons learned from the platform-agnostic automated detection of political content in textual data

The growing availability of data about online information behaviour enab...
research
05/06/2022

Combining Humor and Sarcasm for Improving Political Parody Detection

Parody is a figurative device used for mimicking entities for comedic or...
research
04/04/2022

A pipeline and comparative study of 12 machine learning models for text classification

Text-based communication is highly favoured as a communication method, e...
research
05/04/2020

Storing, preprocessing and analyzing Tweets: Finding the suitable NoSQL system

NoSQL systems are a new generation of databases that aim to handle a lar...
research
05/11/2018

Machine Learning for Public Administration Research, with Application to Organizational Reputation

Machine learning methods have gained a great deal of popularity in recen...
research
04/18/2023

Understand Data Preprocessing for Effective End-to-End Training of Deep Neural Networks

In this paper, we primarily focus on understanding the data preprocessin...

Please sign up or login with your details

Forgot password? Click here to reset