DeepAI AI Chat
Log In Sign Up

No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media

by   Maximilian Spliethöver, et al.

News articles both shape and reflect public opinion across the political spectrum. Analyzing them for social bias can thus provide valuable insights, such as prevailing stereotypes in society and the media, which are often adopted by NLP models trained on respective data. Recent work has relied on word embedding bias measures, such as WEAT. However, several representation issues of embeddings can harm the measures' accuracy, including low-resource settings and token frequency differences. In this work, we study what kind of embedding algorithm serves best to accurately measure types of social bias known to exist in US online news articles. To cover the whole spectrum of political bias in the US, we collect 500k articles and review psychology literature with respect to expected social bias. We then quantify social bias using WEAT along with embedding algorithms that account for the aforementioned issues. We compare how models trained with the algorithms on news articles represent the expected social bias. Our results suggest that the standard way to quantify bias does not align well with knowledge from psychology. While the proposed algorithms reduce the gap, they still do not fully match the literature.


page 1

page 2

page 3

page 4


Quantifying Political Bias in News Articles

Search bias analysis is getting more attention in recent years since sea...

Political Depolarization of News Articles Using Attribute-aware Word Embeddings

Political polarization in the US is on the rise. This polarization negat...

Unsupervised Detection of Contextualized Embedding Bias with Application to Ideology

We propose a fully unsupervised method to detect bias in contextualized ...

Argument from Old Man's View: Assessing Social Bias in Argumentation

Social bias in language - towards genders, ethnicities, ages, and other ...

Rating Reliability and Bias in News Articles: Does AI Assistance Help Everyone?

With the spread of false and misleading information in current news, man...

Neural Media Bias Detection Using Distant Supervision With BABE – Bias Annotations By Experts

Media coverage has a substantial effect on the public perception of even...