How Auditing Methodologies Can Impact Our Understanding of YouTube's Recommendation Systems

by   Sarmad Chandio, et al.

Data generated by audits of social media websites have formed the basis of our understanding of the biases presented in algorithmic content recommendation systems. As legislators around the world are beginning to consider regulating the algorithmic systems that drive online platforms, it is critical to ensure the correctness of these inferred biases. However, as we will show in this paper, doing so is a challenging task for a variety of reasons related to the complexity of configuration parameters associated with the audits that gather data from a specific platform. Focusing specifically on YouTube, we show that conducting audits to make inferences about YouTube's recommendation systems is more methodologically challenging than one might expect. There are many methodological decisions that need to be considered in order to obtain scientifically valid results, and each of these decisions incur costs. For example, should an auditor use (expensive to obtain) logged-in YouTube accounts while gathering recommendations from the algorithm to obtain more accurate inferences? We explore the impact of this and many other decisions and make some startling discoveries about the methodological choices that impact YouTube's recommendations. Taken all together, our research suggests auditing configuration compromises that YouTube auditors and researchers can use to reduce audit overhead, both economically and computationally, without sacrificing accuracy of their inferences. Similarly, we also identify several configuration parameters that have a significant impact on the accuracy of measured inferences and should be carefully considered.


page 1

page 2

page 3

page 4


Auditing the Biases Enacted by YouTube for Political Topics in Germany

With YouTube's growing importance as a news platform, its recommendation...

Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization

The role that YouTube and its behind-the-scenes recommendation algorithm...

Middle-Aged Video Consumers' Beliefs About Algorithmic Recommendations on YouTube

User beliefs about algorithmic systems are constantly co-produced throug...

Evaluating the scale, growth, and origins of right-wing echo chambers on YouTube

Although it is understudied relative to other social media platforms, Yo...

A Longitudinal Analysis of YouTube's Promotion of Conspiracy Videos

Conspiracy theories have flourished on social media, raising concerns th...

Tubes Bubbles – Topological confinement of YouTube recommendations

The role of recommendation algorithms in online user confinement is at t...

Please sign up or login with your details

Forgot password? Click here to reset