Some people aren't worth listening to: periodically retraining classifiers with feedback from a team of end users

04/27/2020
by   Joshua Lockhart, et al.
13

Document classification is ubiquitous in a business setting, but often the end users of a classifier are engaged in an ongoing feedback-retrain loop with the team that maintain it. We consider this feedback-retrain loop from a multi-agent point of view, considering the end users as autonomous agents that provide feedback on the labelled data provided by the classifier. This allows us to examine the effect on the classifier's performance of unreliable end users who provide incorrect feedback. We demonstrate a classifier that can learn which users tend to be unreliable, filtering their feedback out of the loop, thus improving performance in subsequent iterations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/12/2020

SURF: Improving classifiers in production by learning from busy and noisy end users

Supervised learning classifiers inevitably make mistakes in production, ...
research
07/25/2020

Feedback Loop and Bias Amplification in Recommender Systems

Recommendation algorithms are known to suffer from popularity bias; a fe...
research
03/17/2023

Feedback Effect in User Interaction with Intelligent Assistants: Delayed Engagement, Adaption and Drop-out

With the growing popularity of intelligent assistants (IAs), evaluating ...
research
10/27/2016

Icon: An Interactive Approach to Train Deep Neural Networks for Segmentation of Neuronal Structures

We present an interactive approach to train a deep neural network pixel ...
research
03/08/2012

Multi source feedback based performance appraisal system using Fuzzy logic decision support system

In Multi-Source Feedback or 360 Degree Feedback, data on the performance...
research
10/07/2021

Improving Prediction Confidence in Learning-Enabled Autonomous Systems

Autonomous systems use extensively learning-enabled components such as d...
research
06/29/2017

Runaway Feedback Loops in Predictive Policing

Predictive policing systems are increasingly used to determine how to al...

Please sign up or login with your details

Forgot password? Click here to reset