DeepAI AI Chat
Log In Sign Up

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

10/13/2017
by   Pedram Daee, et al.
0

In human-in-the-loop machine learning, the user provides information beyond that in the training data. Many algorithms and user interfaces have been designed to optimize and facilitate this human--machine interaction; however, fewer studies have addressed the potential defects the designs can cause. Effective interaction often requires exposing the user to the training data or its statistics. The design of the system is then critical, as this can lead to double use of data and overfitting, if the user reinforces noisy patterns in the data. We propose a user modelling methodology, by assuming simple rational behaviour, to correct the problem. We show, in a user study with 48 participants, that the method improves predictive performance in a sparse linear regression sentiment analysis task, where graded user knowledge on feature relevance is elicited. We believe that the key idea of inferring user knowledge with probabilistic user models has general applicability in guarding against overfitting and improving interactive machine learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/26/2019

Benign Overfitting in Linear Regression

The phenomenon of benign overfitting is one of the key mysteries uncover...
12/07/2016

Interactive Elicitation of Knowledge on Feature Relevance Improves Predictions in Small Data Sets

Providing accurate predictions is challenging for machine learning algor...
03/18/2022

But that's not why: Inference adjustment by interactive prototype deselection

Despite significant advances in machine learning, decision-making of art...
12/25/2020

Intuitiveness in Active Teaching

Machine learning is a double-edged sword: it gives rise to astonishing r...
08/08/2022

Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality

Avoiding overfitting is a central challenge in machine learning, yet man...
08/07/2019

A modelling methodology for social interaction experiments

Analysis of temporal network data arising from online interactive social...
05/20/2018

Human-guided data exploration using randomisation

An explorative data analysis system should be aware of what the user alr...

Code Repositories

human-overfitting-in-IML

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction


view repo