Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds

08/29/2019
by   John P. Lalor, et al.
0

Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where human expectation about item difficulty does not match difficulty as estimated from the machine RPs.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/15/2017

An Analysis of Ability in Deep Neural Networks

Deep neural networks (DNNs) have made significant progress in a number o...
research
05/28/2016

Building an Evaluation Scale using Item Response Theory

Evaluation of NLP methods requires testing against a previously vetted g...
research
07/30/2023

Text Analysis Using Deep Neural Networks in Digital Humanities and Information Science

Combining computational technologies and humanities is an ongoing effort...
research
06/24/2021

Item Response Thresholds Models

A comprehensive class of models is proposed that can be used for continu...
research
05/11/2023

KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment

Recent legislation of the "right to be forgotten" has led to the interes...
research
08/26/2021

Modeling Item Response Theory with Stochastic Variational Inference

Item Response Theory (IRT) is a ubiquitous model for understanding human...
research
09/04/2022

Quantitative Stopword Generation for Sentiment Analysis via Recursive and Iterative Deletion

Stopwords carry little semantic information and are often removed from t...

Please sign up or login with your details

Forgot password? Click here to reset