iSEA: An Interactive Pipeline for Semantic Error Analysis of NLP Models

03/08/2022
by   Jun Yuan, et al.
9

Error analysis in NLP models is essential to successful model development and deployment. One common approach for diagnosing errors is to identify subpopulations in the dataset where the model produces the most errors. However, existing approaches typically define subpopulations based on pre-defined features, which requires users to form hypotheses of errors in advance. To complement these approaches, we propose iSEA, an Interactive Pipeline for Semantic Error Analysis in NLP Models, which automatically discovers semantically-grounded subpopulations with high error rates in the context of a human-in-the-loop interactive system. iSEA enables model developers to learn more about their model errors through discovered subpopulations, validate the sources of errors through interactive analysis on the discovered subpopulations, and test hypotheses about model errors by defining custom subpopulations. The tool supports semantic descriptions of error-prone subpopulations at the token and concept level, as well as pre-defined higher-level features. Through use cases and expert interviews, we demonstrate how iSEA can assist error understanding and analysis.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/02/2023

NLP Workbench: Efficient and Extensible Integration of State-of-the-art Text Mining Tools

NLP Workbench is a web-based platform for text mining that allows non-ex...
research
11/22/2020

GNNVis: A Visual Analytics Approach for Prediction Error Diagnosis of Graph Neural Networks

Graph Neural Networks (GNNs) aim to extend deep learning techniques to g...
research
10/11/2022

SEAL : Interactive Tool for Systematic Error Analysis and Labeling

With the advent of Transformers, large language models (LLMs) have satur...
research
06/02/2021

When and Why does a Model Fail? A Human-in-the-loop Error Detection Framework for Sentiment Analysis

Although deep neural networks have been widely employed and proven effec...
research
01/13/2022

Towards Automated Error Analysis: Learning to Characterize Errors

Characterizing the patterns of errors that a system makes helps research...
research
10/20/2014

Interactive Error Correction in Implicative Theories

Errors in implicative theories coming from binary data are studied. Firs...
research
05/14/2018

Functional Baby Talk: Analysis of Code Fragments from Novice Haskell Programmers

What kinds of mistakes are made by novice Haskell developers, as they le...

Please sign up or login with your details

Forgot password? Click here to reset