DeepAI AI Chat
Log In Sign Up

Towards Human-Centred Explainability Benchmarks For Text Classification

by   Viktor Schlegel, et al.

Progress on many Natural Language Processing (NLP) tasks, such as text classification, is driven by objective, reproducible and scalable evaluation via publicly available benchmarks. However, these are not always representative of real-world scenarios where text classifiers are employed, such as sentiment analysis or misinformation detection. In this position paper, we put forward two points that aim to alleviate this problem. First, we propose to extend text classification benchmarks to evaluate the explainability of text classifiers. We review challenges associated with objectively evaluating the capabilities to produce valid explanations which leads us to the second main point: We propose to ground these benchmarks in human-centred applications, for example by using social media, gamification or to learn explainability metrics from human judgements.


page 1

page 2

page 3

page 4


A Semantic Approach to Negation Detection and Word Disambiguation with Natural Language Processing

This study aims to demonstrate the methods for detecting negations in a ...

A Diagnostic Study of Explainability Techniques for Text Classification

Recent developments in machine learning have introduced models that appr...

SHAP values for Explaining CNN-based Text Classification Models

Deep neural networks are increasingly used in natural language processin...

Towards Prediction Explainability through Sparse Communication

Explainability is a topic of growing importance in NLP. In this work, we...

Weakly Supervised Domain Detection

In this paper we introduce domain detection as a new natural language pr...

Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples

We describe and validate a metric for estimating multi-class classifier ...

What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods

A multitude of explainability methods and theoretical evaluation scores ...