Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples

06/05/2019
by   Kit Kuksenok, et al.
0

We describe and validate a metric for estimating multi-class classifier performance based on cross-validation and adapted for improvement of small, unbalanced natural-language datasets used in chatbot design. Our experiences draw upon building recruitment chatbots that mediate communication between job-seekers and recruiters by exposing the ML/NLP dataset to the recruiting team. Evaluation approaches must be understandable to various stakeholders, and useful for improving chatbot performance. The metric, nex-cv, uses negative examples in the evaluation of text classification, and fulfils three requirements. First, it is actionable: it can be used by non-developer staff. Second, it is not overly optimistic compared to human ratings, making it a fast method for comparing classifiers. Third, it allows model-agnostic comparison, making it useful for comparing systems despite implementation differences. We validate the metric based on seven recruitment-domain datasets in English and German over the course of one year.

READ FULL TEXT
research
03/25/2020

Adversarial Multi-Binary Neural Network for Multi-class Classification

Multi-class text classification is one of the key problems in machine le...
research
11/25/2022

Comparison Study Between Token Classification and Sequence Classification In Text Classification

Unsupervised Machine Learning techniques have been applied to Natural La...
research
06/15/2023

MetricPrompt: Prompting Model as a Relevance Metric for Few-shot Text Classification

Prompting methods have shown impressive performance in a variety of text...
research
11/10/2022

Towards Human-Centred Explainability Benchmarks For Text Classification

Progress on many Natural Language Processing (NLP) tasks, such as text c...
research
10/06/2020

Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder

This paper demonstrates a fatal vulnerability in natural language infere...
research
02/13/2023

Towards Agile Text Classifiers for Everyone

Text-based safety classifiers are widely used for content moderation and...

Please sign up or login with your details

Forgot password? Click here to reset