Exploring Out-of-Distribution Generalization in Text Classifiers Trained on Tobacco-3482 and RVL-CDIP

08/05/2021
by   Stefan Larson, et al.
0

To be robust enough for widespread adoption, document analysis systems involving machine learning models must be able to respond correctly to inputs that fall outside of the data distribution that was used to generate the data on which the models were trained. This paper explores the ability of text classifiers trained on standard document classification datasets to generalize to out-of-distribution documents at inference time. We take the Tobacco-3482 and RVL-CDIP datasets as a starting point and generate new out-of-distribution evaluation datasets in order to analyze the generalization performance of models trained on these standard datasets. We find that models trained on the smaller Tobacco-3482 dataset perform poorly on our new out-of-distribution data, while text classification models trained on the larger RVL-CDIP exhibit smaller performance drops.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/14/2022

Evaluating Out-of-Distribution Performance on Document Image Classifiers

The ability of a document classifier to handle inputs that are drawn fro...
research
09/26/2019

Learning the Difference that Makes a Difference with Counterfactually-Augmented Data

Despite alarm over the reliance of machine learning systems on so-called...
research
12/16/2022

Fake it till you make it: Learning(s) from a synthetic ImageNet clone

Recent large-scale image generation models such as Stable Diffusion have...
research
01/20/2023

Transforming Unstructured Text into Data with Context Rule Assisted Machine Learning (CRAML)

We describe a method and new no-code software tools enabling domain expe...
research
03/23/2022

Robust Text Line Detection in Historical Documents: Learning and Evaluation Methods

Text line segmentation is one of the key steps in historical document un...
research
09/20/2023

Construction of Paired Knowledge Graph-Text Datasets Informed by Cyclic Evaluation

Datasets that pair Knowledge Graphs (KG) and text together (KG-T) can be...
research
04/29/2020

Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weighting

With the recent proliferation of the use of text classifications, resear...

Please sign up or login with your details

Forgot password? Click here to reset