Using calibrator to improve robustness in Machine Reading Comprehension

02/24/2022
by   Jing Jin, et al.
0

Machine Reading Comprehension(MRC) has achieved a remarkable result since some powerful models, such as BERT, are proposed. However, these models are not robust enough and vulnerable to adversarial input perturbation and generalization examples. Some works tried to improve the performance on specific types of data by adding some related examples into training data while it leads to degradation on the original dataset, because the shift of data distribution makes the answer ranking based on the softmax probability of model unreliable. In this paper, we propose a method to improve the robustness by using a calibrator as the post-hoc reranker, which is implemented based on XGBoost model. The calibrator combines both manual features and representation learning features to rerank candidate results. Experimental results on adversarial datasets show that our model can achieve performance improvement by more than 10% and also make improvement on the original and generalization datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/28/2019

Sogou Machine Reading Comprehension Toolkit

Machine reading comprehension have been intensively studied in recent ye...
research
06/14/2019

Learning to Ask Unanswerable Questions for Machine Reading Comprehension

Machine reading comprehension with unanswerable questions is a challengi...
research
07/01/2021

Ensemble Learning-Based Approach for Improving Generalization Capability of Machine Reading Comprehension Systems

Machine Reading Comprehension (MRC) is an active field in natural langua...
research
05/31/2019

MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension

A large number of reading comprehension (RC) datasets has been created r...
research
11/16/2019

Robust Reading Comprehension with Linguistic Constraints via Posterior Regularization

In spite of great advancements of machine reading comprehension (RC), ex...
research
12/07/2020

Semantics Altering Modifications for Evaluating Comprehension in Machine Reading

Advances in NLP have yielded impressive results for the task of machine ...
research
02/12/2019

Machine Reading Comprehension for Answer Re-Ranking in Customer Support Chatbots

Recent advances in deep neural networks, language modeling and language ...

Please sign up or login with your details

Forgot password? Click here to reset