DeepAI AI Chat
Log In Sign Up

Exploring BERT Parameter Efficiency on the Stanford Question Answering Dataset v2.0

by   Eric Hulburd, et al.
berkeley college

In this paper we explore the parameter efficiency of BERT arXiv:1810.04805 on version 2.0 of the Stanford Question Answering dataset (SQuAD2.0). We evaluate the parameter efficiency of BERT while freezing a varying number of final transformer layers as well as including the adapter layers proposed in arXiv:1902.00751. Additionally, we experiment with the use of context-aware convolutional (CACNN) filters, as described in arXiv:1709.08294v3, as a final augmentation layer for the SQuAD2.0 tasks. This exploration is motivated in part by arXiv:1907.10597, which made a compelling case for broadening the evaluation criteria of artificial intelligence models to include various measures of resource efficiency. While we do not evaluate these models based on their floating point operation efficiency as proposed in arXiv:1907.10597, we examine efficiency with respect to training time, inference time, and total number of model parameters. Our results largely corroborate those of arXiv:1902.00751 for adapter modules, while also demonstrating that gains in F1 score from adding context-aware convolutional filters are not practical due to the increase in training and inference time.


page 1

page 2

page 3

page 4


Contextual Aware Joint Probability Model Towards Question Answering System

In this paper, we address the question answering challenge with the SQuA...

BERTVision – A Parameter-Efficient Approach for Question Answering

We present a highly parameter efficient approach for Question Answering ...

Ensemble ALBERT on SQuAD 2.0

Machine question answering is an essential yet challenging task in natur...

A Study of BERT for Non-Factoid Question-Answering under Passage Length Constraints

We study the use of BERT for non-factoid question-answering, focusing on...

Building a Question and Answer System for News Domain

This project attempts to build a Question- Answering system in the News ...

Pruning a BERT-based Question Answering Model

We investigate compressing a BERT-based question answering system by pru...

Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length

Limited computational budgets often prevent transformers from being used...