Measuring Domain Portability and ErrorPropagation in Biomedical QA

09/12/2019
by   Stefan Hosein, et al.
0

In this work we present Google's submission to the BioASQ 7 biomedical question answering (QA) task (specifically Task 7b, Phase B). The core of our systems are based on BERT QA models, specifically the model of alberti2019bert. In this report, and via our submissions, we aimed to investigate two research questions. We start by studying how domain portable are QA systems that have been pre-trained and fine-tuned on general texts, e.g., Wikipedia. We measure this via two submissions. The first is a non-adapted model that uses a public pre-trained BERT model and is fine-tuned on the Natural Questions data set kwiatkowski2019natural. The second system takes this non-adapted model and fine-tunes it with the BioASQ training data. Next, we study the impact of error propagation in end-to-end retrieval and QA systems. Again we test this via two submissions. The first uses human annotated relevant documents and snippets as input to the model and the second predicted documents and snippets. Our main findings are that domain specific fine-tuning can benefit Biomedical QA. However, the biggest quality bottleneck is at the retrieval stage, where we see large drops in metrics -- over 10pts absolute -- when using non gold inputs to the QA model.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/26/2017

Neural Question Answering at BioASQ 5B

This paper describes our submission to the 2017 BioASQ challenge. We par...
research
02/24/2020

Predicting Subjective Features from Questions on QA Websites using BERT

Modern Question-Answering websites, such as StackOverflow and Quora, hav...
research
06/16/2022

An Open-Domain QA System for e-Governance

The paper presents an open-domain Question Answering system for Romanian...
research
11/30/2022

A Pipeline for Generating, Annotating and Employing Synthetic Data for Real World Question Answering

Question Answering (QA) is a growing area of research, often used to fac...
research
07/31/2023

Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering

Retriever-augmented instruction-following models are attractive alternat...
research
09/10/2021

Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT

Infusing factual knowledge into pre-trained models is fundamental for ma...
research
07/01/2020

Transferability of Natural Language Inference to Biomedical Question Answering

Biomedical question answering (QA) is a challenging problem due to the s...

Please sign up or login with your details

Forgot password? Click here to reset