Towards objectively evaluating the quality of generated medical summaries

04/09/2021
by   Francesco Moramarco, et al.
0

We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

06/22/2020

Shared Task on Evaluating Accuracy in Natural Language Generation

We propose a shared task on methodologies and algorithms for evaluating ...
11/08/2020

A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems

Most Natural Language Generation systems need to produce accurate texts....
11/21/2020

Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference

A major challenge in evaluating data-to-text (D2T) generation is measuri...
03/08/2020

Better Captioning with Sequence-Level Exploration

Sequence-level learning objective has been widely used in captioning tas...
05/30/2019

Assessing The Factual Accuracy of Generated Text

We propose a model-based metric to estimate the factual accuracy of gene...
09/01/2019

Employ Multimodal Machine Learning for Content quality analysis

The task of identifying high-quality content becomes increasingly import...
10/05/2021

The Quality of the 2020 Census: An Independent Assessment of Census Bureau Activities Critical to Data Quality

This report summarizes major findings from an independent evaluation of ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.