RISE: Leveraging Retrieval Techniques for Summarization Evaluation

12/17/2022
by   David Uthus, et al.
0

Evaluating automatically-generated text summaries is a challenging task. While there have been many interesting approaches, they still fall short of human evaluations. We present RISE, a new approach for evaluating summaries by leveraging techniques from information retrieval. RISE is first trained as a retrieval task using a dual-encoder retrieval setup, and can then be subsequently utilized for evaluating a generated summary given an input document, without gold reference summaries. RISE is especially well suited when working on new datasets where one may not have reference summaries available for evaluation. We conduct comprehensive experiments on the SummEval benchmark (Fabbri et al., 2021) and the results show that RISE has higher correlation with human evaluations compared to many past approaches to summarization evaluation. Furthermore, RISE also demonstrates data-efficiency and generalizability across languages.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/18/2023

Summarization is (Almost) Dead

How well can large language models (LLMs) generate summaries? We develop...
research
04/03/2019

Jointly Extracting and Compressing Documentswith Summary State Representations

We present a new neural model for text summarization that first extracts...
research
10/07/2021

GeSERA: General-domain Summary Evaluation by Relevance Analysis

We present GeSERA, an open-source improved version of SERA for evaluatin...
research
03/03/2022

Automatic Text Summarization Methods: A Comprehensive Review

One of the most pressing issues that have arisen due to the rapid growth...
research
06/04/2019

HighRES: Highlight-based Reference-less Evaluation of Summarization

There has been substantial progress in summarization research enabled by...
research
02/05/2021

"I Don't Think So": Disagreement-Based Policy Summaries for Comparing Agents

With Artificial Intelligence on the rise, human interaction with autonom...

Please sign up or login with your details

Forgot password? Click here to reset