DeepAI AI Chat
Log In Sign Up

RISE: Leveraging Retrieval Techniques for Summarization Evaluation

by   David Uthus, et al.

Evaluating automatically-generated text summaries is a challenging task. While there have been many interesting approaches, they still fall short of human evaluations. We present RISE, a new approach for evaluating summaries by leveraging techniques from information retrieval. RISE is first trained as a retrieval task using a dual-encoder retrieval setup, and can then be subsequently utilized for evaluating a generated summary given an input document, without gold reference summaries. RISE is especially well suited when working on new datasets where one may not have reference summaries available for evaluation. We conduct comprehensive experiments on the SummEval benchmark (Fabbri et al., 2021) and the results show that RISE has higher correlation with human evaluations compared to many past approaches to summarization evaluation. Furthermore, RISE also demonstrates data-efficiency and generalizability across languages.


page 1

page 2

page 3

page 4


Jointly Extracting and Compressing Documentswith Summary State Representations

We present a new neural model for text summarization that first extracts...

EmailSum: Abstractive Email Thread Summarization

Recent years have brought about an interest in the challenging task of s...

HighRES: Highlight-based Reference-less Evaluation of Summarization

There has been substantial progress in summarization research enabled by...

GeSERA: General-domain Summary Evaluation by Relevance Analysis

We present GeSERA, an open-source improved version of SERA for evaluatin...

Comparing Methods for Extractive Summarization of Call Centre Dialogue

This paper provides results of evaluating some text summarisation techni...

"I Don't Think So": Disagreement-Based Policy Summaries for Comparing Agents

With Artificial Intelligence on the rise, human interaction with autonom...