Keyphrase Generation with Fine-Grained Evaluation-Guided Reinforcement Learning

04/18/2021
by   Yichao Luo, et al.
0

Aiming to generate a set of keyphrases, Keyphrase Generation (KG) is a classical task for capturing the central idea from a given document. Typically, traditional KG evaluation metrics are only aware of the exact correctness of predictions on phrase-level and ignores the semantic similarities between similar predictions and targets, which inhibits the model from learning deep linguistic patterns. In this paper, we propose a new fine-grained evaluation metric that considers different granularity: token-level F_1 score, edit distance, duplication, and prediction quantities. For learning more recessive linguistic patterns, we use a pre-trained model (e.g., BERT) to compute the continuous similarity score between predicted keyphrases and target keyphrases. On the whole, we propose a two-stage Reinforcement Learning (RL) training framework with two reward functions: our proposed fine-grained evaluation score and the vanilla F_1 score. This framework helps the model identifying some partial match phrases which can be further optimized as the exact match ones. Experiments on four KG benchmarks show that our proposed training framework outperforms the traditional RL training frameworks among all evaluation scores. In addition, our method can effectively ease the synonym problem and generate a higher quality prediction.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/07/2020

Detecting Fine-Grained Cross-Lingual Semantic Divergences without Supervision by Learning to Rank

Detecting fine-grained differences in content conveyed in different lang...
research
08/17/2022

SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation

This paper introduces a novel Self-supervised Fine-grained Dialogue Eval...
research
08/16/2022

Reinforcement Learning to Rank with Coarse-grained Labels

Ranking lies at the core of many Information Retrieval (IR) tasks. While...
research
09/09/2019

Clickbait? Sensational Headline Generation with Auto-tuned Reinforcement Learning

Sensational headlines are headlines that capture people's attention and ...
research
10/26/2020

Fine-grained Information Status Classification Using Discourse Context-Aware BERT

Previous work on bridging anaphora recognition (Hou et al., 2013a) casts...
research
08/30/2023

Fine-Grained Socioeconomic Prediction from Satellite Images with Distributional Adjustment

While measuring socioeconomic indicators is critical for local governmen...
research
07/18/2018

Is it worth it? Budget-related evaluation metrics for model selection

Creating a linguistic resource is often done by using a machine learning...

Please sign up or login with your details

Forgot password? Click here to reset