DeepAI AI Chat
Log In Sign Up

Keyphrase Generation with Fine-Grained Evaluation-Guided Reinforcement Learning

by   Yichao Luo, et al.

Aiming to generate a set of keyphrases, Keyphrase Generation (KG) is a classical task for capturing the central idea from a given document. Typically, traditional KG evaluation metrics are only aware of the exact correctness of predictions on phrase-level and ignores the semantic similarities between similar predictions and targets, which inhibits the model from learning deep linguistic patterns. In this paper, we propose a new fine-grained evaluation metric that considers different granularity: token-level F_1 score, edit distance, duplication, and prediction quantities. For learning more recessive linguistic patterns, we use a pre-trained model (e.g., BERT) to compute the continuous similarity score between predicted keyphrases and target keyphrases. On the whole, we propose a two-stage Reinforcement Learning (RL) training framework with two reward functions: our proposed fine-grained evaluation score and the vanilla F_1 score. This framework helps the model identifying some partial match phrases which can be further optimized as the exact match ones. Experiments on four KG benchmarks show that our proposed training framework outperforms the traditional RL training frameworks among all evaluation scores. In addition, our method can effectively ease the synonym problem and generate a higher quality prediction.


page 1

page 2

page 3

page 4


Detecting Fine-Grained Cross-Lingual Semantic Divergences without Supervision by Learning to Rank

Detecting fine-grained differences in content conveyed in different lang...

SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation

This paper introduces a novel Self-supervised Fine-grained Dialogue Eval...

Reinforcement Learning to Rank with Coarse-grained Labels

Ranking lies at the core of many Information Retrieval (IR) tasks. While...

Clickbait? Sensational Headline Generation with Auto-tuned Reinforcement Learning

Sensational headlines are headlines that capture people's attention and ...

Fine-grained Information Status Classification Using Discourse Context-Aware BERT

Previous work on bridging anaphora recognition (Hou et al., 2013a) casts...

Fine-Grained Socioeconomic Prediction from Satellite Images with Distributional Adjustment

While measuring socioeconomic indicators is critical for local governmen...

Is it worth it? Budget-related evaluation metrics for model selection

Creating a linguistic resource is often done by using a machine learning...