DeepAI AI Chat
Log In Sign Up

KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning

by   Haonan Li, et al.
The University of Melbourne

Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural language generation tasks with high-quality requirements on the output, such as commonsense generation and ad keyword generation. In this work, we present a novel Knowledge Filtering and Contrastive learning Network (KFCNet) which references external knowledge and achieves better generation performance. Specifically, we propose a BERT-based filter model to remove low-quality candidates, and apply contrastive learning separately to each of the encoder and decoder, within a general encoder–decoder architecture. The encoder contrastive module helps to capture global target semantics during encoding, and the decoder contrastive module enhances the utility of retrieved prototypes while learning general features. Extensive experiments on the CommonGen benchmark show that our model outperforms the previous state of the art by a large margin: +6.6 points (42.5 vs. 35.9) for BLEU-4, +3.7 points (33.3 vs. 29.6) for SPICE, and +1.3 points (18.3 vs. 17.0) for CIDEr. We further verify the effectiveness of the proposed contrastive module on ad keyword generation, and show that our model has potential commercial value.


page 1

page 2

page 3

page 4


Empathetic BERT2BERT Conversational Model: Learning Arabic Language Generation with Little Data

Enabling empathetic behavior in Arabic dialogue agents is an important a...

Contrastive Triple Extraction with Generative Transformer

Triple extraction is an essential task in information extraction for nat...

Empathetic Dialogue Generation with Pre-trained RoBERTa-GPT2 and External Knowledge

One challenge for dialogue agents is to recognize feelings of the conver...

Title-Guided Encoding for Keyphrase Generation

Keyphrase generation (KG) aims to generate a set of keyphrases given a d...

DictBERT: Dictionary Description Knowledge Enhanced Language Model Pre-training via Contrastive Learning

Although pre-trained language models (PLMs) have achieved state-of-the-a...

Fine-grained Contrastive Learning for Definition Generation

Recently, pre-trained transformer-based models have achieved great succe...

Inseq: An Interpretability Toolkit for Sequence Generation Models

Past work in natural language processing interpretability focused mainly...