DeepAI AI Chat
Log In Sign Up

Empathetic BERT2BERT Conversational Model: Learning Arabic Language Generation with Little Data

by   Tarek Naous, et al.

Enabling empathetic behavior in Arabic dialogue agents is an important aspect of building human-like conversational models. While Arabic Natural Language Processing has seen significant advances in Natural Language Understanding (NLU) with language models such as AraBERT, Natural Language Generation (NLG) remains a challenge. The shortcomings of NLG encoder-decoder models are primarily due to the lack of Arabic datasets suitable to train NLG models such as conversational agents. To overcome this issue, we propose a transformer-based encoder-decoder initialized with AraBERT parameters. By initializing the weights of the encoder and decoder with AraBERT pre-trained weights, our model was able to leverage knowledge transfer and boost performance in response generation. To enable empathy in our conversational model, we train it using the ArabicEmpatheticDialogues dataset and achieve high performance in empathetic response generation. Specifically, our model achieved a low perplexity value of 17.0 and an increase in 5 BLEU points compared to the previous state-of-the-art model. Also, our proposed model was rated highly by 85 human evaluators, validating its high capability in exhibiting empathy while generating relevant and fluent responses in open-domain settings.


page 1

page 2

page 3

page 4


cTBL: Augmenting Large Language Models for Conversational Tables

An open challenge in multimodal conversational AI requires augmenting la...

Affective Neural Response Generation

Existing neural conversational models process natural language primarily...

KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning

Pre-trained language models have led to substantial gains over a broad r...

Conversational AI Chatbot Based on Encoder-Decoder Architectures with Attention Mechanism

Conversational AI Chatbot development using Artificial Intelligence and M...

Steering Output Style and Topic in Neural Response Generation

We propose simple and flexible training and decoding methods for influen...

Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models

Automated evaluation of open domain natural language generation (NLG) mo...

Inseq: An Interpretability Toolkit for Sequence Generation Models

Past work in natural language processing interpretability focused mainly...