Facial Expression Recognition and Image Description Generation in Vietnamese

by   Khang Nhut Lam, et al.

This paper discusses a facial expression recognition model and a description generation model to build descriptive sentences for images and facial expressions of people in images. Our study shows that YOLOv5 achieves better results than a traditional CNN for all emotions on the KDEF dataset. In particular, the accuracies of the CNN and YOLOv5 models for emotion recognition are 0.853 and 0.938, respectively. A model for generating descriptions for images based on a merged architecture is proposed using VGG16 with the descriptions encoded over an LSTM model. YOLOv5 is also used to recognize dominant colors of objects in the images and correct the color words in the descriptions generated if it is necessary. If the description contains words referring to a person, we recognize the emotion of the person in the image. Finally, we combine the results of all models to create sentences that describe the visual content and the human emotions in the images. Experimental results on the Flickr8k dataset in Vietnamese achieve BLEU-1, BLEU-2, BLEU-3, BLEU-4 scores of 0.628; 0.425; 0.280; and 0.174, respectively.


page 1

page 2

page 3

page 4


Facial Emotions Recognition using Convolutional Neural Net

Human beings displays their emotions using facial expressions. For human...

Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examples

NLP tasks are typically defined extensionally through datasets containin...

A Recursive Framework for Expression Recognition: From Web Images to Deep Models to Game Dataset

In this paper, we propose a recursive framework to recognize facial expr...

Performance Analysis and Evaluation of Cloud Vision Emotion APIs

Facial expression is a way of communication that can be used to interact...

Liminal Scape, an interactive visual installation with expressive AI

Liminal Scape is a visual art installation with an expressive AI sys- t...

Interpretable Deep Neural Networks for Facial Expression and Dimensional Emotion Recognition in-the-wild

In this project, we created a database with two types of annotations use...

Personality Detection of Applicants And Employees Using K-mode Algorithm And Ocean Model

The combination of conduct, emotion, motivation, and thinking is referre...

Please sign up or login with your details

Forgot password? Click here to reset