SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network

10/24/2021
by   Jingyuan Yang, et al.
Xidian University
0

Visual Emotion Analysis (VEA) aims at finding out how people feel emotionally towards different visual stimuli, which has attracted great attention recently with the prevalence of sharing images on social networks. Since human emotion involves a highly complex and abstract cognitive process, it is difficult to infer visual emotions directly from holistic or regional features in affective images. It has been demonstrated in psychology that visual emotions are evoked by the interactions between objects as well as the interactions between objects and scenes within an image. Inspired by this, we propose a novel Scene-Object interreLated Visual Emotion Reasoning network (SOLVER) to predict emotions from images. To mine the emotional relationships between distinct objects, we first build up an Emotion Graph based on semantic concepts and visual features. Then, we conduct reasoning on the Emotion Graph using Graph Convolutional Network (GCN), yielding emotion-enhanced object features. We also design a Scene-Object Fusion Module to integrate scenes and objects, which exploits scene features to guide the fusion process of object features with the proposed scene-based attention mechanism. Extensive experiments and comparisons are conducted on eight public visual emotion datasets, and the results demonstrate that the proposed SOLVER consistently outperforms the state-of-the-art methods by a large margin. Ablation studies verify the effectiveness of our method and visualizations prove its interpretability, which also bring new insight to explore the mysteries in VEA. Notably, we further discuss SOLVER on three other potential datasets with extended experiments, where we validate the robustness of our method and notice some limitations of it.

READ FULL TEXT VIEW PDF

page 1

page 2

page 4

page 9

page 11

page 13

page 14

page 16

09/04/2021

Stimuli-Aware Visual Emotion Analysis

Visual emotion analysis (VEA) has attracted great attention recently, du...
06/23/2021

A Circular-Structured Representation for Visual Emotion Distribution Learning

Visual Emotion Analysis (VEA) has attracted increasing attention recentl...
07/25/2022

Seeking Subjectivity in Visual Emotion Distribution Learning

Visual Emotion Analysis (VEA), which aims to predict people's emotions t...
05/22/2017

Building Emotional Machines: Recognizing Image Emotions through Deep Neural Networks

An image is a very effective tool for conveying emotions. Many researche...
04/25/2022

Multimodal Dual Emotion with Fusion of Visual Sentiment for Rumor Detection

In recent years, rumors have had a devastating impact on society, making...
09/11/2019

PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression

Existing methods on visual emotion analysis mainly focus on coarse-grain...
01/19/2021

ArtEmis: Affective Language for Visual Art

We present a novel large-scale dataset and accompanying machine learning...

I Introduction

Human beings are born with one of the greatest power — emotion [12]

, which is invisible yet indispensable in our daily lives. With the prevalence of social networks, images have become a major medium for people to express emotions and to understand others as well. Nowadays, computer vision algorithms mostly teach networks how to “see” like a human while scarcely tell them how to “feel” like a human, which has been considered as a crucial step towards the understanding of human cognition 

[45]. As shown in Fig. 1, when encountering an image, not only do we see objects and scenes, but we are also evoked by a certain emotion behind them involuntarily. Therefore, aiming at finding out how people feel emotionally towards different visual stimuli, Visual Emotion Analysis (VEA) has become an important research topic with increasing attention recently. Progress in VEA may benefit other related tasks (e.g., image aesthetic assessment [34, 72]

, stylized image captioning 

[7, 15], and social relation inference [76]), and will have a great impact on many applications, including decision making [54], smart advertising [41], opinion mining [37], and mental disease treatment [56].

Fig. 1: Four affective images with different emotions from FI dataset. These images deliver not only objects and scenes, but also, more significantly for us humans, emotions that are evoked involuntarily.
Fig. 2: Examples from FI dataset. We believe that visual emotions are evoked by the interactions between objects (a) as well as the interactions between objects and scenes (b) within an image. In (a), red flowers interact with different objects to convey distinct emotions. In (b), under the same scene of lawn, diverse scene-object interactions evoke different emotions.

Researchers have been engaged in VEA for more than two decades [31]

, during which methods have varied from the early traditional ones to the recent deep learning ones. In the early years, researchers designed hand-crafted features (

e.g., color, texture, shape, composition, balance, and emphasis) based on art and psychological theories [39, 81, 50, 85, 3], attempting to find out the potential impact factors on human visual emotions. However, it was hard to cover all the important factors by implementing manually designed features, which led to sub-optimal results. With the prevalence of deep learning networks, more and more researchers in VEA employed Convolutional Neural Networks (CNNs) in an end-to-end manner to predict emotions [5, 46, 65, 63, 64, 66, 69, 43]. Earlier attempts directly implemented a general CNN to extract holistic features from affective images [5, 69], which neglected the fact that visual emotions can also be evoked by local regions. Recently, in order to trace emotions more concretely, researchers in VEA adopted detection methods and attention mechanisms to focus on local regions [43, 65, 63]. Most of the aforementioned methods mapped holistic or regional features to emotion labels directly. However, since human emotions involves a highly complex and abstract cognitive process, we argue that a direct mapping may underestimate the wide affective gap [65] between low-level pixels and high-level emotions.

In addition to computer vision, researchers in other fields also devoted themselves to exploring the mysteries of visual emotions, including psychology [52], neuroscience [30] and sociology [19]. It has been demonstrated in psychology that scenes and objects can be regarded as emotional stimuli in affective images [4]. In [11], psychologist Frijda suggested that emotional state is not about a specific object, but the perception of multiple emotionally meaningful objects. Neuroscientist Moshe suggested that visual objects occur in rich surroundings are often embedded with other related objects, which serves as a key cognitive process in the human brain [2].

Based on the above studies, we believe that visual emotions are evoked by the interactions between objects as well as the interactions between objects and scenes within an image. To be specific, we argue that rather than an isolated object, multiple objects in an image interact with each other and jointly contribute to the final emotion. Besides, as scenes affect the emotional tone of an image, we further take scenes into account by mining the scene-object interactions, assuming that it is scenes that guide objects to evoke distinct emotions. As shown in Fig. 2(a), there are red roses in both left images and red tulips in the ones on the right, but we have different emotions towards them. Take red roses as an example, when red roses appear with a bride and a groom, it can be inferred as a wedding ceremony, which brings people a positive emotion, i.e., contentment. Oppositely, when red roses and a deadee comes together, people may feel sad with a negative emotion. Therefore, it is not the red roses alone that evoke a specific emotion, but the interactions between red roses and other objects jointly determine the final result. Fig. 2(b) shows that under the same scene of lawn, different objects may evoke different emotions as well. In particular, running kids on the lawn may bring us excitement while a sly crocodile lying on the lawn makes us fear. It is obvious that different interactions between objects and scenes may evoke distinct emotions, from which we attach importance to both objects and scenes when analyzing visual emotions.

Motivated by the above facts, we propose a novel Scene-Object interreLated Visual Emotion Reasoning network (SOLVER), aiming at predicting visual emotions from the interactions between objects and objects as well as objects and scenes. In order to mine the emotional relationships between different objects within an image, we construct an Emotion Graph and conduct reasoning on it. To be specific, by transforming and filtering detected object features, we construct the Emotion Graph with objects as nodes and emotional relationships as edges. Subsequently, we adopt Graph Convolutional Network (GCN) to perform reasoning on the Emotion Graph, which correlates objects with their emotional relationships and eventually yields emotion-enhanced object features. Furthermore, we propose a Scene-Object Fusion Module to interrelate scenes and objects with each other. Specifically, a novel scene-based attention mechanism is designed by exploiting scene features as guidance in object fusion process, which not only fuses multiple object features into a single one, but also serves as a pivot to interact between objects and scenes.

Our contributions can be summarized as follows:

  • We propose a novel framework, namely SOLVER, to predict visual emotions from the interactions between objects and objects as well as objects and scenes within an image, which outperforms the state-of-the-art methods on eight public visual emotion datasets. To the best of our knowledge, it is the first work that reasons between objects and scenes to infer emotions.

  • We construct an Emotion Graph to depict emotional relationships between different objects, and subsequently conduct GCN reasoning on it, aiming to interrelate objects with their emotional relationships.

  • We propose a Scene-Object Fusion Module by exploiting scene features to guide object fusion process with a designed scene-based attention mechanism, which serves as a pivot to mine the emotional relationships between objects and scenes.

The rest of the paper is organized as follows. Section II overviews the existing methods on visual emotion analysis, object detection, and graph reasoning. In Section III, we introduce the proposed SOLVER by constructing an Emotion Graph and a Scene-based Fusion Module. Extensive experiments, including comparisons, ablation studies, visualizations, and further discussions are conducted on visual emotion datasets and other potential datasets given in Section IV. Finally, we conclude our work in Section V.

Ii Related work

This work addresses the problem of visual emotion analysis, which is also closely related to object detection and graph reasoning. In this section, we review the existing methods from the above three aspects.

Ii-a Visual Emotion Analysis

In Visual Emotion Analysis (VEA), related work can be divided into several aspects according to different criteria. Based on different psychological models, i.e., Categorical Emotion States (CES) and Dimensional Emotion Space (DES), VEA can be grouped into classification task [5, 46, 65, 63, 64, 66, 69, 43, 62] and distribution learning task [67, 78, 87, 84, 77, 61]. Oriented to distinct objects, VEA can be divided into personalized VEA [83, 79] and dominant VEA [5, 46, 65, 63, 64, 66, 69, 43], which concentrate on one individual’s emotion or the averaged one of the public. Besides, there are also some surveys concerning this topic [23, 80, 82, 86]. Our work focuses on dominant emotion classification problem.

Ii-A1 Traditional Methods

Earlier works on VEA mainly focused on designing hand-crafted features to mine emotions from affective images. Inspired by psychology and art theory, Machajdik et al[39] extracted specific image features and combined them to predict emotions, which consisted of color, texture, composition and content. Borth et al[3] introduced Adjective Noun Pairs (ANPs) and proposed a visual concept detector, i.e., Sentibank, to filter out visual concepts strongly related to emotions from a semantic level. Considering both bag-of-visual word representations and color distributions, Siersdorfer et al[50] proposed an emotion analysis method based on information theory. By leveraging object detection and concept modeling methods, Chen et al[6] first recognized the top six frequent objects, i.e., car, dog, dress, face, flower, food, and then modeled the concept similarity between those ANPs. In order to understand the relationship between artistic principles and emotions, Zhao et al[81] proposed a method for both classification and regression tasks in VEA by extracting principle-of-art-based emotional features, including balance, emphasis, harmony, variety, gradation, and movement. Besides, Zhao et al[85] extracted low-level generic and elements-of-art features, mid-level attributes and principles-of-art features, high-level semantic and facial features, followed by a multi-graph learning framework. While these methods have been proven to be effective on several small-scale datasets, the hand-crafted features are still limited in covering all important factors in visual emotions.

Fig. 3: Framework of the proposed Scene-Object interreLated Visual Emotion Reasoning network (SOLVER). An object detector is first employed to extract semantic concepts and visual features of distinct objects (Sec. III-A). We then construct an Emotion Graph with objects as nodes and emotional relationships as edges (Sec. III-B1), and conduct GCN reasoning on it to yield emotion-enhanced object features (Sec. III-B2). Scene-Object Fusion Module is further designed by exploiting scene features to guide object fusion process with scene-based attention mechanism (Sec. III-C), where denotes concatenation.

Ii-A2 Deep Learning Methods

Recently, with the great success of deep learning networks, researchers in VEA have adopted Convolutional Neural Network (CNN) to predict emotions and have achieved significant progress. Based on their previous SentiBank 

[3], Chen et al[5] implemented deep networks to construct a visual sentiment concept classification method named DeepSentiBank. Leveraging half a million images labeled with website meta data, You et al[69] proposed a novel progressive CNN architecture (PCNN) to predict emotions. Rao et al[46] constructed a multi-level deep representation network (MldrNet), which extracted emotional features from image semantics, aesthetics and low-level visual features simultaneously through multiple instance learning framework. Earlier attempts simply implemented a general CNN to extract holistic features from affective images, neglecting the fact that visual emotions can also be evoked by local regions. Different from previous methods, You et al[68] utilized attention mechanism to discover emotion-relevant regions, which serves as a prior attempt to focus on local regions in VEA. Similarly, Yang et al[65] constructed a local branch to discover affective regions by implementing the off-the-shelf detection tools. A weakly supervised coupled network (WSCNet) [63] was further proposed by Yang et al., which discovers emotion regions through attention mechanism and leverages both holistic and regional features to predict emotions in an end-to-end manner. Besides, by employing deep metric learning, Yang et al[64] proposed a multi-task deep framework for tackling both retrieval and classification tasks in VEA. Zhang et al[74] proposed a novel CNN model to extract and integrate content information as well as style information to infer visual emotions. Considering different kinds of emotional stimuli, Yang et al[62] proposed a stimuli-aware VEA network together with a hierarchical cross-entropy loss. Most of the existing deep learning methods directly used holistic feature or regional features to predict visual emotions. However, considering the complexity and abstractness involved in the cognitive process of human emotions, we argue that a direct mapping may underestimate the wide affective gap between low-level pixels and high-level emotions.

Psychological studies have demonstrated that visual emotions are evoked by the interactions between objects and objects as well as objects and scenes [4, 11, 2]. Based on these observations, we propose a novel Scene-Object interreLated Visual Emotion Reasoning network (SOLVER) to predict visual emotions. To mine the emotional relationships between different objects, we first construct an Emotion Graph and then conduct GCN reasoning on it to yield emotion-enhanced object features. Besides, a Scene-Object Fusion Module is further proposed to effectively interrelate objects and scenes with a novel scene-based attention mechanism.

Ii-B Object Detection

Object detection serves as a core problem in computer vision, which has shown dramatic progress recently. Further, object detection methods have been implemented into various related tasks as a pre-processing step, including image captioning [71, 88], scene graph [57, 60], visual reasoning [59, 73], person re-identification [75, 16]etc. Object detection methods can be roughly divided into one-stage methods (e.g., YOLO [47], SSD [38]) and two-stage methods (e.g., R-CNN [13], Fast R-CNN [14], Faster R-CNN [48]

) according to whether region proposals are generated. R-CNN was a pioneer work in two-stage detection methods, which boosted detection accuracy exceedingly by implementing deep networks for the first time. Based on their previous work, Girshick proposed Fast R-CNN to reduce the redundant computation of feature extraction in R-CNN and further improve its accuracy and speed. Region Proposal Network (RPN) was introduced by Faster R-CNN to replace the original time-consuming Selective Search, which not only unified object detection in an end-to-end network, but also improved its efficiency simultaneously. Since Faster R-CNN is widely used for its great performance on both accuracy and speed 

[35, 20, 33, 21, 22], we adopt Faster R-CNN to extract object features and then construct an Emotion Graph based on them.

Ii-C Graph-based Reasoning

Recently, Graph Neural Networks (GNNs) have been proved to be an effective framework to exchange and propagate information through structured graphs, which have been widely used in various computer vision tasks [18, 9, 8]. In Gated Graph Neural Network (GGNN) [36], Li et al

. adopted Gate Recurrent Units (GRUs) to update hidden states of graph nodes and implemented modern optimization techniques to yield output sequences. Aiming to learn the graph-structured data via convolutional operations, Graph Convolutional Network (GCN) 

[26] was proposed as a scalable approach for semi-supervised classification. In Graph Attention Networks (GAT) [55], self-attention mechanism was introduced to update node features by attending over its neighbors, which was often used in bidirectional graph. In this paper, we apply GCN to conduct reasoning on the Emotion Graph, which correlates objects with their emotional relationships and yields emotion-enhanced object features.

Iii Methodology

In this section, we propose a novel Scene-Object interreLated Visual Emotion Reasoning network (SOLVER) to predict visual emotions through mining the interrelationships between objects and objects as well as objects and scenes. Rather than adopting scene and object features directly, we develop a reasoning mechanism to infer deep emotional relationships between objects and scenes, which is helpful to bridge the existing affective gap. Fig. 3 shows the architecture of the proposed network. We first employ an object detector, i.e., Faster R-CNN, to extract semantic concepts and visual features of distinct objects (Sec. III-A). After transforming and filtering those object features, we construct an Emotion Graph to depict emotional relationships between different objects (Sec. III-B1) and subsequently conduct GCN reasoning on it to correlate different objects with their pairwise relationships, yielding emotion-enhanced object features (Sec. III-B2). Finally, we design a Scene-Object Fusion Module by exploiting scene features as guidance to fuse object features with scene-based attention mechanism, from which scene-object interrelationships are built simultaneously (Sec. III-C).

Iii-a Object Detector

We adopt Faster R-CNN [48] as the object detector to select a set of candidate regions using ResNet-101 [17] as backbone. Our Faster R-CNN model is pre-trained on Visual Genome dataset [28], which outputs attribute classes in addition to general object classes, providing more detailed information. For example, object classes include: dog, building, mountains, womanetc., while attribute classes include: white, large, rocky, beautifuletc. In the following process, we employ object classes alone, aiming to build up connections between emotions and objects. For simplicity, we only use the top-10 RoIs and further prove its effectiveness in Sec. IV-D2. After applying Faster R-CNN, each image can be represented as a set of object semantic concepts with corresponding confidence scores , and a set of object visual features , in which , , and . Object semantic concepts are prediction results of the top-10 RoIs, which can be regarded as pseudo object-level labels since the involved visual emotion datasets only contain image-level labels. Each confidence score indicates the degree of confidence towards its corresponding prediction result . For each selected region , we extract the feature after the average pooling layer to serve as object visual feature .

Iii-B Emotion Graph

In this section, we build up an Emotion Graph and perform reasoning on it to mine the emotional relationships between different objects. Based on the detection results of Faster R-CNN, we construct the Emotion Graph with objects as nodes and emotional relationships as edges, by transforming and filtering those detected object features in Sec. III-A. Subsequently, we conduct GCN reasoning on the Emotion Graph, which propagates object information under different emotional relationships and eventually yields emotion-enhanced object features.

Iii-B1 Emotion Graph Construction

Fig. 4: Construction of the Emotion Graph. Based on the detection results of Faster R-CNN, our Emotion Graph is constructed with objects as nodes and emotional relationships as edges.

We construct our Emotion Graph by setting objects as nodes and building up emotional relationships between them based on previous detection results as shown in Fig. 4. For word embedding, following the recent work [9, 21]

, we employ Global Vectors for Word Representation (GloVe) 

[44]

, an unsupervised learning algorithm to obtain vector representations for words. Thus, semantic concepts

are embedded to semantic features , where and . Notably, towards a specific object (e.g., cat, tree, balloon), visual features may vary from instance to instance while semantic features always remain unchanged. Thus, instead of visual features , we adopt semantic features as nodes to construct the Emotion Graph, aiming to map objects to emotions with one-to-one relationships. Oppositely, we build up edges based on visual features , so as to depict diverse emotional relationships between distinct objects. Considering the existing gap between semantics and emotions, we first embed visual features from semantic space to emotional space using a linear function followed by a non-linear -norm function:

(1)

which may bring a richer description towards emotional space. In Eq. (1), , is a learnable embedding matrix, and is a learnable embedding bias.

Therefore, the emotional visual features are denoted as , in which . Following [33, 58]

, we build up adjacency matrix by calculating affinity matrix to construct pairwise emotional relationships between objects for the Emotion Graph:

(2)

where , , represents two emotional visual features, and denotes the emotional relationships between them. Notably, and are two embedding functions with different parameters. Since visual features may vary greatly under the same emotion, we further embed them with different parameters to eliminate the bias of distinct features, hoping that they would be more comparable in the emotional space. The settings in Eq. (2), i.e., two functions with different parameters, is further ablated in Section IV-D1.

Our Emotion Graph is eventually constructed as where nodes denote the objects with their semantic features:

(3)

and edges represent the emotional relationships between different objects, which are described by affinity matrix:

(4)

which means there will be an edge with high affinity score connecting two objects if they have strong emotional relationships and are thus highly correlated. In order to remove redundancy of object nodes, we set a confidence threshold of 0.3 for to filter out less confident ones from the nodes, resulting in filtered nodes:

(5)
(6)

where and . Notably, not only do we remove redundant nodes, but we also delete adjacent edges of these nodes through masking operations as shown in Eq. (8), where denotes sign function and denotes absolute value function. After applying the above two functions, outputs either 1 or 0, suggesting whether or not edge still exists after the masking operation:

(7)
(8)

where and takes the maximum element of and . We then apply mask matrix to affinity matrix and obtain the masked affinity matrix as

(9)

where denotes the element-wise multiplication. The validity of mask operation is further proved in Sec. IV-D1. In order to depict emotional relationships between objects, the Emotion Graph is eventually built up with as nodes and as edges and further propagates information to yield emotion-enhanced features in Sec. III-B2.

Iii-B2 Emotion Graph Reasoning

In order to mine the emotional relationships between different objects, we apply Graph Convolutional Network (GCN) [26] to perform reasoning on the Emotion Graph, which is capable of exchanging and propagating information through structured graph. In traditional GCN, as shown in Eq. (10), adjacency matrix is denoted as while degree matrix is denoted as , where symmetric normalized Laplacian matrix depicts the relationships between different nodes:

(10)

where denotes the node features of the l-th layer and

denotes a nonlinear activation function. Different from the supervised graph learning tasks, we need to establish a set of rules to calculate edges of the Emotion Graph, which is described in Eqs. (

1) (2) (9). Following similar update mechanism with Eq. (10), our GCN layer is defined as

(11)

where denotes the input node features of the l-th layer and denotes the input edge features. Notably, we add a residual block in our GCN following [33] to better maintain the original node features, drawing lessons from residual block in ResNet. For the l-th layer, is the weight matrix of GCN and is the weight matrix of the residual block. It can be inferred from Eq. (11) that each node is updated based on both its neighbors and itself, from which object features propagate throughout the whole Emotion Graph under their emotional relationships. We conduct reasoning on our Emotion Graph by applying several GCN layers to update object features iteratively:

(12)

where . The output of the final GCN layer is regarded as emotion-enhanced object features, as object features are iteratively updated under their emotional relationships. In our experiment, L is set to 4, which is further ablated in Sec. IV-D2. Therefore, we successfully construct the Emotion Graph and conduct reasoning on it to correlate objects with their emotional relationships.

Iii-C Scene-Object Fusion Module

Fig. 5: Scene-based attention mechanism. By exploiting scene features to guide object fusion process, scene-based attention mechanism interrelates scenes and objects with each other, where represents matrix product.

In previous sections, we model the emotional relationships between distinct objects and yield emotion-enhanced object features. Besides salient objects, scenes are regarded as another major stimulus in emotion evocation process, which largely affect the emotional tone of an image and thus cannot be ignored in VEA. In this section, assuming that it is scenes that guide objects to evoke distinct emotions, we propose a novel scene-based attention mechanism to mine the scene-object interrelationships. By implementing ResNet-50 [17] as backbone, scene feature extractor takes an affective image as input and outputs with its corresponding scene feature, which is denoted as with . In order to mine the deep interrelationships between scenes and objects, we propose a scene-based attention mechanism by exploiting scene features as guidance to fuse object features under different scene-object emotional relationships. To be specific, we first project the scene feature and emotion-enhanced object features into an embedding space to narrow the gap between scenes and objects, and calculate object attention weights by mining the emotional relationships between them:

(13)
(14)

where embedding functions are constructed by two learnable matrices , followed by an -norm function. Besides, so as to normalize each attention weight to

, we apply Sigmoid function as the nonlinear activation function

. The closer relationship a specific object is to a scene , the greater attention weight it will gain. Considering that different interactions between objects and scenes may evoke different emotions, we employ their interrelationships, i.e., the attention weights , as guidance to fuse object features:

(15)

After applying attention weights to all object features, we obtain the attended object feature with . The overall process of the scene-based attention mechanism is shown in Fig. 5. Since both scenes and objects are indispensable in emotion evocation process, we further concatenate the scene feature with the object feature as

(16)

The concatenated emotion feature

is then fed into the emotion classifier and a Softmax function successively:

(17)
(18)

where denotes the number of emotion categories and denotes the number of affective images in a specific visual emotion dataset. Moreover, represents emotion labels in the dataset while represents emotion prediction results of the proposed SOLVER. is a learnable weight matrix in the emotion classifier, which is further optimized by the cross-entropy (CE) loss together with the whole network in an end-to-end manner.

Iv Experimental results

Iv-a Datasets

Dataset # Images # Classes Type
FI [70] 23,164 8 Social
Flickr [24] 60,730 2 Social
Instagram [24] 42,848 2 Social
EmotionROI [43] 1,980 6 Social
Twitter I [69] 1,269 2 Social
Twitter II [3] 603 2 Social
ArtPhoto [39] 806 8 Artistic
IAPSa [32, 40] 395 8 Natural
TABLE I: Statistics of the involved visual emotion datasets.
Method FI Flickr Instagram EmotionROI Twitter I Twitter II
Sentibank [3] 49.23 69.26 66.53 35.24 66.63 65.93
Zhao et al[81] 46.13 66.61 64.17 34.84 67.92 67.51
DeepSentibank [5] 51.54 70.16 67.13 42.53 71.25 70.23
Fine-tuned AlexNet [29] 59.85 79.73 77.29 44.19 75.20 75.63
Fine-tuned VGG-16 [51] 65.52 80.75 78.72 49.75 78.35 77.31
Fine-tuned ResNet-50 [17] 67.53 82.73 81.45 52.27 79.53 78.15
MldrNet [46] 65.23
Sun et al[53] 79.85 78.67 80.33 78.97
Yang et al[66] 67.48 52.40
Yang et al[64] 67.64
WSCNet [63] 70.07 81.36 81.81 58.25 84.25 81.35
Zhang et al[74] 71.77
SOLVER (Ours) 72.33 86.20 85.60 62.12 85.43 83.19
TABLE II: Comparison with the state-of-the-art methods.
Results are reported in classification accuracy (%) on six visual emotion datasets.

We evaluate the proposed SOLVER on eight public visual emotion datasets, including the Flickr and Instagram (FI) [70], Flickr, Instagram [24], EmotionROI [43], Twitter I [69], Twitter II [3], ArtPhoto [39] and IAPSa [32, 40]. The involved datasets can be roughly divided into large-scale datasets (i.e., FI, Flickr, Instagram) and small-scale datasets (i.e., EmotionROI, Twitter I, Twitter II, Artphoto, IAPSa), for which more details are shown in TABLE I.

FI. The FI dataset, with 23,164 images, is one of the largest well-labeled datasets, which is collected from the Flickr and Instagram by searching eight emotion categories as keywords, i.e., Amusement, Anger, Awe, Contentment, Disgust, Excitement, Fear and Sad. The collected images are then well-labeled by 225 Amazon Mechanical Turk (AMT) workers through keeping the weakly labels and their corresponding images with at least three of the five are agreed.

Flickr and Instagram. Using image ID or emotional words as query keywords, the Flickr and Instagram datasets are crawled from the internet, containing 60,730 and 42,848 affective images respectively. Labeled by crowed-sourcing human annotation, these datasets provide sentiment labels with two emotion categories, i.e., positive, negative. With approximately 100 thousand images in total, the Flickr and Instagram datasets provide us with a large dataset to train with deep learning methods.

EmotionROI. The EmotionROI dataset contains 1,980 images with six emotion categories (i.e., anger, disgust, fear, joy, sad, surprise), which is a widely-used emotion prediction benchmark collected from Flickr. Besides, each image is also annotated with 15 bounding boxes as emotional regions, which act as pixel-level supervisions besides image-level labels.

Twitter I. Collected from social websites, Twitter I contains 1,269 affective images in total. Five AMT workers are recruited to generate sentiment labels for each candidate image in Twitter I.

Twitter II. Twitter II consists of 603 images downloaded from Twitter website, which is also labeled by AMT participants. With sentiment labels, there are 470 positive images and 133 negative images in total.

ArtPhoto. Rather than collecting images from social networks, ArtPhoto dataset are taken by professional artists, aiming to deliberately evoke certain emotions in their photos. Labeled with eight emotion categories, Artphoto dataset contains 806 images in total.

IAPSa. The International Affective Picture System (IAPS) dataset [32] is widely used in visual emotion analysis research. As a subset of the IAPS dataset, The IAPSa dataset [40] contains 395 images and is labeled with eight emotion categories.

Iv-B Implementation Details

Based on ResNet-50, our scene feature extractor is pre-trained on a large-scale visual recognition dataset, ImageNet 

[10]. We adopt Faster R-CNN as our object detector based on ResNet-101, which is pre-trained on Visual Genome dataset [28]. The whole network, including the scene branch and the object branch, is then jointly trained in an end-to-end manner with affective datasets. Following the same setting in [70], FI dataset is randomly split into training set (80%), validation set (5%) and testing set (15%). Flickr and Instagram datasets are randomly split into training set (90%) and testing set (10%), which follows the same configuration in [24]. Most of the small-scale datasets are split into training set (80%) and testing set (20%) randomly, except for those with specified training/testing separations [3][43]. For training/validation/testing sets, we first resize each image to 480 on its shorter side and then crop it to 448448 randomly followed by a horizontal flip [17]. Our SOLVER is trained by the adaptive optimizer Adam [25]

. With a weight decay of 5e-5, the learning rate starts from 5e-5 and is decayed by 0.1 every 5 epochs, and the total epoch number is set to 50. Our framework is implemented using PyTorch 

[42] and our experiments are performed on an NVIDIA GTX 1080Ti GPU.

Iv-C Comparison with the State-of-the-art Methods

To evaluate the effectiveness of the proposed SOLVER, we conduct experiments compared with the state-of-the-art methods on eight visual emotion datasets, which are shown in TABLE II and Fig. 6.

In TABLE II, we first compare our SOLVER with the state-of-the-art methods in classification accuracy on six visual emotion datasets, including FI, Flickr, Instagram, EmotionROI, Twitter I and Twitter II. which can be divided into traditional methods and deep learning ones. For traditional methods, Sentibank [3] and Zhao et al[81] adopted a set of emotion-related hand-crafted features, which were early attempts to explore the mysteries in VEA. We also conduct experiments on several typical CNN backbones by fine-tuning the network parameters with visual emotion datasets, including AlexNet [29], VGG-16 [51] and ResNet-50 [17]. Benefiting from its powerful representation ability, deep learning methods gained significant performance boosts compared with those traditional ones. With coupled global-local branches, Yang et al[63] proposed WSCNet and greatly improved the classification performance in VEA. Zhang et al[74] further boosted the classification performance by exploring content as well as style information to predict visual emotions. Notably, the missing data in TABLE II

is due to the lack of both classification results and open source codes. We can infer from TABLE 

II that our SOLVER achieves greater performance boosts on large-scale datasets compared with those small-scale ones, as deep scene-object interrelationships can be better mined with larger datasets. Overall, the proposed SOLVER outperforms the state-of-the-art methods by a large margin on six visual emotion datasets.

Fig. 6: Classification results on Artphoto and IAPSa datasets, compared with the state-of-the-art methods.

We further conduct detailed comparisons with the-state-of-the-art methods on Artphoto [39] and IAPSa [32, 40], as shown in Fig. 6. Considering the limited and imbalanced data in the above datasets, we employ the “one against all” strategy to train our network following the previous method [39] for fair comparison. Moreover, images in each category are randomly split into five batches and a 5-fold cross-validation is further implemented for emotion classification. We further remove the category of anger from IAPSa dataset following [39, 81, 46], as there are only eight samples in this category, where our SOLVER outperforms the state-of-the-art methods, including Machajdik[39], Zhao et al.[81] and MldrNet [46]. In addition to the optimal overall accuracy, our method is generally robust and applicable to each emotion with smaller accuracy deviation between categories. The above analysis proves that our SOLVER is applicable on Artphoto and IAPSa datasets as well.

The proposed SOLVER consistently outperforms the state-of-the-art methods on eight visual emotion datasets, which proves the effectiveness and robustness of our method.

Iv-D Ablation Study

Our SOLVER is further ablated to verify the validity of its network structure and the involved hyper-parameters.

Module Ablated Combinations Acc (%)
Emotion Graph single object 34.98
multiple objects 47.60
multiple objects + GCN + one embedding 62.40
multiple objects + GCN + two embeddings 64.47
multiple objects + GCN + mask + one embedding 63.86
multiple objects + GCN + mask + two embeddings 66.28
Scene-Object Fusion Module scene 67.53
scene + single object 68.13
scene + multiple objects 70.20
scene + multiple objects + scene-based attention 70.66
scene + multiple objects + GCN + one embedding + scene-based attention 71.06
scene + multiple objects + GCN + two embedding + scene-based attention 71.67
scene + multiple objects + GCN + mask + one embedding + scene-based attention 71.93
scene + multiple objects + GCN + mask + two embeddings + scene-based attention 72.33
TABLE III: Ablation study of network structure on FI dataset.

Iv-D1 Network Architecture Analysis

As shown in TABLE III, we conduct ablation study on FI dataset, aiming to verify the effectiveness of each proposed module. Our SOLVER mainly consists of two modules: Emotion Graph (i.e., object-object interaction) and Scene-Object Fusion Module (i.e., scene-object interaction), as shown in the first column. In each module, there are some detailed network designs, which are depicted as ablated combinations in the second column in TABLE III. In the Emotion Graph, we first conduct ablation studies concerning a single object and multiple objects, which indicates that multiple objects bring a performance boost compared with a single one. After that, we introduce our GCN reasoning mechanism with detailed designs into comparisons, i.e., one/two embedding(s), w/wo mask, suggesting that the object-object interactions indeed improve the performance in predicting emotions. From the above experiments, it is obvious that two embedding functions with different parameters achieve better performance than one embedding function and mask operation can further bring a performance boost. In the Scene-Object Fusion Module, it is obvious that both scene branch and scene-based attention mechanism make a great contribution to emotion classification, which suggests that it is the scene that guide object fusion process. From the above ablation studies, we can conclude that each detailed design of the proposed method is complementary and indispensable, which jointly contributes to the final result.

L 1 2 3 4 5 6 7 8
Acc (%) 71.03 71.49 72.04 72.33 72.36 72.01 72.13 71.58
TABLE IV: Ablation study of GCN layers (L) on FI dataset.
Fig. 7: Hyper-parameter analysis of node number in the Emotion Graph.

Iv-D2 Hyper-Parameter Analysis

We conduct experiments to validate the choice of node number and the layer number in our Emotion Graph, as shown in Fig. 7 and TABLE IV. In general settings of Visual Genome dataset [28], top-36 RoIs are selected from the pre-trained object detector. However, we believe that such a large number of nodes can be redundant in our Emotion Graph and thus further conduct experiments to figure out the optimal , balancing both accuracy and computational cost. To be specific, we design experiments on both scene-based attention mechanism and the GAP layer, aiming to reduce the randomness in a single experiment. Setting as the step value, is varied from 0 to 20. We find that the accuracy constantly grows as varies from 0 to 10, while it slightly drops after , due to the redundancy of RoIs in object detection. It is worth mentioning that the growth of accuracy from 0 to 10 further proves that visual emotion analysis is not about a single object, but the interactions between multiple objects. From the above analysis, we choose as the node number in our Emotion Graph. For GCN layers, we ablate the number of GCN layers for better illustration. Setting as the step value, L is varied from 1 to 8. It is obvious that the accuracy consistently grows as the GCN layers increase, contributing to the excellent reasoning ability GCN owns. However, the performance meets a bottleneck when the number of GCN layers is too large, which may be caused by over fitting of too many parameters.

Iv-E Visualization

The effectiveness of the proposed SOLVER has been quantitatively evaluated by comparing to the state-of-the-art methods and performing detailed ablation studies. As we are motivated by the psychological evidences that emotions are evoked by the two categories of interactions, in this section, we try to figure out how objects and objects, objects and scenes are interrelated with each other by visualizing the intermediate process of the SOLVER. To be specific, we visualize the emotional object concepts (Sec. IV-E1) and emotional object regions (Sec. IV-E2) for each category (i.e., Amusement, Awe, Contentment, Excitement, Anger, Disgust, Fear and Sad), which further validates the interpretability of our method and explores the mysteries of visual emotions.

Iv-E1 Emotional Object Concepts

Considering that there exist relationships between objects, scenes and emotions, we visualize the top-10 emotional object concepts for each category. After implementing Faster R-CNN on FI dataset, we first calculate the frequency for each object in each emotion category , where denotes the overall object number and denotes the emotion categories in the whole dataset:

(19)

where and represents the count of object in category . Subsequently, we derive the scene-based attention coefficient from the trained model and calculate the averaged attention coefficient for each object in each emotion category :

(20)

where denotes the j-th instance of object in category . While object frequency represents the objects distribution of each emotion in the dataset, scene-based attention coefficient represents the correlations between objects and emotions learned by our SOLVER. Taking both object frequency and scene-based attention coefficient into account, we obtain the weighted frequency for each object in each emotion category :

(21)

It is obvious that not all the objects are emotional, some non-emotional objects appear in every emotion category, e.g., man, woman, people, etc. Thus, we adopt the TF-IDF [49] technique to separate the emotion-specific objects from the non-emotional ones, where the importance of an object increases as it appears in a specific emotion category and decreases inversely with its appearance in the whole dataset. As shown in Fig. 8, we list the top-10 emotional object concepts for each emotion category with their corresponding weighted frequencies. Besides, we present three typical images for each emotion to further illustrate these emotional object concepts in a concrete and vivid form. Taking awe as an example, there are mountains, castle, ocean in the first image, cliff, horizon, sunset in the second one, and ocean, wave, shore in the third one, which indicates that interactions between these emotional objects may evoke awe to a large extent. The most relevant concepts towards excitement include raft, surfboard, plane, microphone, player, etc., which implies sports events or entertainments and consequently evokes excitement.

Fig. 8: Visualization of emotional object concepts on FI. Each emotion is described with the top-10 emotional object concepts and their corresponding weighted frequencies. We further illustrate these emotional object concepts with three typical images from FI dataset.
Fig. 9: Visualization of emotional object regions on FI. Each emotion is represented with the input affective image and the output emotional object regions, together with their corresponding semantic concepts and attention weights.

Iv-E2 Emotional Object Regions

Besides emotional object concepts, we also visualize emotional object regions by presenting one image for each emotion category as a representative. As shown in Fig. 9, after implemented the proposed SOLVER, an affective image (i.e., the upper one) is turned into a set of emotional object regions (i.e., the lower one) with corresponding semantic concepts and attention weights. Notably, we first adopt object detectors to refine emotions to object level and then mine the interrelationships between objects and emotions with scene-based attention mechanism, which takes the scene feature as a guidance to fuse objects. Attention weights reflect the relevance between objects, scenes and emotions. The more an object is related to a specific emotion, the higher the attention weight will gain. For example, in amusement, tower (0.957), building (0.938) get higher attention scores while cloud (0.764) gets a lower score, as when tower and building comes together, they often indicate castle and amusement park, which surely bring people joy. When people or other animals are angry, the most striking part of their facial expression is an opened mouth. Thus, angry is represented by an furious leopard with a distinguished opened mouth, and accordingly our SOLVER focuses more on its mouth (0.851) than its head (0.684), eyes (0.654).

Iv-F Further Discussions

To validate the robustness of our method, we further extend experiments on three other potential datasets, i.e., LUCFER [1], EMOTIC [27], and Flickr CC [3], with more diverse emotional settings. Meanwhile, we also notice some limitations of our method and discuss them for future work.

Iv-F1 Potential Datasets

LUCFER [1] contains over 3.6M images with 3-dimensional labels including emotion, context, and valence, which is currently the largest emotion recognition dataset. With a sum of 18,316 images, EMOTIC [27] serves as a pioneer dataset in emotion state recognition (ESR) field, which is labeled with both discrete categories (i.e., 26 defined emotions) and continuous dimensions (i.e., valence, arousal, and dominance (VAD)). Containing 500,000 images in total, Flickr CC [3] is one of the largest visual sentiment datasets labeled with 1,553 fine-grained ANPs (Adjective Noun Pairs).

Iv-F2 Data Preprocessing

Since SOLVER is oriented to single-label emotion classification tasks, we preprocess the three potential datasets in distinct and reasonable ways to fit our task. According to the rules in LUCFER, we degenerate the 275 fine-grained emotion-context pairs into eight basic emotions (i.e., Anger, Anticipation, Disgust, Fear, Joy, Sadness, Surprise, and Trust) in Plutchik’s wheel. Besides, we group the eight emotions into two sentiments (i.e., positive, and negative) according to their polarities. Following the same setting in [1], LUCFER is split into training set (80%) and testing set (20%). Each image in EMOTIC is annotated with multiple emotion labels, which is different from our single-label classification task. Besides, it is irrational to degrade a multi-label task to a single-label one. Fortunately, valence in VAD measures the positive degree of an emotion, where corresponds to positive and negative. Thus, we degenerate valence to sentiment labels (i.e., positive, and negative) according to psychological model. EMOTIC is split into training set (70%), validation set (10%), and testing set (20%) [27]. Based on Visual Sentiment Ontology (VSO) [3], we degenerate its 1,553 ANPs into sentiment labels (i.e., positive, and negative). The Flickr CC dataset is split into training set (80%) and testing set (20%), which follows the same setting in [3].

Dataset Scene Object SOLVER
LUCFER-8 [1] 12.98 17.06 14.69
LUCFER-2 [1] 65.46 67.19 69.38
EMOTIC-2 [27] 36.11 35.99 36.13
Flickr CC-2 [3] 37.05 36.45 36.86
TABLE V: Classification accuracy (%) on three potential datasets
using models trained on FI.
Dataset Scene Object SOLVER
LUCFER-8 [1] 72.04 71.25 75.59
LUCFER-2 [1] 87.94 86.84 90.55
EMOTIC-2 [27] 63.47 62.22 65.07
Flickr CC-2 [3] 69.74 68.27 71.68
TABLE VI: Classification accuracy (%) on three potential datasets
using models trained on themselves.

Iv-F3 Classification Accuracy

We conduct experiments in classification accuracy on three potential datasets in TABLE V and TABLE VI. Specifically, results using models trained on FI dataset are reported in TABLE V, and results using models trained on different datasets themselves are shown in TABLE VI. In particular, LUCFER is conducted with two experimental settings, eight emotions (i.e., LUCFER-8) and two sentiments (i.e., LUCFER-2), for richer comparisons and analyses. In order to validate the effectiveness of our method and to explore the emotional diversity in different datasets, we ablated our experiments to three settings including Scene branch, Object branch, and SOLVER.

Fig. 10: Visualization of emotional object regions on LUCFER. Each image (i.e., the first row) is presented with two sets of emotional object regions (i.e., the second and third rows). Specifically, the second row uses model trained on FI while the model in the third row is trained on LUCFER itself.

In TABLE V

, it can be noticed that the FI trained model achieves relatively good results on LUCFER-2 (69.38%), while meets unsatisfactory results on LUCFER-8, EMOTIC-2 and Flickr CC-2 (14.69%, 36.13%, and 36.86%). Dataset shift is a common problem in machine learning. Since emotions are complex and ambiguous to be described and labeled, this gap becomes even larger. Besides, VEA aims at finding out how people feel emotionally towards different visual stimuli, while ESR focuses on recognizing peoples’ emotional states from their frames. The mismatches between these two areas may lead to a performance degradation in TABLE 

V. Compared with the other 2-sentiment tasks, LUCFER-2 achieves quite a good result, which attributes to the well-constructed dataset with more emotional stimuli and less noise. It is obvious that Object branch performs generally better than Scene branch on LUCFER. The possible reason may be that there are distinct and obvious objects conveying emotions, while scene features are not clearly distinguishable to separate different emotions apart. The three results on EMOTIC dataset are close to each other, which indicates that our SOLVER failed to find key features related to ESR. Moreover, we guess that the key features concerning ESR may be facial expressions or body language, which our method does not involve. Similarly, SOLVER performs poorly on Flickr CC, resulting from its very large data scale compared with FI. Besides, most images in Flickr CC contain a distinct object or a simple scene, which our SOLVER may fail to match with.

For fair comparisons with datasets in Sec IV-C, we also report the results using models trained on three potential datasets and tested on themselves in TABLE VI. Compared with TABLE V, it is obvious that the corresponding results in TABLE VI have significant performance improvements, which are comparable to the results in TABLE II. In TABLE VI, it is undoubted that LUCFER is a well-labeled dataset, whose 8-emotion accuracy (75.59%) is higher than others’ 2-sentiment ones (65.07%, and 71.68%). TABLE VI shows that SOLVER constantly outperforms the other ablated branches on all experimental settings, which further validates the effectiveness and the robustness of each proposed branch.

From TABLE V and TABLE VI, we can conclude that SOLVER achieves competitive results on three potential datasets. Meanwhile, we also noticed some limitations of our method. Firstly, when facing ESR tasks, SOLVER fails to leverage the facial expressions as well as body language of people within the images, resulting in degraded performance. Secondly, SOLVER is proposed to mine the emotional correlations between objects and scenes, which fails to deal with the situation of a single object or a simple scene.

Fig. 11: Failure cases on LUCFER. (a) Rather than specific scenes (i.e., stadium in the first column) or objects (i.e., tennis in the second column), emotions are largely evoked by facial expressions and body language of people within the image. (b) Different people in an image do not necessarily share one emotion. (c) Some images do not necessarily evoke strong emotions.

Iv-F4 Visualizations

In addition to classification accuracy, we also visualize emotional object regions on potential datasets from different trained models (i.e., FI and themselves). In Fig. 10, we present the visualizations of emotional object regions on LUCFER. While the first row represents the input images, the second row uses model trained on FI and the third row is trained on LUCFER itself. By analyzing the difference in attention weights, we discover some possible reasons for the performance improvements in TABLE VI compared with TABLE V. Take the first image as an example, while the second row focuses more on grass, tree, and other objects, the third row attends more weights on man, shirt, and hair. It is obvious that FI trained model concentrate more on objects unrelated to people while LUCFER trained model emphasizes more on human-centered ones, which further shows the robustness of our method. As ESR datasets are constructed by images all containing people, it is understandable that increased attention weights on people and their related objects may bring a performance boost in such datasets. As is mentioned above, there are some similarities in VEA and ESR tasks yet much more differences in details. The visualizations also prove that emotions are always evoked by people within the images in ESR datasets, especially their facial expressions and body language, which is different from our VEA task.

Iv-F5 Failure Cases

Fig. 11 shows three types of failure cases on LUCFER. In Fig. 11 (a), two images in the first column share the same scene (i.e., stadium), and two images in the second column share the same object (i.e., tennis). However, under the same scene/object, emotions are varying according to different facial expressions and body language. Take first row as an example, by observing the waving fists and the confident smile, the coach is angry with his players while the player is satisfied with herself Since SOLVER mines emotions from the interactions between objects and scenes, facial expressions and body language are not considered in our method. This failure is mainly caused by the mismatch between VEA and ESR, which will be considered in our future work. In Fig. 11 (b), we can see that people in an image do not necessarily share the same emotion. Take the second row as an example, the woman seems happy while the man is angry. While ESR concentrates on the emotions of characters within the image, our VEA focuses on the emotions of viewers outside the image. This failure also results from the mismatch between two tasks. Besides, there are also some images where emotions may be weak and obscure as in Fig. 11 (c). In other words, these images hardly arouse any of our emotions. This failure is universal in both tasks. We may find a reasonable way to separate emotional images and emotionless images in our future work.

V Conclusion

We have proposed a Scene-Object interreLated Visual Reasoning network (SOLVER) to mine emotions from the interactions between objects and objects as well as objects and scenes. We first constructed an Emotion Graph based on detected features and conducted GCN reasoning on it, aiming to extract the emotional relationships between different objects. Besides, we proposed a Scene-Object Fusion Module to fuse objects with the guidance of scene-based attention mechanism. Extensive experiments and comparisons have shown that the proposed SOLVER consistently outperforms the state-of-the-art methods on eight public visual emotion datasets. Notably, visualization results on emotional object concepts and regions not only proved the interpretability of our network, but also offered new insight to explore the mysteries in visual emotion analysis. We further extended our experiments on three other potential datasets, where we validated the effectiveness and robustness of our method with more diverse emotional settings. Additionally, we also noticed some limitations of our method. Since SOLVER aims at mining emotions from the interactions between objects and scenes, it may fail to deal with situations including images with a single object/scene and drawings without objects/scenes. In ESR datasets, though scenes and objects are still important in predicting emotions, human-centered attributes, i.e., facial expressions and body language, play a leading role that cannot be ignored. These limitations will be considered in a unified network for a better performance in our future work.

References

  • [1] P. Balouchian, M. Safaei, and H. Foroosh (2019) LUCFER: a large-scale context-sensitive image dataset for deep learning of visual emotions. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1645–1654. Cited by: §IV-F1, §IV-F2, §IV-F, TABLE V, TABLE VI.
  • [2] M. Bar (2004) Visual objects in context. Nature Reviews Neuroscience 5 (8), pp. 617–629. Cited by: §I, §II-A2.
  • [3] D. Borth, R. Ji, T. Chen, T. Breuel, and S. Chang (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the 21st ACM international conference on Multimedia, pp. 223–232. Cited by: §I, §II-A1, §II-A2, §IV-A, §IV-B, §IV-C, §IV-F1, §IV-F2, §IV-F, TABLE I, TABLE II, TABLE V, TABLE VI.
  • [4] T. Brosch, G. Pourtois, and D. Sander (2010) The perception and categorisation of emotional stimuli: a review. Cognition and Emotion 24 (3), pp. 377–400. Cited by: §I, §II-A2.
  • [5] T. Chen, D. Borth, T. Darrell, and S. Chang (2014) Deepsentibank: visual sentiment concept classification with deep convolutional neural networks. arXiv preprint arXiv:1410.8586. Cited by: §I, §II-A2, §II-A, TABLE II.
  • [6] T. Chen, F. X. Yu, J. Chen, Y. Cui, Y. Chen, and S. Chang (2014) Object-based visual sentiment concept analysis and application. In ACMMM, pp. 367–376. Cited by: §II-A1.
  • [7] T. Chen, Z. Zhang, Q. You, C. Fang, Z. Wang, H. Jin, and J. Luo (2018) “Factual”or“emotional”: stylized image captioning with adaptive learning and attention. In ECCV, pp. 519–535. Cited by: §I.
  • [8] Y. Chen, M. Rohrbach, Z. Yan, Y. Shuicheng, J. Feng, and Y. Kalantidis (2019) Graph-based global reasoning networks. In CVPR, pp. 433–442. Cited by: §II-C.
  • [9] Z. Chen, X. Wei, P. Wang, and Y. Guo (2019) Multi-label image recognition with graph convolutional networks. In

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    ,
    pp. 5177–5186. Cited by: §II-C, §III-B1.
  • [10] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In CVPR, pp. 248–255. Cited by: §IV-B.
  • [11] N. H. Frijda (2009) Emotion experience and its varieties. Emotion Review 1 (3), pp. 264–271. Cited by: §I, §II-A2.
  • [12] E. R. Gerber (1985) Rage and obligation: samoan emotion in conflict. Person, self, and experience: Exploring Pacific ethnopsychologies 121, pp. 167. Cited by: §I.
  • [13] R. Girshick, J. Donahue, T. Darrell, and J. Malik (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pp. 580–587. Cited by: §II-B.
  • [14] R. Girshick (2015) Fast r-cnn. In ICCV, pp. 1440–1448. Cited by: §II-B.
  • [15] L. Guo, J. Liu, P. Yao, J. Li, and H. Lu (2019) MSCap: multi-style image captioning with unpaired stylized text. In CVPR, pp. 4204–4213. Cited by: §I.
  • [16] Y. Hao, N. Wang, J. Li, and X. Gao (2019) HSME: hypersphere manifold embedding for visible thermal person re-identification. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33, pp. 8385–8392. Cited by: §II-B.
  • [17] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In CVPR, pp. 770–778. Cited by: §III-A, §III-C, §IV-B, §IV-C, TABLE II.
  • [18] X. He, Q. Liu, and Y. Yang (2020) MV-gnn: multi-view graph neural network for compression artifacts reduction. IEEE Transactions on Image Processing. Cited by: §II-C.
  • [19] A. R. Hochschild (1998) The sociology of emotion as a way of seeing. Emotions in social life: Critical themes and contemporary issues, pp. 3–15. Cited by: §I.
  • [20] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei (2018) Relation networks for object detection. In CVPR, pp. 3588–3597. Cited by: §II-B.
  • [21] Y. Huang, J. Chen, W. Ouyang, W. Wan, and Y. Xue (2020) Image captioning with end-to-end attribute detection and subsequent attributes prediction. IEEE Transactions on Image Processing 29, pp. 4013–4026. Cited by: §II-B, §III-B1.
  • [22] J. Ji, C. Xu, X. Zhang, B. Wang, and X. Song (2020) Spatio-temporal memory attention for image captioning. IEEE Transactions on Image Processing. Cited by: §II-B.
  • [23] D. Joshi, R. Datta, E. Fedorovskaya, Q. Luong, J. Z. Wang, J. Li, and J. Luo (2011) Aesthetics and emotions in images. IEEE Signal Processing Magazine 28 (5), pp. 94–115. Cited by: §II-A.
  • [24] M. Katsurai and S. Satoh (2016)

    Image sentiment analysis using latent correlations among visual, textual, and sentiment views

    .
    In ICASSP, pp. 2837–2841. Cited by: §IV-A, §IV-B, TABLE I.
  • [25] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §IV-B.
  • [26] T. N. Kipf and M. Welling (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907. Cited by: §II-C, §III-B2.
  • [27] R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza (2017) Emotion recognition in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1667–1675. Cited by: §IV-F1, §IV-F2, §IV-F, TABLE V, TABLE VI.
  • [28] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. A. Shamma, et al. (2017) Visual genome: connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123 (1), pp. 32–73. Cited by: §III-A, §IV-B, §IV-D2.
  • [29] A. Krizhevsky, I. Sutskever, and G. E. Hinton (2012) Imagenet classification with deep convolutional neural networks. In NeurIPS, pp. 1097–1105. Cited by: §IV-C, TABLE II.
  • [30] R. D. Lane and L. Nadel (2002) Cognitive neuroscience of emotion. Oxford University Press. Cited by: §I.
  • [31] P. J. Lang, M. M. Bradley, B. N. Cuthbert, et al. (1997) International affective picture system (iaps): technical manual and affective ratings. NIMH Center for the Study of Emotion and Attention 1, pp. 39–58. Cited by: §I.
  • [32] P. J. Lang, M. M. Bradley, B. N. Cuthbert, et al. (1999) International affective picture system (iaps): instruction manual and affective ratings. The center for research in psychophysiology, University of Florida. Cited by: §IV-A, §IV-A, §IV-C, TABLE I.
  • [33] K. Li, Y. Zhang, K. Li, Y. Li, and Y. Fu (2019) Visual semantic reasoning for image-text matching. In ICCV, pp. 4654–4662. Cited by: §II-B, §III-B1, §III-B2.
  • [34] L. Li, H. Zhu, S. Zhao, G. Ding, and W. Lin (2020) Personality-assisted multi-task learning for generic and personalized image aesthetics assessment. IEEE Transactions on Image Processing 29, pp. 3898–3910. Cited by: §I.
  • [35] Y. Li, S. Zhou, X. Huang, L. Xu, Z. Ma, H. Fang, Y. Wang, and C. Lu (2019) Transferable interactiveness knowledge for human-object interaction detection. In CVPR, pp. 3585–3594. Cited by: §II-B.
  • [36] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel (2015) Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493. Cited by: §II-C.
  • [37] Z. Li, Y. Fan, B. Jiang, T. Lei, and W. Liu (2019) A survey on sentiment analysis and opinion mining for social multimedia. Multimedia Tools and Applications 78 (6), pp. 6939–6967. Cited by: §I.
  • [38] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A. C. Berg (2016) Ssd: single shot multibox detector. In ECCV, pp. 21–37. Cited by: §II-B.
  • [39] J. Machajdik and A. Hanbury (2010) Affective image classification using features inspired by psychology and art theory. In ACMMM, pp. 83–92. Cited by: §I, §II-A1, §IV-A, §IV-C, TABLE I.
  • [40] J. A. Mikels, B. L. Fredrickson, G. R. Larkin, C. M. Lindberg, S. J. Maglio, and P. A. Reuter-Lorenz (2005) Emotional category data on images from the international affective picture system. Behavior research methods 37 (4), pp. 626–630. Cited by: §IV-A, §IV-A, §IV-C, TABLE I.
  • [41] A. A. Mitchell (1986) The effect of verbal and visual components of advertisements on brand attitudes and attitude toward the advertisement. Journal of Consumer Research 13 (1), pp. 12–24. Cited by: §I.
  • [42] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer (2017) Automatic differentiation in pytorch. Cited by: §IV-B.
  • [43] K. Peng, A. Sadovnik, A. Gallagher, and T. Chen (2016) Where do emotions come from? predicting the emotion stimuli map. In ICIP, pp. 614–618. Cited by: §I, §II-A, §IV-A, §IV-B, TABLE I.
  • [44] J. Pennington, R. Socher, and C. D. Manning (2014) Glove: global vectors for word representation. In EMNLP, pp. 1532–1543. Cited by: §III-B1.
  • [45] E. A. Phelps (2006) Emotion and cognition: insights from studies of the human amygdala. Annual Review of Psychology 57, pp. 27–53. Cited by: §I.
  • [46] T. Rao, X. Li, and M. Xu (2016) Learning multi-level deep representations for image emotion classification. Neural Processing Letters, pp. 1–19. Cited by: §I, §II-A2, §II-A, §IV-C, TABLE II.
  • [47] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi (2016) You only look once: unified, real-time object detection. In CVPR, pp. 779–788. Cited by: §II-B.
  • [48] S. Ren, K. He, R. Girshick, and J. Sun (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In NeurIPS, pp. 91–99. Cited by: §II-B, §III-A.
  • [49] G. Salton and C. T. Yu (1973) On the construction of effective vocabularies for information retrieval. Acm Sigplan Notices 10 (1), pp. 48–60. Cited by: §IV-E1.
  • [50] S. Siersdorfer, E. Minack, F. Deng, and J. Hare (2010) Analyzing and predicting sentiment of images on the social web. In ACMMM, pp. 715–718. Cited by: §I, §II-A1.
  • [51] K. Simonyan and A. Zisserman (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Cited by: §IV-C, TABLE II.
  • [52] K. T. Strongman (1996) The psychology of emotion: theories of emotion in perspective. John Wiley & Sons. Cited by: §I.
  • [53] M. Sun, J. Yang, K. Wang, and H. Shen (2016) Discovering affective regions in deep convolutional neural networks for visual sentiment prediction. In ICME, pp. 1–6. Cited by: TABLE II.
  • [54] R. Szczepanowski, J. Traczyk, M. Wierzchoń, and A. Cleeremans (2013) The perception of visual emotion: comparing different measures of awareness. Consciousness and Cognition 22 (1), pp. 212–220. Cited by: §I.
  • [55] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio (2017) Graph attention networks. arXiv preprint arXiv:1710.10903. Cited by: §II-C.
  • [56] M. J. Wieser, E. Klupp, P. Weyers, P. Pauli, D. Weise, D. Zeller, J. Classen, and A. Mühlberger (2012) Reduced early visual emotion discrimination as an index of diminished emotion processing in parkinson’s disease?–evidence from event-related brain potentials. Cortex 48 (9), pp. 1207–1217. Cited by: §I.
  • [57] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei (2017) Scene graph generation by iterative message passing. In CVPR, pp. 5410–5419. Cited by: §II-B.
  • [58] H. Xu, C. Jiang, X. Liang, and Z. Li (2019) Spatial-aware graph relation network for large-scale object detection. In CVPR, pp. 9298–9307. Cited by: §III-B1.
  • [59] H. Xu, C. Jiang, X. Liang, L. Lin, and Z. Li (2019) Reasoning-rcnn: unifying adaptive global reasoning into large-scale object detection. In CVPR, pp. 6419–6428. Cited by: §II-B.
  • [60] J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh (2018) Graph r-cnn for scene graph generation. In ECCV, pp. 670–685. Cited by: §II-B.
  • [61] J. Yang, J. Li, L. Li, X. Wang, and X. Gao (2021) A circular-structured representation for visual emotion distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4237–4246. Cited by: §II-A.
  • [62] J. Yang, J. Li, X. Wang, Y. Ding, and X. Gao (2021) Stimuli-aware visual emotion analysis. IEEE Transactions on Image Processing. Cited by: §II-A2, §II-A.
  • [63] J. Yang, D. She, Y. Lai, P. L. Rosin, and M. Yang (2018) Weakly supervised coupled networks for visual sentiment analysis. In CVPR, pp. 7584–7592. Cited by: §I, §II-A2, §II-A, §IV-C, TABLE II.
  • [64] J. Yang, D. She, Y. Lai, and M. Yang (2018) Retrieving and classifying affective images via deep metric learning. In AAAI, Cited by: §I, §II-A2, §II-A, TABLE II.
  • [65] J. Yang, D. She, M. Sun, M. Cheng, P. L. Rosin, and L. Wang (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Transactions on Multimedia 20 (9), pp. 2513–2525. Cited by: §I, §II-A2, §II-A.
  • [66] J. Yang, D. She, and M. Sun (2017) Joint image emotion classification and distribution learning via deep convolutional neural network.. In IJCAI, pp. 3266–3272. Cited by: §I, §II-A, TABLE II.
  • [67] J. Yang, M. Sun, and X. Sun (2017)

    Learning visual sentiment distributions via augmented conditional probability neural network

    .
    In AAAI, Vol. 31. Cited by: §II-A.
  • [68] Q. You, H. Jin, and J. Luo (2017) Visual sentiment analysis by attending on local image regions. In AAAI, Cited by: §II-A2.
  • [69] Q. You, J. Luo, H. Jin, and J. Yang (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks. In AAAI, Cited by: §I, §II-A2, §II-A, §IV-A, TABLE I.
  • [70] Q. You, J. Luo, H. Jin, and J. Yang (2016) Building a large scale dataset for image emotion recognition: the fine print and the benchmark. In AAAI, Cited by: §IV-A, §IV-B, TABLE I.
  • [71] N. Yu, X. Hu, B. Song, J. Yang, and J. Zhang (2018) Topic-oriented image captioning based on order-embedding. IEEE Transactions on Image Processing 28 (6), pp. 2743–2754. Cited by: §II-B.
  • [72] H. Zeng, Z. Cao, L. Zhang, and A. C. Bovik (2019) A unified probabilistic formulation of image aesthetic assessment. IEEE Transactions on Image Processing 29, pp. 1548–1561. Cited by: §I.
  • [73] J. Zhang, F. Shen, X. Xu, and H. T. Shen (2020) Temporal reasoning graph for activity recognition. IEEE Transactions on Image Processing 29, pp. 5491–5506. Cited by: §II-B.
  • [74] W. Zhang, X. He, and W. Lu (2019) Exploring discriminative representations for image emotion recognition with cnns. IEEE Transactions on Multimedia 22 (2), pp. 515–523. Cited by: §II-A2, §IV-C, TABLE II.
  • [75] W. Zhang, X. He, X. Yu, W. Lu, Z. Zha, and Q. Tian (2019)

    A multi-scale spatial-temporal attention model for person re-identification in videos

    .
    IEEE Transactions on Image Processing 29, pp. 3365–3373. Cited by: §II-B.
  • [76] Z. Zhang, P. Luo, C. Loy, and X. Tang (2015) Learning social relation traits from face images. In ICCV, pp. 3631–3639. Cited by: §I.
  • [77] S. Zhao, G. Ding, Y. Gao, and J. Han (2017) Learning visual emotion distributions via multi-modal features fusion. In ACMMM, pp. 369–377. Cited by: §II-A.
  • [78] S. Zhao, G. Ding, Y. Gao, X. Zhao, Y. Tang, J. Han, H. Yao, and Q. Huang (2018)

    Discrete probability distribution prediction of image emotions with shared sparse learning

    .
    IEEE Transactions on Affective Computing 11 (4), pp. 574–587. Cited by: §II-A.
  • [79] S. Zhao, G. Ding, J. Han, and Y. Gao (2018) Personality-aware personalized emotion recognition from physiological signals. In IJCAI, pp. 1660–1667. Cited by: §II-A.
  • [80] S. Zhao, G. Ding, Q. Huang, T. Chua, B. W. Schuller, and K. Keutzer (2018) Affective image content analysis: a comprehensive survey. In IJCAI, pp. 5534–5541. Cited by: §II-A.
  • [81] S. Zhao, Y. Gao, X. Jiang, H. Yao, T. Chua, and X. Sun (2014) Exploring principles-of-art features for image emotion recognition. In ACMMM, pp. 47–56. Cited by: §I, §II-A1, §IV-C, §IV-C, TABLE II.
  • [82] S. Zhao, S. Wang, M. Soleymani, D. Joshi, and Q. Ji (2019) Affective computing for large-scale heterogeneous multimedia data: a survey. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 15 (3s), pp. 1–32. Cited by: §II-A.
  • [83] S. Zhao, H. Yao, Y. Gao, G. Ding, and T. Chua (2016) Predicting personalized image emotion perceptions in social networks. IEEE transactions on affective computing 9 (4), pp. 526–540. Cited by: §II-A.
  • [84] S. Zhao, H. Yao, Y. Gao, R. Ji, and G. Ding (2016) Continuous probability distribution prediction of image emotions via multitask shared sparse regression. IEEE transactions on multimedia 19 (3), pp. 632–645. Cited by: §II-A.
  • [85] S. Zhao, H. Yao, Y. Yang, and Y. Zhang (2014)

    Affective image retrieval via multi-graph learning

    .
    In ACMMM, pp. 1025–1028. Cited by: §I, §II-A1.
  • [86] S. Zhao, X. Yao, J. Yang, G. Jia, G. Ding, T. Chua, B. W. Schuller, and K. Keutzer (2021) Affective image content analysis: two decades review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §II-A.
  • [87] S. Zhao, X. Zhao, G. Ding, and K. Keutzer (2018) EmotionGAN: unsupervised domain adaptation for learning discrete probability distributions of image emotions. In ACMMM, pp. 1319–1327. Cited by: §II-A.
  • [88] L. Zhou, Y. Zhang, Y. Jiang, T. Zhang, and W. Fan (2019) Re-caption: saliency-enhanced image captioning through two-phase learning. IEEE Transactions on Image Processing 29, pp. 694–709. Cited by: §II-B.