Diverse and Styled Image Captioning Using SVD-Based Mixture of Recurrent Experts

07/07/2020 ∙ by Marzieh Heidari, et al. ∙ AUT 0

With great advances in vision and natural language processing, the generation of image captions becomes a need. In a recent paper, Mathews, Xie and He [1], extended a new model to generate styled captions by separating semantics and style. In continuation of this work, here a new captioning model is developed including an image encoder to extract the features, a mixture of recurrent networks to embed the set of extracted features to a set of words, and a sentence generator that combines the obtained words as a stylized sentence. The resulted system that entitled as Mixture of Recurrent Experts (MoRE), uses a new training algorithm that derives singular value decomposition (SVD) from weighting matrices of Recurrent Neural Networks (RNNs) to increase the diversity of captions. Each decomposition step depends on a distinctive factor based on the number of RNNs in MoRE. Since the used sentence generator gives a stylized language corpus without paired images, our captioning model can do the same. Besides, the styled and diverse captions are extracted without training on a densely labeled or styled dataset. To validate this captioning model, we use Microsoft COCO which is a standard factual image caption dataset. We show that the proposed captioning model can generate a diverse and stylized image captions without the necessity of extra-labeling. The results also show better descriptions in terms of content accuracy.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 5

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Generating human-like captions for images automatically, namely, image captioning, has risen as an interdisciplinary research issue at the crossing point of computer vision and natural language processing

[1, 2, 3, 4, 5, 6, 7]. It has numerous imperative industrial applications, such as assistant facilities for visually impaired individuals, visual knowledge in chatting robots, and photo sharing on social media. For producing genuine human-like image captions, an image captioning framework is required to understand the visual content of input image and write captions with proper linguistic properties. Nonetheless, most existing image captioning frameworks center around the vision side that describes the visual content in an objective and neutral manner (factual captions), while the language side, e.g. linguistic style, is regularly disregarded. In fact, linguistic style [8] is an essential factor that reflects human personality [9], influences purchasing decisions[10] and fosters social interactions [11]. The different styles in image captioning is also an important problem that has been referred by [11, 12, 5].

Generating styled, diverse and accurate captions for an image is an open challenge. For gaining diversity in generated captions some works require manually created, densely labeled, image caption datasets[13, 14, 15], some use GANs[16] to achieve diversity which mostly suffers from poor accuracy. Also For gaining style most works use styled datasets[17, 18].

We address the problem of necessity of gathering styled and densely labeled datasets to generate styled and diverse captions by presenting a novel unified model architecture that can generate styled, diverse and accurate captions without using extra labels and trained only on a standard image captioning dataset and a styled corpus.

Central to our approach is reducing the requirement of immense, densely labeled and styled dataset for image captioning. We propose a model for generating styled, diverse, and semantically relevant image captions containing an image embedder, a Term Generator, and a Sentence Generator. Image embedder is the one before the last layer of a pre-trained CNN that takes an image as input and outputs the visual features of the image. Term Generator is an MoRE that is responsible for diversity. Each RNN expert generates a specific word sequence. During the training of each RNN in Term Generator, at the end of each epoch, we filter out a part of deep network weights using SVD decomposition to generate diverse captions without an extra label. Previously SVD has been used for network compression

[19] and overfitting controlling [20], but this is the first time SVD is used for diverse captioning. Sentence Generator is responsible for controlling style. It learns style from a corpus of stylized text without aligned images. We evaluate our model on COCO dataset[21]. After the evaluation of sentences by each Term Generator expert, we extracted the vocabulary from their sentences. The vocabulary sets are different in both of the lengths and their content.

Our contribution is developing a new model that can generate styled, diverse, and accurate captions for images without training on a densely labeled dataset.

To discuss on some related works, in earlier image captioning studies, template-based models [22, 23] or retrieval-based models [24] were commonly used. The template-based models distinguish visual concepts from a given image and fill them into some well-defined formats to make sentences. In this way, the generations suffer from lack of diversity. The retrieval-based models discover the foremost reliable sentences from existing ones and are not able to produce new descriptions.

On the other hand, end-to-end trainable image captioning models are the result of recent advances in deep learning and the release of large scale datasets such as COCO

[25] and Flickr30k[26]. Most modern image captioning models use the encoder-decoder framework [2, 5, 7, 27, 28]

, where a convolutional neural network (CNN) encodes the input image in vector space feature embeddings which are fed into an RNN. The RNN takes the image encoding as input and the word generated in the current time step to generate a complete sentence one word at a time. Maximum likelihood estimation is typically used for training. It has been shown that attention mechanisms

[5, 29, 30] and high-level attributes/concepts [31, 32]

can help image captioning. Recently, reinforcement learning has been introduced for image captioning models to directly optimize task-specific metrics

[33, 34]. Some works adopt Generative Adversarial Networks (GANs)[16] to diverse or generate human-like captions [35]. Some works have been also adapted by weakly-supervised training methods [36] for making richer captions.

1.1 Diverse Image Captioning

Generating diverse captions for images and videos has been studied in the recent years[37, 38, 39, 40, 41, 42]. Some techniques such as GAN-based methods [39, 37] and VAE-based methods [38, 41] are used to improve the diversity and accuracy of descriptions. Some others [43, 37] studied generating descriptive paragraphs for images. Also in [44], a method has been proposed to apply the part-of-speech of words in generated sentences. In [45], generated sentences can contain words of different topics. The research of [13], generates descriptions for each semantic informative region in images. In addition, in [46], a particular guiding object that is presented in the image is chosen to be necessarily presented in the generated description. Most of these approaches require additional labels in a dataset, e.g. Visual Genome [47]. In what follows, we propose a new scheme without the necessity to give additional labels.

1.2 Stylized Image Captioning

Stylized image captioning aims at generating captions that are successfully stylized and describe the image content accurately. The works proposed for tackling this task can be divided into two categories: models using parallel stylized image-caption data (the supervised mode) [17, 48, 49, 50] and models using non-parallel stylized corpus (the semi-supervised mode) [18, 1]. SentiCap [17]

handles the positive/negative styles and proposes to model word changes with two parallel Long Short Term Memory (LSTM) networks and word-level supervision. StyleNet

[18] handles the humorous/romantic styles by factoring the input weight matrices to contain a style specific factor matrix. SF-LSTM [48] experiments on the above four caption styles and proposes to learn two groups of matrices to capture the factual and stylized knowledge, respectively. You et al. [50] proposed two simple methods to inject sentiments into image captions. They could control the sentiment by providing different sentiment labels. See and Chat [51] in the first step retrieved the visually similar images form their dataset which contains 426K images with 11 million associated comments, to input image using k nearest neighbors and then ranked their comments to get the most relevant comment for the input image. There are more than 25 comments for each image on average. Since the comments of the dataset are styled, the resulted caption was styled too.

2 A New Model with a Mixture of Experts

As stated in Section 1.1, the current image captioning methods need extra labels on the training data to generate stylized captions. In this section, we propose a novel image captioning model that applies diversity and style to the generated captions without requiring additional labels. The goal of our proposed model is to take an image, a target style, and a diversity factor as input and generate a sentence within the target style. To find better results, we use an ensemble of neural networks. To this aim, different kinds of ensemble models can be used. For example, ensemble of neural networks in tree structure [52, 53] or mixture of experts [54, 55] can be followed. We follow a Mixture of Recurrent Experts (MoRE) in what follows.

The architecture of the proposed model is illustrated in Figure 2

. The model is comprised of three basic components, i.e., an image encoder, a Term Generator, and a Sentence Generator. The image encoder is a CNN that extracts visual features from the input image. The Term Generator is a mixture of some RNNs that takes visual features extracted by CNN and SVD factors as input and gives a sequence of semantic terms. The Sentence Generator is an attention-based RNN that takes this sequence of semantic terms and the target style and decodes them into a sentence in natural language with a specific style that describes the image. Each RNN of the Term Generator component has a specific SVD factor. The SVD factor shows what portion of the RNN weight matrix is saved during training. At test time, the SVD factors determine which one of the experts is responsible to generate the sequence of semantic terms from visual features.

Figure 1: The results of applying different SVD factors to approximate weight matrix. The image is from COCO [21].

Figure 2: The architecture of MoRE to generate the multi-style captions with target style. SVD Factor indicates what portion of matrix rank is remained in the deep network. The output of each expert is supposed as the input of Sentence Generator.The attention is from SemStyle [1]

Each Term Generator uses a different SVD factor to cause diversity in extracted words and consequently diversity in the generated captions. Moreover, we designed a two-stage learning strategy to train the Term Generator networks and the Sentence Generator network separately. We train the Term Generators on a dataset of image caption pairs and the Sentence Generator on a corpus of styled text data such as romantic novels.

2.1 Image Encoder

This module encodes any image to get features utilizing a deep CNN. Previous studies use different types of image features. The image features could be local visual features for every semantic segment of image [7] or a static global representation of the image [5]. A visual context vector is obtained by directly using the static feature or calculating adaptively with a soft-attention mechanism[7, 56]. In this paper, we use the static features to remain consistency with the previous works. The image features are extracted from the second last layer of the Inception-v3[57]

of CNN pre-trained on ImageNet

[58].

2.2 Term Generator

The Term Generator network is an MoRE that maps an input image, denoted by to an ordered sequences of semantic terms . Each semantic term is a word with a part-of-speech tag. These words indicate the objects, scene, and activity in the image. This generator should completely capture the visual semantics and should be independent of linguistic style; because Sentence Generator is responsible for applying a style to caption.

Our MoRE is trained by an SVD based approximation method inspired by[59]. For a learnt weight matrix , by approximating by an SVD, one can find

(1)

where is a diagonal matrix with singular values on the diagonal in the decreasing order. columns of and columns of are called the left-singular vectors and the right-singular vectors of , respectively. By approximating by the greatest components of this decomposition, one can substitute the following instead of

(2)

By changing , the different variations of the weighting matrix can be defined for Term Generator networks and each approximation interprets any image differently. Thus, the outputs will be variant. In our model, for RNN, we define a diversity factor where is the number of RNNs in MoRE. For each expert of MoRE, shows the portion of the principal components of matrix that remains in the learning model. Noe that denotes the rank of The effect of different diversity factors on the generating the sequence of semantic terms is shown in Fig 1.

The architecture of all experts of MoRE are similar and is a CNN+RNN inspired by Show and Tell [5], see the middle part of Fig.  2

. The image feature vector passes through a densely connected layer and then through an RNN with Gated Recurrent Unit (GRU) cells

[60]. The word list is shorter than a full sentence, which speeds up training and alleviates the effect of forgetting long sequences. At each time-step , there are two inputs to the GRU cell. The first is the previous hidden state summarizing the image and word history . The second is the GloVe embedding vector of the current word. A fully connected layer with softmax takes the output and produces a categorical distribution for the next word in the sequence

. Argmax decoding can be used to recover the entire word sequence from the conditional probabilities. See Eq. (1) in

[1]. We set as the beginning-of-sequence token and terminate when the sequence exceeds a maximum length or when the end-of-sequence token is generated.

2.3 Sentence Generator

The Sentence Generator, shown in the upper part of Fig. 2, maps the sequence of semantic terms to a sentence with a specific style. For example, given the word list “girl”, “posture”, “refrigerator”, and “DESCRIPTIVE” as the requested style, a suitable caption is “A girl standing in a kitchen beside a refrigerator.” Also the same list of words with “STORY” as the expected style as the input is “I saw the girl standing in the kitchen, and I was staring at the refrigerator”. Given the list of words and a target style we generate an output caption where and is the output word vocabulary. To do so, the idea of [1] is used by considering an RNN sequence-to-sequence sentence generator network with attention over the input sequence. This is an auto-encoder that maps input word sequence to a vector space and decodes the sequence to a sentence to describe the image in a suitable style. Encoder component for sequence

consists of a GloVe vector embedding followed by a batch normalization layer

[61] and a bidirectional RNN [62] with GRU cells. The Bidirectional RNN [1] is implemented as two independent RNNs. They run in opposite directions with shared embeddings. For details, we refer to Eq. (4) of [1].

Figure 3: Some results to compare the description of our model with that of SemStyle[1]. Green color shows enhancement in accuracy and blue color indicates the improvement of style. The images are from COCO[21].

3 Experimental Setup

We conduct experiments on publicly available image caption dataset, Microsoft COCO [25]. COCO is a large image captioning dataset, containing 82783, 40504, and 40775 images for training, validation, and test, respectively. Each image is labeled with 5 human-generated descriptions for image captioning. All labels are converted to lower case and tokenized. We use Semstyle training and testing split sets [1] for both factual and stylized captions.

We consider 9 baseline methods and compare them with 5 variants of our proposed captioning model. The considered baselines are as the following:

  • Show and Tell [5] constructed a CNN as an encoder and an RNN as the decoder.

  • Neural Talk [63] used the images and their regions alignments to captions to learn and to generate descriptions of image regions.

  • StyleNet [18] originally has been trained on FlickrStyle10K [18]. The implementation of StyleNet and StyleNet-COCO are from [18]. This reimplementation makes the trained datasets match and consequently makes the approaches comparable. StyleNet generates styled captions, while StyleNet-COCO generates descriptive captions.

  • Neural-storyteller [64] is a model trained on romance text (from the same source as ours).

  • JointEmbedding maps images and sentences to a continuous multi-modal vector space [65] and uses a separate decoder, that has been trained on the romance text, to decode from this space.

  • SemStyle [1] is our reference to develop the model and maps image features to a word sequence and then maps the sequence to a caption.

  • SGC [4] applies the Scene Graph Captioner (SGC) framework for the image captioning task.

  • Hierarchical Attention [3]

    uses a hierarchical attention model by utilizing both of the global CNN features and the local object features for more effective feature representation and reasoning in image captioning.

  • VSV-VRA-POS [2] the adapts the language models for word generation to the specific syntactic structure of sentences and visual skeleton of the image.

The variants of our proposed captioning model are as the following:

  1. Shuffled words model is genuinely base-line that during the training of the Sentence Generator, the input words are out of order. This gives a little noise to input and the results are less overfitted.

  2. Shuffles words+batch normalization model is "Shuffled words" model that uses a batch normalization layer after the embedding layer of Sentence Generator. This makes features more general and consequently more general captions.

  3. Shuffles words+Glove+batch normalization model uses freezed weights of Glove pre-trained embedding.

  4. Full model applies a specific drop-out to embedding layers.

  5. Kaldi GRU is a full model that uses Kaldi Speech Recognition [66] GRUs as encoder and decoder in Sentence Generator instead of typical GRUs.

3.1 Evaluation Metrics

We use two types of metrics to evaluate the proposed image captioning model. The first type is automatic relevance metrics. In this part, similar to [1], we consider captioning metrics including BLEU [67], METEOR[68], ROUGE_L [69], and CIDEr [70]

which are based on n-gram overlap and SPICE

[71]

based on f-score over semantic tuples extracted from COCO reference sentences

[21]

. As the second type of metric, we consider automatic style metrics. In this part, we measure how often a generated caption has the correct target-style according to a pre-trained style classifier. The CLassifier Fraction (CLF) metric

[1]

, is the fraction of generated captions classified as styled by a binary classifier. This classifier is logistic regression with 1,2-gram occurrence features trained on styled sentences and COCO training captions. Its cross-validation precision is 0.992 at a recall of 0.991.

3.2 Training details

In our experiments, the model is optimized with Adam [72]. The learning rate is set to 1e-3. We clip gradients to [-5, 5] and apply dropout to image and sentence embeddings. The mini-batch size is 64 for both the Term Generator and the Sentence Generator. Both the Term Generator and Sentence Generator use separate 512-dimensional GRUs and word embedding vectors. The Term Generator has a vocabulary of 10000 words while the Sentence Generator has two vocabularies: one for encoder input another for the decoder – both vocabularies have 20000 entries to account for a broader scope. The number of intersecting words between the Term Generator and the Sentence Generator is 8266 with both datasets, and 6736 without. Image embeddings come from the second last layer of the Inception-v3 CNN[57] and are 2048 dimensional.

We used Glove [73] frozen weights for embedding layers of both Term Generator and Sentence Generator of Semstyle’s baseline. The model suffered from overfitting so we adopt the following regularization techniques in order to fix this problem:

  • Instead of normal drop-out, we used embedding-specific drop-out proposed by Merity et al. [74].

  • We adopt batch normalization [61] for both modules of the model including Term Generator and Sentence Generator.

  • For Term Generator, we used weight decay [75] with a coefficient of 1e-6.

  • In training Sentence Generator instead of feeding ordered semantic terms, we shuffled them so the model learns to generate sentences from unordered semantic terms that improve the generalization.

We trained three Term Generator experts. Each expert of MoRE applies SVD on the weighting matrices in the Term Generator model, and afterward reconstruct the model based on the inherent sparseness of the original matrices. For each expert, we saved of principal components of learned weight matrix for SVD approximation. This approach has been used by [59] for the neural network training process. After reconstruction, the accuracy decreases but the final classification results improve. After every epoch, the reduced weights are replaced in the model. Afterward, we fine-tune the reconstructed model using the back-propagation method to receive better accuracy.

3.3 Results

The results of measurements are presented in Table 1. Table 2 shows the results of the automatic metrics on caption style learned from romance novels. This comparison is similar to [1]. Also, our full model generates descriptive captions. It accomplishes semantic relevance scores comparable to the Show and Tell [5], with a SPICE of 0.166 vs 0.154, and BLEU-4 of 0.252 vs 0.238. Thus utilizing semantic words is a competitive way to distill image semantics. Really, the Term Generator and the Sentence Generator constitute a compelling vision-to-language pipeline. Additionally, our model can create different caption styles, where the CLF metric for captions classification is 99.995%, when the target style is descriptive. Figure 3 demonstrates some quantitative results for different styles alongside results of the same images generated by SemStyle which is the most similar approach to ours.

In addition, our model generates styled captions in 74.1% of cases, based on CLF. SPICE score is 0.145 which is better than the presented baselines. Results are shown in Table  2.

As one can see in Table 2 there are three models (SGC [4], VSV-VRA-POS [2] and Hierarchical Attention [3] ) with better relevance scores compared with our work. These three works are presented to show the impact of using unpaired captions and a unified model for multi-style captioning. The trade-off for such a model would be the loss of relevance scores. Because when you have only one objective (which is generating similar captions to ground-truth) the similarity score would be higher compared with the works that peruse more objects. Since, our model tries to improve the captions for all styles, our method can outperform other one-objective methods, in real situations.

3.4 Component Analysis

According to Table 3 and Table 4 the poorest result is gotten when Kaldi GRU is used instead of regular GRU cells. By shuffling the input words of Sentence Generator input in training, we can see a little improvement which is the result of decreasing overfit caused by adding this noise. Adding a Batch Normalization layer boosts the results so that in some metrics such as BLEU3, BLEU4, and Cider the best result is achieved by this model. Adding Glove frozen weights only improves styled captioning by increasing the number of styled captions by 10%. But it decreases other scores slightly. This score-dropping is the result of overfitting so in the full model, we added embedding-specific dropout layer to fix this issue. As a result, relevance scores boost up again and in addition, another 12% added to the styled caption on style evaluation.

Figure 4: Words scattering for generated captions by three models with three different SVD factors on the weighting matrices. The numbers are the count of words and different colors on different Term Generators inside the MoRE.
model

Multi-Style

Unpaired

BLEU1

BLEU2

BLEU3

BLEU4

METEOR

ROUGE_L

CIDEr

SPICE

Show and Tell [5] no no 0.667 0.238 0.224 0.772 0.154
Neural Talk[63] no no 0.625 0.45 0.23 0.23 0.195 0.66
StyleNet (COCO) [18] no yes 0.643 0.212 0.218 0.664 0.135
SGC [4] no no 0.679 0.493 0.347 0.243 0.222 0.754 0.488 -
Hierarchical Attention [3] no no 0.7261 0.5277 0.3780 0.2724 0.2473 0.8814 0.5604 -
VSV-VRA-POS [2] no no 0.782 0.619 0.477 0.368 0.277 0.572 1.159 0.207
SemStyle (COCO) [1] yes yes 0.653 0.478 0.337 0.238 0.219 0.482 0.769 0.157
MoRE (ours) yes yes 0.679 0.501 0.356 0.252 0.226 0.501 0.844 0.166
Table 1: Evaluating caption relevance on the COCO dataset.
Model Multi-Style Unpaired SPICE CLF
StyleNet [18] no yes 0.010 0.415
neural-storyteller[64] no no 0.057 0.983
JointEmbedding [65] no no 0.046 0.99
SemStyle (ROM) [1] yes yes 0.144 0.589
MoRE(ours) yes yes 0.145 0.741
Table 2: Evaluating styled captions with automated metrics.
model

BLEU1

BLEU2

BLEU3

BLEU4

METEOR

ROUGE_L

CIDEr

SPICE

kaldi GRU 0.606 0.437 0.297 0.201 0.204 0.464 0.632 0.145
shuffles words 0.655 0.486 0.347 0.247 0.223 0.497 0.813 0.163
shuffles words+batch normalization 0.664 0.494 0.357 0.258 0.226 0.5 0.837 0.164
shuffles words+embedding+batch normalization 0.657 0.49 0.353 0.254 0.224 0.498 0.83 0.163
full model 0.679 0.501 0.356 0.252 0.226 0.501 0.844 0.166
Table 3: Experimental result for descriptive style.
model SPICE CLF
shuffles words 0.147 0.553
shuffles words+batch normalization 0.153 0.503
shuffles words+Glove+batch normalization 0.150 0.605
full model 0.145 0.741
Table 4: Experimental result for romance style.

SVD factor

total words count

wps mean

wps std

BLEU1

BLEU2

BLEU3

BLEU4

METEOR

ROUGE_L

CIDEr

SPICE

656 8.35 1.49 0.672 0.493 0.348 0.244 0.222 0.497 0.808 0.161
700 8.45 1.59 0.674 0.497 0.354 0.250 0.226 0.500 0.827 0.162
1 791 8.43 1.53 0.679 0.501 0.356 0.252 0.226 0.501 0.844 0.166
Table 5: Experimental result for diversity on descriptive style. wps is the word per sentence

For evaluating diversity in generated sentences using different SVD factors, we counted words of generated captions. As shown in Table 5

, the number of words and the average of words count per sentence are decreasing as much as the factor decreases. Different factors for model aims at producing more diverse and novel captions which may not appear in the ground truths. Then, their similarity metric scores are generally less than the full model since fewer n-grams match with ground truths. Therefore, these metrics particularly represent a quality of pattern matching, instead of overall quality from the human perspective.

In Figure 4, all models are trained by the same set of vocabulary. The extracted vocabulary from generated captions for all models is not completely similar. This indicates diversity in generated captions by different models of MoRE.

4 Conclusion

We have proposed a multi-style, diverse image captioning model by using the unpaired stylized corpus. This model includes the following components:

  • A CNN as the feature extractor

  • A Mixture of RNNs (MoRE) to embed features into a set of words

  • An RNN that gets the output of MoRE and generates a sentence as the final output

Our model can generate human-like, appropriately stylized, visually grounded, and style-controllable captions. Besides, the captions made rich and diverse using a mixture of experts. The results on the COCO dataset, show that the performance of our proposed captioning model is better than previous works in case of accuracy, diversity, and styled captions. This improves the results of the previous works in BLEU, CIDEr, SPICE, ROUGE_L, and CLF metrics. For the future works, one can consider the different ensemble methods instead of MoRE. Also to avoid overfitting, the different methods can be compared [76] to obtain the most effective one.

References

  • [1] Alexander Mathews, Lexing Xie, and Xuming He. Semstyle: Learning to generate stylised image captions using unaligned text. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    , pages 8591–8600, 2018.
  • [2] Liang Yang and Haifeng Hu. Visual skeleton and reparative attention for part-of-speech image captioning system. Computer Vision and Image Understanding, 189:102819, 2019.
  • [3] Shiyang Yan, Yuan Xie, Fangyu Wu, Jeremy S Smith, Wenjin Lu, and Bailing Zhang. Image captioning via hierarchical attention mechanism and policy gradient optimization. Signal Processing, 167:107329, 2020.
  • [4] Ning Xu, An-An Liu, Jing Liu, Weizhi Nie, and Yuting Su. Scene graph captioner: Image captioning based on structural visual representation. Journal of Visual Communication and Image Representation, 58:477–485, 2019.
  • [5] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015.
  • [6] Jinhui Tang, Xiangbo Shu, Zechao Li, Guo-Jun Qi, and Jingdong Wang. Generalized deep transfer networks for knowledge propagation in heterogeneous domains. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 12(4s):68
  • [7] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In

    International conference on machine learning

    , pages 2048–2057, 2015.
  • [8] Allan Bell. Language style as audience design. Language in society, 13(2):145–204
  • [9] James W. Pennebaker and Laura A. King. Linguistic styles: Language use as an individual difference. Journal of personality and social psychology, 77(6):1296 1939–1315, 1999.
  • [10] Stephan Ludwig, Ko De Ruyter, Mike Friedman, Elisabeth C. Brüggen, Martin Wetzels, and Gerard Pfann. More than words: The influence of affective content and linguistic style matches in online reviews on conversion rates. Journal of Marketing, 77(1):87–103
  • [11] Cristian Danescu-Niculescu-Mizil, Michael Gamon, and Susan Dumais. Mark my words!: linguistic style accommodation in social media. In Proceedings of the 20th international conference on World wide web, pages 745–754. ACM, 2011.
  • [12] Ellie Pavlick and Ani Nenkova. Inducing lexical style properties for paraphrase and genre differentiation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 218–224, 2015.
  • [13] Justin Johnson, Andrej Karpathy, and Li Fei-Fei.

    Densecap: Fully convolutional localization networks for dense captioning.

    In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4565–4574, 2016.
  • [14] Mark Yatskar, Michel Galley, Lucy Vanderwende, and Luke Zettlemoyer. See no evil, say no evil: Description generation from densely labeled images. In Proceedings of the Third Joint Conference on Lexical and Computational Semantics (* SEM 2014), pages 110–120, 2014.
  • [15] Linjie Yang, Kevin Tang, Jianchao Yang, and Li-Jia Li. Dense captioning with joint inference and visual context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2193–2202, 2017.
  • [16] Dianqi Li, Qiuyuan Huang, Xiaodong He, Lei Zhang, and Ming-Ting Sun. Generating diverse and accurate visual captions by comparative adversarial learning. arXiv preprint arXiv:1804.00861, 2018.
  • [17] Alexander Patrick Mathews, Lexing Xie, and Xuming He. Senticap: Generating image descriptions with sentiments. In

    Thirtieth AAAI conference on artificial intelligence

    , 2016.
  • [18] Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. Stylenet: Generating attractive visual captions with styles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3137–3146, 2017.
  • [19] Koen Goetschalckx, Bert Moons, Patrick Wambacq, and Marian Verhelst. Efficiently combining svd, pruning, clustering and retraining for enhanced neural network compression. In Proceedings of the 2nd International Workshop on Embedded and Mobile Deep Learning, pages 1–6, 2018.
  • [20] Mohammad Mahdi Bejani and Mehdi Ghatee. Adaptive svd regularization for deep neural networks learning systems. Neural Networks, 128:33–46, 2020.
  • [21] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
  • [22] Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Babytalk: Understanding and generating simple image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2891–2903, 2013.
  • [23] Desmond Elliott and Arjen de Vries. Describing images using inferred visual dependency representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 42–52, 2015.
  • [24] Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell. Language models for image captioning: The quirks and what works. arXiv preprint arXiv:1505.01809, 2015.
  • [25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
  • [26] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  • [27] Jing Wang, Jianlong Fu, Jinhui Tang, Zechao Li, and Tao Mei. Show, reward and tell: Automatic generation of narrative paragraph from photo stream by adversarial training. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  • [28] Zhilin Yang, Ye Yuan, Yuexin Wu, William W Cohen, and Ruslan R Salakhutdinov. Review networks for caption generation. In Advances in Neural Information Processing Systems, pages 2361–2369, 2016.
  • [29] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375–383, 2017.
  • [30] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6077–6086, 2018.
  • [31] Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. Boosting image captioning with attributes. In Proceedings of the IEEE International Conference on Computer Vision, pages 4894–4902, 2017.
  • [32] Luowei Zhou, Chenliang Xu, Parker Koch, and Jason J. Corso. Image caption generation with text-conditional semantic attention. arXiv preprint arXiv:1606.04621, 2, 2016.
  • [33] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008–7024, 2017.
  • [34] Li Zhang, Flood Sung, Feng Liu, Tao Xiang, Shaogang Gong, Yongxin Yang, and Timothy M. Hospedales. Actor-critic sequence training for image captioning. arXiv preprint arXiv:1706.09601, 2017.
  • [35] Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine to human captions by adversarial training. In Proceedings of the IEEE International Conference on Computer Vision, pages 4135–4144, 2017.
  • [36] Hai-Tao Zheng, Zhe Wang, Ningning Ma, Jinyuan Chen, Xi Xiao, and Arun Kumar Sangaiah. Weakly-supervised image captioning based on rich contextual information. Multimedia Tools and Applications, 77(14):18583–18599, 2018.
  • [37] Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. Towards diverse and natural image descriptions via a conditional gan. In Proceedings of the IEEE International Conference on Computer Vision, pages 2970–2979, 2017.
  • [38] Unnat Jain, Ziyu Zhang, and Alexander G Schwing.

    Creativity: Generating diverse questions using variational autoencoders.

    In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6485–6494, 2017.
  • [39] Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine to human captions by adversarial training. In Proceedings of the IEEE International Conference on Computer Vision, pages 4135–4144, 2017.
  • [40] Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424, 2016.
  • [41] Liwei Wang, Alexander Schwing, and Svetlana Lazebnik. Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space. In Advances in Neural Information Processing Systems, pages 5756–5766, 2017.
  • [42] Mingxing Zhang, Yang Yang, Hanwang Zhang, Yanli Ji, Heng Tao Shen, and Tat-Seng Chua. More is better: Precise and detailed image captioning using online positive recall and missing concepts mining. IEEE Transactions on Image Processing, 28(1):32–44, 2018.
  • [43] Moitreya Chatterjee and Alexander G Schwing. Diverse and coherent paragraph generation from images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 729–744, 2018.
  • [44] Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander Schwing, and David A Forsyth. Diverse and controllable image captioning with part-of-speech guidance. arXiv preprint arXiv:1805.12589, 2(8), 2018.
  • [45] Yuzhao Mao, Chang Zhou, Xiaojie Wang, and Ruifan Li. Show and tell more: Topic-oriented multi-sentence image captioning. In IJCAI, pages 4258–4264, 2018.
  • [46] Yue Zheng, Yali Li, and Shengjin Wang. Intention oriented image captions with guiding objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8395–8404, 2019.
  • [47] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017.
  • [48] Tianlang Chen, Zhongping Zhang, Quanzeng You, Chen Fang, Zhaowen Wang, Hailin Jin, and Jiebo Luo. “factual”or“emotional”: Stylized image captioning with adaptive learning and attention. In Proceedings of the European Conference on Computer Vision (ECCV), pages 519–535, 2018.
  • [49] Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. Engaging image captioning via personality. arXiv preprint arXiv:1810.10665, 2018.
  • [50] Quanzeng You, Hailin Jin, and Jiebo Luo. Image captioning at will: A versatile scheme for effectively injecting sentiments into image descriptions. arXiv preprint arXiv:1801.10121, 2018.
  • [51] Jingwen Chen, Ting Yao, and Hongyang Chao. See and chat: automatically generating viewer-level comments on images. Multimedia Tools and Applications, 78(3):2689–2702, 2019.
  • [52] Shadi Abpeykar and Mehdi Ghatee.

    An ensemble of rbf neural networks in decision tree structure with knowledge transferring to accelerate multi-classification.

    Neural Computing and Applications, 31(11):7131–7151, 2019.
  • [53] Shadi Abpeykar, Mehdi Ghatee, and Hadi Zare.

    Ensemble decision forest of rbf networks via hybrid feature clustering approach for high-dimensional data classification.

    Computational Statistics & Data Analysis, 131:12–36, 2019.
  • [54] Elham Abbasi, Mohammad Ebrahim Shiri, and Mehdi Ghatee. A regularized root–quartic mixture of experts for complex classification problems. Knowledge-Based Systems, 110:98–109, 2016.
  • [55] Ali Pashaei, Mehdi Ghatee, and Hedieh Sajedi. Convolution neural network joint with mixture of extreme learning machines for feature extraction and classification of accident images. Journal of Real-Time Image Processing, pages 1–16, 2019.
  • [56] Xinwei He, Yang Yang, Baoguang Shi, and Xiang Bai. Vd-san: Visual-densely semantic attention network for image caption generation. Neurocomputing, 328:48–55, 2019.
  • [57] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  • [58] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.
  • [59] Jian Xue, Jinyu Li, and Yifan Gong. Restructuring of deep neural network acoustic models with singular value decomposition. In Interspeech, pages 2365–2369, 2013.
  • [60] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
  • [61] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
  • [62] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681, 1997.
  • [63] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
  • [64] Jamie Ryan Kiros. neural-storyteller, a recurrent neural network for generating little stories about images. available at<>, GitHub, Inc., retrieved on Nov, 26:4, 2016.
  • [65] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014.
  • [66] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. The kaldi speech recognition toolkit. In

    IEEE 2011 workshop on automatic speech recognition and understanding

    , number CONF. IEEE Signal Processing Society, 2011.
  • [67] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics, 2002.
  • [68] Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376–380, 2014.
  • [69] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  • [70] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  • [71] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European Conference on Computer Vision, pages 382–398. Springer, 2016.
  • [72] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  • [73] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  • [74] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm language models. arXiv preprint arXiv:1708.02182, 2017.
  • [75] Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pages 950–957, 1992.
  • [76] Mohammad Mahdi Bejani and Mehdi Ghatee. Overfitting control in shallow and deep neural networks: A systematic review. Artificial Intelligence Review, 2020.