Gated Convolutional Neural Networks for Domain Adaptation

05/16/2019 ∙ by Avinash Madasu, et al. ∙ SAMSUNG 0

Domain Adaptation explores the idea of how to maximize performance on a target domain, distinct from source domain, upon which the classifier was trained. This idea has been explored for the task of sentiment analysis extensively. The training of reviews pertaining to one domain and evaluation on another domain is widely studied for modeling a domain independent algorithm. This further helps in understanding correlation between domains. In this paper, we show that Gated Convolutional Neural Networks (GCN) perform effectively at learning sentiment analysis in a manner where domain dependant knowledge is filtered out using its gates. We perform our experiments on multiple gate architectures: Gated Tanh ReLU Unit (GTRU), Gated Tanh Unit (GTU) and Gated Linear Unit (GLU). Extensive experimentation on two standard datasets relevant to the task, reveal that training with Gated Convolutional Neural Networks give significantly better performance on target domains than regular convolution and recurrent based architectures. While complex architectures like attention, filter domain specific knowledge as well, their complexity order is remarkably high as compared to gated architectures. GCNs rely on convolution hence gaining an upper hand through parallelization.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

With the advancement in technology and invention of modern web applications like Facebook and Twitter, users started expressing their opinions and ideologies at a scale unseen before. The growth of e-commerce companies like Amazon, Walmart have created a revolutionary impact in the field of consumer business. People buy products online through these companies and write reviews for their products. These consumer reviews act as a bridge between consumers and companies. Through these reviews, companies polish the quality of their services. Sentiment Classification (SC) is one of the major applications of Natural Language Processing (NLP) which aims to find the polarity of text. In the early stages


of text classification, sentiment classification was performed using traditional feature selection techniques like Bag-of-Words (BoW)


or TF-IDF. These features were further used to train machine learning classifiers like Naive Bayes (NB)


and Support Vector Machines (SVM)

[4]. They are shown to act as strong baselines for text classification [5]

. However, these models ignore word level semantic knowledge and sequential nature of text. Neural networks were proposed to learn distributed representations of words

[6]. Skip-gram and CBOW architectures [7] were introduced to learn high quality word representations which constituted a major breakthrough in NLP. Several neural network architectures like recursive neural networks [8] and convolutional neural networks [9]

achieved excellent results in text classification. Recurrent neural networks which were proposed for dealing sequential inputs suffer from vanishing


and exploding gradient problems


. To overcome this problem, Long Short Term Memory (LSTM) was introduced


All these architectures have been successful in performing sentiment classification for a specific domain utilizing large amounts of labelled data. However, there exists insufficient labelled data for a target domain of interest. Therefore, Domain Adaptation (DA) exploits knowledge from a relevant domain with abundant labeled data to perform sentiment classification on an unseen target domain. However, expressions of sentiment vary in each domain. For example, in Books domain, words thoughtful and comprehensive are used to express sentiment whereas cheap and costly are used in Electronics domain. Hence, models should generalize well for all domains. Several methods have been introduced for performing Domain Adaptation. Blitzer [13] proposed Structural Correspondence Learning (SCL) which relies on pivot features between source and target domains. Pan [14] performed Domain Adaptation using Spectral Feature Alignment (SFA) that aligns features across different domains. Glorot [15]

proposed Stacked Denoising Autoencoder (SDA) that learns generalized feature representations across domains. Zheng

[16] proposed end-to-end adversarial network for Domain Adaptation. Qi [17] proposed a memory network for Domain Adaptation. Zheng [18] proposed a Hierarchical transfer network relying on attention for Domain Adaptation.

However, all the above architectures use a different sub-network altogether to incorporate domain agnostic knowledge and is combined with main network in the final layers. This makes these architectures computationally intensive. To address this issue, we propose a Gated Convolutional Neural Network (GCN) model that learns domain agnostic knowledge using gated mechanism [19]. Convolution layers learns the higher level representations for source domain and gated layer selects domain agnostic representations. Unlike other models, GCN doesn’t rely on a special sub-network for learning domain agnostic representations. As, gated mechanism is applied on Convolution layers, GCN is computationally efficient.

Figure 1: Architecture of the proposed model

2 Related Work

Traditionally methods for tackling Domain Adaptation are lexicon based. Blitzer

[20] used a pivot method to select features that occur frequently in both domains. It assumes that the selected pivot features can reliably represent the source domain. The pivots are selected using mutual information between selected features and the source domain labels. SFA [14] method argues that pivot features selected from source domain cannot attest a representation of target domain. Hence, SFA tries to exploit the relationship between domain-specific and domain independent words via simultaneously co-clustering them in a common latent space. SDA [15] performs Domain Adaptation by learning intermediate representations through auto-encoders. Yu [21] used two auxiliary tasks to help induce sentence embeddings that work well across different domains. These embeddings are trained using Convolutional Neural Networks (CNN).

Gated convolutional neural networks have achieved state-of-art results in language modelling [19]. Since then, they have been used in different areas of natural language processing (NLP) like sentence similarity [22] and aspect based sentiment analysis [23].

3 Gated Convolutional Neural Networks

In this section, we introduce a model based on Gated Convolutional Neural Networks for Domain Adaptation. We present the problem definition of Domain Adaptation, followed by the architecture of the proposed model.

3.1 Problem Definition

Given a source domain represented as = {,….} where represents the vector of source text and represents the corresponding source domain label. Let represent the task in source domain. Given a target domain represented as = {,….}, where represents the vector of target text and represents corresponding target domain label. Let represent the task in target domain. Domain Adaptation (DA) is defined by the target predictive function calculated using the knowledge of and where but . It is imperative to note that the domains are different but only a single task. In this paper, the task is sentiment classification.

3.2 Model Architecture

The proposed model architecture is shown in the Figure 1

. Recurrent Neural Networks like LSTM, GRU update their weights at every timestep sequentially and hence lack parallelization over inputs in training. In case of attention based models, the attention layer has to wait for outputs from all timesteps. Hence, these models fail to take the advantage of parallelism either. Since, proposed model is based on convolution layers and gated mechanism, it can be parallelized efficiently. The convolution layers learn higher level representations for the source domain. The gated mechanism learn the domain agnostic representations. They together control the information that has to flow through further fully connected output layer after max pooling.

Let denote the input sentence represented as = {} where represents the word in and is the maximum sentence length considered. Let be the vocabulary size for each dataset and denote the word embedding matrix where each is a dimensional vector. Input sentences whose length is less than

are padded with 0s to reach maximum sentence length. Words absent in the pretrained word embeddings

111 are initialized to 0s. Therefore each input sentence is converted to dimensional vector. Convolution operation is applied on with kernel . The convolution operation is one-dimensional, applied with a fixed window size across words. We consider kernel size of 3,4 and 5. The weight initialization of these kernels is done using glorot uniform [24]

. Each kernel is a feature detector which extracts patterns from n-grams. After convolution we obtain a new feature map

= [] for each kernel .



represents the activation function in convolution layer. The gated mechanism is applied on each convolution layer. Each gated layer learns to filter domain agnostic representations for every time step



where is the activation function used in gated convolution layer. The outputs from convolution layer and gated convolution layer are element wise multiplied to compute a new feature representation

(a) GTRU
(b) GTU
(c) GLU
Figure 2: Variations in gates of the proposed GCN architecture.

Maxpooling operation is applied across each filter in this new feature representation to get the most important features [9]. As shown in Figure 1 the outputs from maxpooling layer across all filters are concatenated. The concatenated layer is fully connected to output layer. Sigmoid is used as the activation function in the output layer.

3.3 Gating mechanisms

Gating mechanisms have been effective in Recurrent Neural Networks like GRU and LSTM. They control the information flow through their recurrent cells. In case of GCN, these gated units control the domain information that flows to pooling layers. The model must be robust to change in domain knowledge and should be able to generalize well across different domains. We use the gated mechanisms Gated Tanh Unit (GTU) and Gated Linear Unit (GLU) and Gated Tanh ReLU Unit (GTRU) [23] in proposed model. The gated architectures are shown in figure 2. The outputs from Gated Tanh Unit is calculated as . In case of Gated Linear Unit, it is calculated as where and denotes Tanh and Sigmoid activation functions respectively. In case of Gated Tanh ReLU Unit, output is calculated as

4 Experiments

4.1 Datasets

4.1.1 Multi Domain Dataset(MDD)

Multi Domain Dataset [20] is a short dataset with reviews from distinct domains namely Books(B), DVD(D), Electronics(E) and Kitchen(K). Each domain consists of 2000 reviews equally divided among positive and negative sentiment. We consider 1280 reviews for training, 320 reviews for validation and 400 reviews for testing from each domain.

4.1.2 Amazon Reviews Dataset(ARD)

Amazon Reviews Dataset [25] is a large dataset with millions of reviews from different product categories. For our experiments, we consider a subset of 20000 reviews from the domains Cell Phones and Accessories(C), Clothing and Shoes(S), Home and Kitchen(H) and Tools and Home Improvement(T). Out of 20000 reviews, 10000 are positive and 10000 are negative. We use 12800 reviews for training, 3200 reviews for validation and 4000 reviews for testing from each domain.

4.2 Baselines

To evaluate the performance of proposed model, we consider various baselines like traditional lexicon approaches, CNN models without gating mechanisms and LSTM models.

4.2.1 BoW+LR

Bag-of-words (BoW) is one of the strongest baselines in text classification [5]

. We consider all the words as features with a minimum frequency of 5. These features are trained using Logistic Regression (LR).

4.2.2 Tf-Idf+lr

TF-IDF is a feature selection technique built upon Bag-of-Words. We consider all the words with a minimum frequency of 5. The features selected are trained using Logistic Regression (LR).

4.2.3 Pv+fnn

Paragraph2vec or doc2vec [26]

is a strong and popularly used baseline for text classification. Paragraph2Vec represents each sentence or paragraph in the form of a distributed representation. We trained our own doc2vec model using DBOW model. The paragraph vectors obtained are trained using Feed Forward Neural Network (FNN).

4.2.4 Cnn

To show the effectiveness of gated layer, we consider a CNN model which does not contain gated layers. Hence, we consider Static CNN model, a popular CNN architecture proposed in Kim [9] as a baseline.

4.2.5 Crnn

Wang [27] proposed a combination of Convolutional and Recurrent Neural Network for sentiment Analysis of short texts. This model takes the advantages of features learned by CNN and long-distance dependencies learned by RNN. It achieved remarkable results on benchmark datasets. We report the results using code published by the authors222

4.2.6 Lstm

We offer a comparison with LSTM model with a single hidden layer. This model is trained with equivalent experimental settings as proposed model.

4.2.7 LSTM+Attention

In this baseline, attention mechanism [28] is applied on the top of LSTM outputs across different timesteps.

Model Batchsize

Time for 1 epoch(in Sec)

CRNN 50 50
LSTM 50 70
LSTM+Attention 50 150
GLU 50 10
GRU 50 10
GTRU 50 10
Table 1: Average training time for all the models on ARD

4.3 Implementation details

All the models are experimented with approximately matching number of parameters for a solid comparison using a Tesla K80 GPU.

Input Each word in the input sentence is converted to a 300 dimensional vector using GloVe pretrained vectors [29]. A maximum sentence length 100 is considered for all the datasets. Sentences with length less than 100 are padded with 0s.

Architecture details

: The model is implemented using keras. We considered 100 convolution filters for each of the kernels of sizes 3,4 and 5. To get the same sentence length after convolution operation zero padding is done on the input.

Training Each sentence or paragraph is converted to lower case. Stopword removal is not done. A vocabulary size of 20000 is considered for all the datasets. We apply a dropout layer [30]

with a probability of 0.5, on the embedding layer and probability 0.2, on the dense layer that connects the output layer. Adadelta

[31] is used as the optimizer for training with gradient descent updates. Batch-size of 16 is taken for MDD and 50 for ARD. The model is trained for 50 epochs. We employ an early stopping mechanism based on validation loss for a patience of 10 epochs. The models are trained on source domain and tested on unseen target domain in all experiments.

BD 72.5 73.75 63.749 57.75 68.75 69.5
BE 67.5 68.5 53.25 53.5 63.249 58.75
BK 69.25 72.5 57.75 56.25 66.5 64.75
DB 66 68.5 64.75 54.25 66.75 74.75
DE 71 69.5 56.75 57.25 69.25 64.25
DK 68 69.75 60 58.25 67.5 70
EB 63.249 64 54 57.25 69.5 67.75
ED 65 66 47.25 56.499 64.5 67
EK 76.25 76.75 59.25 63.249 76 76
KB 61.5 67.75 50 57.75 69.25 66.25
KD 68 70.5 52.25 60 64.75 71
KE 81 80 50 59.25 69 76.75
Table 2: Accuracy scores on Multi Domain Dataset.
SourceTarget LSTM.Attention GLU GTU GTRU
BD 76.75 79.5 79.25 77.5
BE 70 71.75 71.25 71.25
BK 74.75 73 72.5 74.25
DB 72.5 78 80.25 77.25
DE 71 73 74.5 69.25
DK 72.75 77 76 74.75
EB 64.75 71.75 68.75 67.25
ED 62.749 71.75 69 68.25
EK 72 82.25 80.5 79
KB 64.75 70 67.75 63.249
KD 75 73.75 73.5 69.25
KE 75.5 82 82 81.25
Table 3: Accuracy scores on Multi Domain Dataset.
CS 79.3 81.175 69.625 62.324 84.95 83.7
CH 81.6 82.875 70.775 59.35 81.8 81.175
CT 76.25 77.475 66.4 54.5 79.025 77.175
SC 76.925 76.525 69.425 55.375 79.975 79.85
ST 80.125 81.575 74.524 62.7 81.45 82.925
SH 74.275 75.175 67.274 61.925 76.05 77.7
HS 76.149 73.575 65.3 53.55 79.074 78.574
HC 81.225 80.925 70.7 58.25 74.275 81.95
HT 79.175 75.449 69.425 59.4 76.325 76.725
TC 75.1 73.875 56.85 56 80.25 76.9
TS 78.875 80.5 59.199 60 85.824 81.8
TH 81.325 81.875 66.8 61.25 83.35 81
Table 4: Accuracy scores on Amazon Reviews Dataset.
SourceTarget LSTM.Attention GLU GTU GTRU
CS 84.15 85.125 84.95 84.8
CH 82.6 84.85 84.2 84.55
CT 77.9 79.5 79.274 80.225
SC 78.075 80.925 80.25 83.1
SH 82.325 83.95 83.399 84.025
ST 78.425 79.475 77.85 79.375
HC 81.375 83.175 81.85 82.1
HS 81.975 82.75 84.1 85.425
HT 80.95 82.55 81.774 81.825
TC 75.55 82.125 80.805 81.825
TS 82.375 82.625 83.975 84.775
TH 80.5 84.7 83.95 85.275
Table 5: Accuracy scores on Amazon Reviews Dataset.

5 Results and Discussion

5.1 Results

The performance of all models on MDD is shown in Tables 2 and 3 while for ARD, in Tables 4 and 5. All values are shown in accuracy percentage. Furthermore time complexity of each model is presented in Table 1.

5.2 Discussion

5.2.1 Gated outperform regular Convolution

We find that gated architectures vastly outperform non gated CNN model. The effectiveness of gated architectures rely on the idea of training a gate with sole purpose of identifying a weightage. In the task of sentiment analysis this weightage corresponds to what weights will lead to a decrement in final loss or in other words, most accurate prediction of sentiment. In doing so, the gate architecture learns which words or n-grams contribute to the sentiment the most, these words or n-grams often co-relate with domain independent words. On the other hand the gate gives less weightage to n-grams which are largely either specific to domain or function word chunks which contribute negligible to the overall sentiment. This is what makes gated architectures effective at Domain Adaptation.

In Figure 3, we have illustrated the visualization of convolution outputs(kernel size = 3) from the sigmoid gate in GLU across domains. As the kernel size is 3, each row in the output corresponds to a trigram from input sentence. This heat map visualizes values of all 100 filters and their average for every input trigram. These examples demonstrate what the convolution gate learns. Trigrams with domain independent but heavy polarity like “_  _  good” and “_  costly would” have higher weightage. Meanwhile, Trigrams with domain specific terms like “quality functional case” and “sell entire kitchen” get some of the least weights. In Figure 3(b) example, the trigram “would have to” just consists of function words, hence gets the least weight. While “sell entire kitchen” gets more weight comparatively. This might be because while function words are merely grammatical units which contribute minimal to overall sentiment, domain specific terms like “sell” may contain sentiment level knowledge only relevant within the domain. In such a case it is possible that the filters effectively propagate sentiment level knowledge from domain specific terms as well.

(a) “good cell phone”
(b) “costly would have to sell entire kitchen”
(c) “great quality functional case”
Figure 3: Visualizing outputs from gated convolutions (filter size = 3) of GLU for example sentences, darker indicates higher weightage

5.2.2 Gated outperform Attention and Linear

We see that gated architectures almost always outperform recurrent, attention and linear models BoW, TFIDF, PV. This is largely because while training and testing on same domains, these models especially recurrent and attention based may perform better. However, for Domain Adaptation, as they lack gated structure which is trained in parallel to learn importance, their performance on target domain is poor as compared to gated architectures. As gated architectures are based on convolutions, they exploit parallelization to give significant boost in time complexity as compared to other models. This is depicted in Table 1.

5.2.3 Comparison among gates

While the gated architectures outperform other baselines, within them as well we make observations. Gated Linear Unit (GLU) performs the best often over other gated architectures. In case of GTU, outputs from Sigmoid and Tanh are multiplied together, this may result in small gradients, and hence resulting in the vanishing gradient problem. However, this will not be the in the case of GLU, as the activation is linear. In case of GTRU, outputs from Tanh and ReLU are multiplied. In ReLU, because of absence of negative activations, corresponding Tanh outputs will be completely ignored, resulting in loss of some domain independent knowledge.

6 Conclusion

In this paper, we proposed Gated Convolutional Neural Network(GCN) model for Domain Adaptation in Sentiment Analysis. We show that gates in GCN, filter out domain dependant knowledge, hence performing better at an unseen target domain. Our experiments reveal that gated architectures outperform other popular recurrent and non-gated architectures. Furthermore, because these architectures rely on convolutions, they take advantage of parellalization, vastly reducing time complexity.


  • [1] Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics, 2002.
  • [2] Zellig S Harris. Distributional structure. Word, 10(2-3):146–162, 1954.
  • [3] Andrew McCallum, Kamal Nigam, et al. A comparison of event models for naive bayes text classification. In AAAI-98 workshop on learning for text categorization, volume 752, pages 41–48. Citeseer, 1998.
  • [4] Thorsten Joachims. Text categorization with support vector machines: Learning with many relevant features. In European conference on machine learning, pages 137–142. Springer, 1998.
  • [5] Sida Wang and Christopher D Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pages 90–94. Association for Computational Linguistics, 2012.
  • [6] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003.
  • [7] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013.
  • [8] Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 129–136, 2011.
  • [9] Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014.
  • [10] Yoshua Bengio, Patrice Simard, Paolo Frasconi, et al. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157–166, 1994.
  • [11] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310–1318, 2013.
  • [12] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  • [13] John Blitzer, Ryan McDonald, and Fernando Pereira. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 120–128. Association for Computational Linguistics, 2006.
  • [14] Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen. Cross-domain sentiment classification via spectral feature alignment. In Proceedings of the 19th international conference on World wide web, pages 751–760. ACM, 2010.
  • [15] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.

    Domain adaptation for large-scale sentiment classification: A deep learning approach.

    In Proceedings of the 28th international conference on machine learning (ICML-11), pages 513–520, 2011.
  • [16] Zheng Li, Yun Zhang, Ying Wei, Yuxiang Wu, and Qiang Yang. End-to-end adversarial memory network for cross-domain sentiment classification. In IJCAI, pages 2237–2243, 2017.
  • [17] Qi Liu, Yue Zhang, and Jiangming Liu. Learning domain representation for multi-domain sentiment classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 541–550, 2018.
  • [18] Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang. Hierarchical attention transfer network for cross-domain sentiment classification. In

    Thirty-Second AAAI Conference on Artificial Intelligence

    , 2018.
  • [19] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 933–941. JMLR. org, 2017.
  • [20] John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 440–447, 2007.
  • [21] Jianfei Yu and Jing Jiang. Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 236–246, 2016.
  • [22] Peixin Chen, Wu Guo, Zhi Chen, Jian Sun, and Lanhua You. Gated convolutional neural network for sentence matching. In Interspeech, 2018.
  • [23] Wei Xue and Tao Li. Aspect based sentiment analysis with gated convolutional networks. arXiv preprint arXiv:1805.07043, 2018.
  • [24] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256, 2010.
  • [25] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507–517. International World Wide Web Conferences Steering Committee, 2016.
  • [26] Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In International conference on machine learning, pages 1188–1196, 2014.
  • [27] Xingyou Wang, Weijie Jiang, and Zhiyong Luo. Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2428–2437, 2016.
  • [28] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  • [29] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  • [30] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
  • [31] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.