Big Neural Networks Waste Capacity

by   Yann N. Dauphin, et al.

This article exposes the failure of some big neural networks to leverage added capacity to reduce underfitting. Past research suggest diminishing returns when increasing the size of neural networks. Our experiments on ImageNet LSVRC-2010 show that this may be due to the fact there are highly diminishing returns for capacity in terms of training error, leading to underfitting. This suggests that the optimization method - first order gradient descent - fails at this regime. Directly attacking this problem, either through the optimization method or the choices of parametrization, may allow to improve the generalization error on large datasets, for which a large capacity is required.



There are no comments yet.


page 1

page 2

page 3

page 4


The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization

Modern deep learning models employ considerably more parameters than req...

Stochastic Markov Gradient Descent and Training Low-Bit Neural Networks

The massive size of modern neural networks has motivated substantial rec...

Beyond Lazy Training for Over-parameterized Tensor Decomposition

Over-parametrization is an important technique in training neural networ...

Qualitatively characterizing neural network optimization problems

Training neural networks involves solving large-scale non-convex optimiz...

Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems

Deep learning recommendation systems at scale have provided remarkable g...

Dynamic Capacity Networks

We introduce the Dynamic Capacity Network (DCN), a neural network that c...

Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization

Training sparse networks to converge to the same performance as dense ne...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Deep learning and neural networks have achieved state-of-the-art results on vision 111Krizhevsky et al. (2012) reduced by almost one half the error rate on the 1000-class ImageNet object recognition benchmark, language 222Mikolov et al. (2011) reduced perplexity on WSJ by 40% and speech recognition absolute word error rate by %. , and audio-processing tasks 333For speech recognition, Seide et al. (2011) report relative word error rates decreasing by about 30% on datasets of 309 hours. All these cases involved fairly large datasets, but in all these cases, even larger ones could be used. One of the major challenges remains to extend neural networks on a much larger scale, and with this objective in mind, this paper asks a simple question: is there an optimization issue that prevents efficiently training larger networks?

Prior evidence of the failure of big networks in the litterature can be found for example in Coates et al. (2011), which shows that increasing the capacity of certain neural net methods quickly reaches a point of diminishing returns on the test error. These results have since been extended to other types of auto-encoders and RBMs (Rifai et al., 2011; Courville et al., 2011). Furthermore, Coates et al. (2011)

shows that while neural net methods fail to leverage added capacity K-Means can. This has allowed K-Means to reach state-of-the-art performance on CIFAR-10 for methods that do not use artificial transformations. This is an unexpected result because K-Means is a much

dumberunsupervised learning algorithm when compared with RBMs and regularized auto-encoders. Coates et al. (2011) argues that this is mainly due to K-Means making better use of added capacity, but it does not explain why the neural net methods failed to do this.

2 Experimental Setup

We will perform experiments with the well known ImageNet LSVRC-2010 object detection dataset444 The subset used in the Large Scale Visual Recognition Challenge 2010 contains 1000 object categories and 1.2 million training images.

This dataset has many attractive features:

  1. The task is difficult enough for current algorithms that there is still room for much improvement. For instance, Krizhevsky et al. (2012) was able to reduce the error by half recently. What’s more the state-of-the-art is at 15.3% error. Assuming minimal error in the human labelling of the dataset, it should be possible to reach errors close to 0%.

  2. Improvements on ImageNet are thought to be a good proxy for progress in object recognition (Deng et al., 2009).

  3. It has a large number of examples. This is the setting that is commonly found in industry where datasets reach billions of examples. Interestingly, as you increase the amount of data, the training error converges to the generalization error. In other words, reducing training error is well correlated with reducing generalization error, when large datasets are available. Therefore, it stands to reason that resolving underfitting problems may yield significant improvements.

We use the features provided by the Large Scale Visual Recognition Challenge 2010555 The images are convolved with SIFT features, then K-Means is used to form a visual vocabulary of 1000 visual words. Following the litterature, we report the Top-5 error rate only.

The experiments focus on the behavior of Multi-Layer Perceptrons (MLP) as capacity is increased. This is done by increasing the number of hidden units in the network. The final classification layer of the network is a softmax over possible classes (

). The hidden layers use the logistic sigmoid activation function (

). We initialize the weights of the hidden layer according to the formula proposed by Glorot and Bengio (2010). The parameters of the classification layer are initialized to 0, along with all the bias (offset) parameters of the MLP.

The hyper-parameters to tune are the learning rate and the number of hidden units. We are interested in optimization performance so we cross-validate them based on the training error. We use a grid search with the learning rates taken from and the number of hiddens from

. When we report the performance of a network with a given number of units we choose the best learning rate. The learning rate is decreased by 5% everytime the training error goes up aftern an epoch. We do not use any regularization because it would typically not help to decrease the training set error. The number of epochs is set to 300 s that it is large enough for the networks to converge.

The experiments are run on a cluster of Nvidia Geforce GTX 580 GPUs with the help of the Theano library

(Bergstra et al., 2010). We make use of HDF5 (Folk et al., 2011) to load the dataset in a lazy fashion because of its large size. The shortest training experiment took 10 hours to run and the longest took 28 hours.

3 Experimental Results

Figure 1: Training error with respect to the capacity of a 1-layer sigmoidal neural network. This curve seems to suggest we are correctly leveraging added capacity.

Figure 1 shows the evolution of the training error as the capacity is increased. The common intuition is that this increased capacity will help fit the training set - possibly to the detriment of generalization error. For this reason practitioners have focused mainly on the problem of overfitting the dataset when dealing with large networks - not underfitting. In fact, much research is concerned with proper regularization of such large networks (Hinton et al., 2012, 2006).

However, Figure 2 reveals a problem in the training of big networks. This figure is the derivative of the curve in Figure 1 (using the number of errors instead of the percentage). It may be interpreted as the return on investment (ROI) for the addition of capacity. The Figure shows that the return on investment of additional hidden units decreases fast, where in fact we would like it to be close to constant. Increasing the capacity from 1000 to 2000 units, the ROI decreases by an order of magnitude. It is harder and harder for the model to make use of additional capacity. The red line is a baseline where the additional unit is used as a template matcher for one of the training errors. In this case, the number of errors reduced per unit is at least 1. We see that the MLP does not manage to beat this baseline after 5000 units, for the given number of training iterations.

Figure 2: Return on investment on the addition of hidden units for a 1-hidden layer sigmoidal neural network. The vertical axis is the number of training errors removed per additional hidden unit, after 300 epochs. We see here that it is harder and harder to use added capacity.

For reference, we also include the learning curves of the networks used for Figure 1 and 2 in Figure 3. We see that the curves for capacities above 5000 all converge towards the same point.

Figure 3: Training error with respect to the number of epochs of gradient descent. Each line is a 1-hidden layer sigmoidal neural network with a different number of hidden units.

4 Future directions

This rapidly decreasing return on investment for capacity in big networks seems to be a failure of first order gradient descent.

In fact, we know that the first order approximation fails when there are a lot of interactions between hidden units. It may be that adding units increases the interactions between units and causes the Hessian to be ill-conditioned. This reasoning suggests two research directions:

  • methods that break interactions between large numbers of units. This helps the Hessian to be better conditioned and will lead to better effectiveness for first-order descent. This type of method can be implemented efficiently. Examples of this approach are sparsity and orthognality penalties.

  • methods that model interactions between hidden units. For example, second order methods (Martens, 2010) and natural gradient methods (Le Roux et al., 2008). Typically, these are expensive approaches and the challenge is in scaling them to large datasets, where stochastic gradient approaches may dominate. The ideal target is a stochastic natural gradient or stochastic second-order method.

The optimization failure may also be due to other reasons. For example, networks with more capacity have more local minima. Future work should investigate tests that help discriminate between ill-conditioning issues and local minima issues.

Fixing this optimization problem may be the key to unlocking better performance of deep networks. Based on past observations (Bengio, 2009; Erhan et al., 2010), we expect this optimization problem to worsen for deeper networks, and our experimental setup should be extended to measure the effect of depth. As we have noted earlier, improvements on the training set error should be well correlated with generalization for large datasets.


  • Bengio (2009) Bengio, Y. (2009). Learning deep architectures for AI. Now Publishers.
  • Bergstra et al. (2010) Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation.
  • Coates et al. (2011) Coates, A., Lee, H., and Ng, A. Y. (2011). An analysis of single-layer networks in unsupervised feature learning. In

    Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2011)

  • Courville et al. (2011) Courville, A., Bergstra, J., and Bengio, Y. (2011). Unsupervised models of images by spike-and-slab RBMs. In

    Proceedings of the Twenty-eight International Conference on Machine Learning (ICML’11)

  • Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.
  • Erhan et al. (2010) Erhan, D., Bengio, Y., Courville, A., Manzagol, P., Vincent, P., and Bengio, S. ((11) 2010). Why does unsupervised pre-training help deep learning? J. Machine Learning Res.
  • Folk et al. (2011) Folk, M., Heber, G., Koziol, Q., Pourmal, E., and Robinson, D. (2011). An overview of the hdf5 technology suite and its applications. In Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases, pages 36–47. ACM.
  • Glorot and Bengio (2010) Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In JMLR W&CP: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010), volume 9, pages 249–256.
  • Hinton et al. (2006) Hinton, G. E., Osindero, S., and Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554.
  • Hinton et al. (2012) Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. Technical report, arXiv:1207.0580.
  • Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. (2012).

    ImageNet classification with deep convolutional neural networks.

    In Advances in Neural Information Processing Systems 25 (NIPS’2012).
  • Le Roux et al. (2008) Le Roux, N., Manzagol, P.-A., and Bengio, Y. (2008). Topmoumoute online natural gradient algorithm. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20 (NIPS’07), Cambridge, MA. MIT Press.
  • Martens (2010) Martens, J. (2010). Deep learning via Hessian-free optimization. In L. Bottou and M. Littman, editors, Proceedings of the Twenty-seventh International Conference on Machine Learning (ICML-10), pages 735–742. ACM.
  • Mikolov et al. (2011) Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and Cernocky, J. (2011). Empirical evaluation and combination of advanced language modeling techniques. In Proc. 12th annual conference of the international speech communication association (INTERSPEECH 2011).
  • Rifai et al. (2011) Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., and Glorot, X. (2011). Higher order contractive auto-encoder. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD).
  • Seide et al. (2011) Seide, F., Li, G., and Yu, D. (2011). Conversational speech transcription using context-dependent deep neural networks. In Interspeech 2011, pages 437–440.