1 Introduction
Deep networks have achieved very good performance in a variety of tasks, e.g. [10, 5, 3]. However, a fundamental limitation of these architectures is that the entire network must be executed for all inputs. This computational burden imposes limits network size. One way to scale these networks up while keeping the computational cost low is to increase the overall number of parameters and hidden units, but use only a small portion of the network for each given input. Then, learn a computationally cheap mapping function from input to the appropriate portions of the network.
The Mixture of Experts model [7] is a continuous version of this: A learned gating network mixes the outputs of “expert” networks to produce a final output. While this model does not itself achieve the computational benefits outlined above, it shows promise as a stepping stone towards networks that can realize this goal.
In this work, we extend the Mixture of Experts to use a different gating network at each layer in a multilayer network, forming a Deep Mixture of Experts (DMoE). This increases the number of effective experts by introducing an exponential number of paths through different combinations of experts at each layer. By associating each input with one such combination, our model uses different subsets of its units for different inputs. Thus it can be both large and efficient at the same time.
We demonstrate the effectiveness of this approach by evaluating it on two datasets. Using a jittered MNIST dataset, we show that the DMoE learns to factor different aspects of the data representation at each layer (specifically, location and class), making effective use of all paths. We also find that all combinations are used when applying our model to a dataset of speech monophones.
2 Related Work
A standard Mixture of Experts (MoE) [7] learns a set of expert networks along with a gating network . Each maps the input to outputs (one for each class ), while is a distribution over experts that sums to 1. The final output is then given by Eqn. 1
(1)  
(2) 
This can also be seen as a probability model, where the final probability over classes is marginalized over the selection of expert: setting
and , we have Eqn. 2.A product of experts (PoE) [6] is similar, but instead combines log probabilities to form a product:
(3) 
Also closely related to our work is the Hierarchical Mixture of Experts [9], which learns a hierarchy of gating networks in a tree structure. Each expert network’s output corresponds to a leaf in the tree; the outputs are then mixed according to the gating weights at each node.
Our model differs from each of these three models because it dynamically assembles a suitable expert combination for each input. This is an instance of the concept of conditional computation put forward by Bengio [1] and examined in a singlelayer stochastic setting by Bengio, Leonard and Courville [2]. By conditioning our gating and expert networks on the output of the previous layer, our model can express an exponentially large number of effective experts.
3 Approach
(a)  (b) 
To extend MoE to a DMoE, we introduce two sets of experts with gating networks and , along with a final linear layer (see Fig. 1). The final output is produced by composing the mixtures at each layer:
We set each to a single linear map with rectification, and each to two layers of linear maps with rectification (but with few hidden units); is a single linear layer. See Section 4 for details.
We train the network using stochastic gradient descent (SGD) with an additional constraint on gating assignments (described below). SGD by itself results in a degenerate local minimum: The experts at each layer that perform best for the first few examples end up overpowering the remaining experts. This happens because the first examples increase the gating weights of these experts, which in turn causes them to be selected with high gating weights more frequently. This causes them to train more, and their gating weights to increase again,
ad infinitum.To combat this, we place a constraint on the relative gating assignments to each expert during training. Let be the running total assignment to expert of layer at step , and let be their mean (here, is the training example at step ). Then for each expert , we set if for a margin threshold , and renormalize the distribution to sum to 1 over experts . This prevents experts from being overused initially, resulting in balanced assignments. After training with the constraint in place, we lift it and further train in a second finetuning phase.
4 Experiments
4.1 Jittered MNIST
We trained and tested our model on MNIST with random uniform translations of pixels, resulting in grayscale images of size
. As explained above, the model was trained to classify digits into ten classes.
For this task, we set all and to onelayer linear models with rectification, , and similarly for . We set to a linear layer, . We varied the number of output hidden units of and between 20 and 100. The final output from has 10 units (one for each class).
The gating networks and are each composed of two linear+rectification layers with either 50 or 20 hidden units, and 4 output units (one for each expert), i.e. , and similarly for .
We evaluate the effect of using a mixture at the second layer by comparing against using only a single fixed expert at the second layer, or concatenating the output of all experts. Note that for a mixture with hidden units, the corresponding concatenated model has hidden units. Thus we expect the concatenated model to perform better than the mixture, and the mixture to perform better than the single network. It is best for the mixture to be as close as possible to the concatenatedexperts bound. In each case, we keep the first layer architecture the same (a mixture).
We also compare the twolayer model against a onelayer model in which the hidden layer is mapped to the final output through linear layer and softmax. Finally, we compare against a fullyconnected deep network with the same total number of parameters. This was constructed using the same number of secondlayer units , but expanding the number first layer units such that the total number of parameters is the same as the DMoE (including its gating network parameters).
4.2 Monophone Speech
In addition, we ran our model on a dataset of monophone speech samples. This dataset is a random subset of approximately one million samples from a larger proprietary database of several hundred hours of US English data collected using Voice Search, Voice Typing and read data [8]
. For our experiments, each sample was limited to 11 frames spaced 10ms apart, and had 40 frequency bins. Each input was fed to the network as a 440dimensional vector. There were 40 possible output phoneme classes.
We trained a model with 4 experts at the first layer and 16 at the second layer. Both layers had 128 hidden units. The gating networks were each two layers, with 64 units in the hidden layer. As before, we evaluate the effect of using a mixture at the second layer by comparing against using only a single expert at the second layer, or concatenating the output of all experts.
5 Results
5.1 Jittered MNIST
Table 1 shows the error on the training and test sets for each model size (the test set is the MNIST test set with a single random translation per image). In most cases, the deeply stacked experts performs between the single and concatenated experts baselines on the training set, as expected. However, the deep models often suffer from overfitting: the mixture’s error on the test set is worse than that of the single expert for two of the four model sizes. Encouragingly, the DMoE performs almost as well as a fullyconnected network (DNN) with the same number of parameters, even though this network imposes fewer constraints on its structure.
In Fig. 2, we show the mean assignment to each expert (i.e. the mean gating output), both by input translation and by class. The first layer assigns experts according to translation, while assignment is uniform by class. Conversely, the second layer assigns experts by class, but is uniform according to translation. This shows that the two layers of experts are indeed being used in complementary ways, so that all combinations of experts are effective. The first layer experts become selective to where the digit appears, regardless of its membership class, while the second layer experts are selective to what the digit class is, irrespective of the digit’s location.
Finally, Fig. 3 shows the nine test examples with highest gating value for each expert combination. Firstlayer assignments run over the rows, while the secondlayer runs over columns. Note the translation of each digit varies by rows but is constant over columns, while the opposite is true for the class of the digit. Furthermore, easily confused classes tend to be grouped together, e.g. 3 and 5.
5.2 Monophone Speech
Table 2 shows the error on the training and test sets. As was the case for MNIST, the mixture’s error on the training set falls between the two baselines. In this case, however, test set performance is about the same for both baselines as well as the mixture.
Fig. 4 shows the 16 test examples with highest gating value for each expert combination (we show only 4 experts at the second layer due to space considerations). As before, firstlayer assignments run over the rows, while the secondlayer runs over columns. While not as interpretable as for MNIST, each expert combination appears to handle a distinct portion of the input. This is further bolstered by Fig. 5, where we plot the average number of assignments to each expert combination. Here, the choice of secondlayer expert depends little on the choice of firstlayer expert.
6 Conclusion
The Deep Mixture of Experts model we examine is a promising step towards developing large, sparse models that compute only a subset of themselves for any given input. We see precisely the gating assignments required to make effective use of all expert combinations: for jittered MNIST, a factorization into translation and class, and distinctive use of each combination for monophone speech data. However, we still use a continuous mixture of the experts’ outputs rather than restricting to the top few — such an extension is necessary to fulfill our goal of using only a small part of the model for each input. A method that accomplishes this for a single layer has been described by Collobert et al. [4], which could possibly be adapted to our multilayer case; we hope to address this in future work.
Acknowledgements
The authors would like to thank Matthiew Zeiler for his contributions on enforcing balancing constraints during training.
References
 [1] Y. Bengio. Deep learning of representations: Looking forward. CoRR, abs/1305.0445, 2013.
 [2] Y. Bengio, N. Léonard, and A. C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013.

[3]
D. C. Cireşan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber.
Flexible, high performance convolutional neural networks for image classification.
In IJCAI, 2011. 
[4]
R. Collobert, Y. Bengio, and S. Bengio.
Scaling large learning problems with hard parallel mixtures.
International Journal on Pattern Recognition and Artificial Intelligence (IJPRAI)
, 17(3):349–365, 2003. 
[5]
A. Graves, A. Mohamed, and G. Hinton.
Speech recognition with deep recurrent neural networks.
In ICASSP, 2013.  [6] G. E. Hinton. Products of experts. ICANN, 1:1–6, 1999.
 [7] R. A. Jacobs, M. I. Jordan, S. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3:1–12, 1991.
 [8] N. Jaitly, P. Nguyen, A. Senior, and V. Vanhoucke. Application of pretrained deep neural networks to large vocabulary speech recognition. Interspeech, 2012.
 [9] M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6:181–214, 1994.
 [10] A. Krizhevsky, I. Sutskever, and G.E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.