In the last few years artificial neural network (ANN) models received huge interest from the research community. In particular, their potential capability of solving complex tasks with extremely simple training strategies has been the initial spark, while convolutional neural networks (CNNs), capable of self-extracting relevant features from images, have been the fuel for the burning flame which is the research around ANNs. Furthermore, thanks to the ever-increasing computational capability of machines with the introduction of GPUs (and, recently, TPUs) in the simulation of neural networks, ANNs might be embedded in many portable devices and, potentially, used in everyday life.
State-of-the-art ANNs are able to learn very complex classification tasks: from the nowadays outdated MNIST 
, moving to CIFAR-10 and then even the ImageNet classification problem. In order to overcome the complexity of these learning tasks, extremely complex architectures have been proposed: some examples are VGG and ResNet . However, due to their extremely high number of parameters, these models are prone to over-fitting the data; hence, they are not able to generalize as they should. In this case, the simple learning strategies (like SGD) are no longer able to guarantee the network to learn the relevant features from the training set and other strategies need to be adopted.
In order to improve the generalization of ANNs, several approaches have been proposed. One of the most typical relies on the introduction of a “regularization” term, whose aim is to add an extra constraint to the overall objective function to be minimized. Recently, other approaches have been proposed: from the introduction of different optimizers  to data augmentation techniques, the proposal of new techniques like dropout  and even changing the basic architecture of the ANN .
In this work, we propose a regularization term inspired by a side effect of the regularization (also known ad weight decay) on the parameters. In particular, we are going to show that, naturally, weight decay makes the post-synaptic potential dropping to zero in ANN models. From this observation, a post-synaptic potential regularization (PSP) is here proposed. Differently from regularization, its effect on the parameters is not local: parameters belonging to layers closer to the input feel the effect of the regularization on the forward layers. Hence, this regularization is aware of the configuration of the whole network and tunes the parameters using a global information. We show that the standard regularization is a special case of the proposed regularizer as well. Empirically we show that, when compared to the standard weight decay regularization, PSP generalizes better.
The rest of this paper is organized as follows. In Sec. 2 we review some of the most relevant regularization techniques aiming at improving generalization. Next, in Sec. 3 we introduce our proposed regularization, starting from some simple considerations on the effect of the regularization on the post synaptic potential and analyzing the potential effects on the learning dynamics. Then, in Sec. 4 we show some empirical results and some extra insights of the proposed regularization. Finally, in Sec. 5 the conclusions are drawn.
2 Related work
Regularization is one of the key features a learning algorithm should particularly take care of in deep learning, in order to prevent data over-fit and boosting the generalization. Even though such a concept is more general and older than the first ANN models 
, we are going to focus on what regularization for deep architectures (trained on finite datasets via supervised learning) is. We can divide the regularization strategies in our context under four main categories:
regularization via network architecture: in this case, the architecture is properly selected in order to fit the particular dataset we are aiming to train. It can involve the choice of single layers (pooling, convolutional , dropout ), it can involve the insertion of entire blocks (residual blocks ), the entire structure can be designed on-purpose    or even pruned  .
regularization via optimization: an optimizer can determine the nature of the local minima and avoid “bad” local minima (if any ), boosting the generalization  . The initialization also seemed to cover an important role , together with cross-validation based techniques like early-stopping .
regularization via regularization term: here a regularization term is added to the loss, and a global objective function (sum of loss and regularization term (4) ) is minimized. This is the scope of our work.
One of the ground-breaking regularization techniques, proposed few years ago, is dropout. Srivava et al. 
proposed, during the training process, to stochastically set a part of the activations in an ANN to zero according to an a-priori set dropout probability. Empirically it was observed that, applying dropout on fully-connected architecture, was significantly improving the generalization, while its effectiveness was less evident in convolutional architectures. Such a technique, however, typically requires a longer training time, and sometimes a proper choice of the dropout probability may change the effectiveness of the technique. However, dropout has many variants aiming to the same goal: one of the most used is dropconnect by Wanet al. .
A completely different approach to boost the generalization is to focus the attention on some regions of the loss functions. It has been suggested by Linet al.  that “sharp” minima of the loss function does not generalize as well as “wide” minima. According to this, the design of an optimizer which does not remain stuck in sharp minima helps in the generalization. Towards this end, some optimizers like SGD or Adam  are already implicitly looking for these kind of solutions. Recently, a specially-purposed optimizer, Entropy-SGD, designed by Chaudhari et al. , showed good generalization results. However, more sophisticated optimizers increase the computational complexity significantly.
A regularization technique, proposed about 30 years ago by Weigend et al. and just recently re-discovered, is weight elimination 
: a penalty term is added to the loss function and the total objective function is minimized. The aim of the regularization term is here to estimate the “complexity” of the model, which is minimized together with the loss function. The learning complexity for an object increases with its number of parameters: there should exist an optimal number of parameters (or, in other words, configuration) for any given classification problem. Supporting this view, while using theirsensitivity-driven regularization  aiming to sparsify deep models, Tartaglione et al. observed an improvement of the generalization for low compression ratios.
Any of the proposed regularization techniques, however, is typically used jointly to the regularization. Such a technique is broadly used during most of the ANN trainings and, despite its simple formulation, under a wide range of different scenarios, it improves the generalization. Furthermore, many recent works suggest that there is a correspondence between regularization and other techniques: for example, Wager et al.  showed an equivalence between dropout and weight decay. Is there something else to understand about regularization? What’s under the hood? This will be our starting point, to be discussed more in details in Sec. 3.2.
3 Post-synaptic potential regularization
In this section we first analyze the effect of weight decay on the output of any neuron in an ANN model. We show thatregularization makes the post-synaptic potential drop to zero. Hence, a regularization over the post-synaptic potential is formulated (PSP). Next, the parameters update term is derived and some considerations for multi-layer architectures are drawn. Finally, we show the concrete effect of post-synaptic potential regularization on both the output of a single neuron and its parameters.
In this section we introduce the notation to be used in the rest of this work. Let us assume we work with an acyclic, multi-layer artificial neural network composed of layers, where layer is the input layer and the output layer. The ensemble of all the trained parameters in the ANN will be indicated as . Each of the layers is made of neurons (or filters for convolutional layers). Hence, the -th neuron () in the -th layer has:
as its own output.
as input vector.
as its own parameters: are the weights (from which we identify the -th as ) and is the bias.
Each of the neurons has its own activation function to be applied after some affine function which can be convolution, dot product, adding residual blocks, batch normalization or any combination of them. Hence, the output of a neuron can be expressed by
We can simplify (1) if we define the post-synaptic potential (or equivalently, the pre-activation potential) as
As we are going to see, the post-synaptic potential will be central in our method and analysis and encloses the true essence of the proposed regularization strategy. A summary of the introduced notation is graphically represented in Fig. 1.
3.2 Effect of weight decay on the post-synaptic potential
Most of the learning strategies use the well-known regularization term
Eq. (3) is minimized together with the loss function ; hence, the overall minimized function is
where is the desired output and are positive real numbers, and commonly in range . All the update contributions are computed using the standard back-propagation strategy. Let us focus here, for sake of simplicity, on the regularization term (3). Minimizing it corresponds to adopt the commonly named weight decay strategy for which we have the following update rule:
This generates a perturbation of the output for the corresponding neuron resulting in a perturbation of the post-synaptic potential:
How does the regularization affect ?
Clearly, minimizing (3) means that . Now, if we have an input pattern for our network , it is straightforward, according to (2) and (5), that , as all the parameters will be zero.
Under this assumption, we can say that the weight decay strategy implicitly aims to focus on peculiar regions of the mostly-used activation functions: in the case we use sigmoid or hyperbolic tangent, we have the maximum value for the derivative for
; while for the ReLU activation we are close to the function discontinuity. Is this the real essence of weight decay and one of the reasons it helps in the generalization?
Starting from these very simple observation, we are now going to formulate a regularization term which explicitly minimizes .
3.3 Post-synaptic regularization
In the previous section we have observed that, in the typical deep learning scenario, weight decay minimizes the post-synaptic potential, focusing the output of the neuron around some particular regions, which might help in the signal back-propagation and, indirectly, favor the generalization.
If we wish to explicitly drive the output of the neuron, or better, its post-synaptic potential, we can impose an regularization on :
where is an index ranging for all the neurons in the -th layer. We can split (7) for each of the neurons in the -th layer:
In case we desire to apply the regularization (8
), we can use the chain rule to check what is the update felt by the parameters of our model:
Expanding (9), we have
Here we need to differentiate between bias and weight cases: if is the bias then (10) can be easily written as
while, if is one of the weights,
From (12) it is possible to recover the usual weight decay assuming and completely neglecting the contribution coming from the other parameters for the same neuron.
The variation in the parameter value, according to (9), is
where as usual. As we are minimizing (7), we can say that is a bounded term. Furthermore, looking at , we need to distinguish two cases:
: in this case, represents the input, which we impose to be a bounded quantity.
: here we should recall that is the output of the -th layer: if we minimize the post-synaptic potentials also in those layers, for the commonly-used activation functions, we guarantee it to be a bounded quantity.
Hence, as product of bounded quantities, also (13) is a limited quantity.
However, what we aim to minimize is not (8), but the whole summation in (7). If we explicitly wish to write what the regularization contribution to the parameter is, we have
Here, three different cases can be analyzed:
: in this case, the gradient term is and the entire contribution is zero.
: here, the gradient term if , zero otherwise.
: this is the most interesting case: regularization on the last layers affects all the previous ones, and such a contribution is automatically computed using back-propagation.
Hence, in the most general case, the total update contribution resulting from the minimization of (7) on the -th weight belonging to the -th neuron at layer is indeed
In this section we have proposed a post-synaptic potential regularization which explicitly minimizes in all the neurons of the ANN. In particular, we have observed that the update term for the single parameter employs a global information coming from forward layers, favoring the regularization. In the next section, results from some simulations in which PSP regularization is tested are shown.
In this section we show the performance reached by some of the mostly-used ANNs with our post-synaptic potential regularization (PSP) and we compare it to the results obtained with weight decay. We have tested our regularization on three different datasets: MNIST, Fashion-MNIST and CIFAR-10 on LeNet5, ResNet-18 , MobileNet v2  and All-CNN-C 
. All the simulationsare performed using the standard SGD with CUDA 8 on a Nvidia Tesla P-100 GPU. Our regularization has been implemented using PyTorch 1.1.111All the source code is publicly available at https://github.com/enzotarta/PSP
4.1 Simulations on MNIST
As very first experiments, we attempted to train the well-known LeNet-5 model over the standard MNIST dataset  (60k training images and 10k test images, all the images are 28x28 pixels, grey-scale). We use SGD with a learning parameter , minibatch size 100. In Fig. 1(a), we show a typically observed scenario in our experimental setting, where we compare standard SGD with no regularization, the effect of regularization () and our pre-activation signal potential regularization (PSP, ). While the weight decay average performance is 0.64% (showing improvements from the standard SGD, which is about 0.71%), PSP final performance is about 0.50%, with peaks reaching 0.46%. We would like to emphasize that, to the best of our knowledge, we hit the best ever recorded performance for the current dataset with the same architecture .
We find interesting the behavior of for all the three techniques (Fig. 1(b)). In the case of standard SGD, the averaged value, as it is not controlled, typically grows until the gradient on the loss will not be zero. For regularization, interestingly, it slowly grows until it reaches a final plateau. Finally, in PSP regularization, the value is extremely low, and still slowly decreases. According to the results in Fig. 1(a), this is helping in the generalization.
At this point we can have a further look at what is happening at the level of the distribution of the parameters layer-by-layer. A typical trained parameters distribution for LeNet5, trained on MNIST, is shown in Fig. 3. While regularization typically shrinks the parameters around zero, PSP regularization does not constrain the parameters with the same strength, while still constrains the pre-activation signal (Fig. 1(b)). However, contrarily to this, the first convolutional layer, with regularization, is less constrained around zero than with PSP (Fig. 2(a)). Such a behavior can be explained by (15): all the regularization contributions coming from all the forward layers (in this case, conv2, fc1 and fc2) affect the parameters in conv1, which are directly conditioning all the computed in forward layers.
4.2 Simulations on Fashion-MNIST
We have decided, as a further step, to test LeNet-5 on a more complex dataset: hence, we have chosen the Fashion-MNIST dataset . It is made of 10 classes of 28x28 grey-scale images representing various pieces of clothing. They are divided in 60k examples for the training set and 10k for the test set. Such a dataset has two main advantages: the problem dimensionality (input, output) is the same as MNIST; hence, the same ANN can be used for both problems, and it is not as trivial as MNIST to solve.
|Technique||Test set error[%]|
Training results are shown in Table 1. The simulations are performed with , batch size and for regularization while for PSP. Here, the difference in the generalization between
and PSP is wider than the one presented for MNIST: we are able, with the same architecture, to improve the performance by around the 1% without any other heuristics.
4.3 Simulations on CIFAR-10
Moving towards deep architectures, we decided to use CIFAR-10 as dataset. It is made of 32x32 color images (3 channels) divided in 10 classes. The training set is made of 50k images and the test set of 10k samples. This dataset is a good compromise to make the first tests on deep architectures as the training is performed from scratch.
Three convolutional architectures have been here tested: MobileNet v2 , ResNet-18  and All-CNN-C . In order to separate the contribution of our regularizer towards other state-of-the-art regularizers, we are going to compare our results with our baseline (same data augmentation, no dropout). All these networks were pre-trained with
for 150 epochs and then learning rate decay policy was applied (drop to 10% every 100 epochs) for 300 epochs. Minibatch size was set toand momentum to . For standard training, the was set to while for PSP regularization to .
As we can observe in Table 2, using PSP-based regularization shows improvements from the baseline and from regularization. We speculate that with a proper setting of , PSP can potentially match, or even overcome, top performance marked by state-of-the-art regularizers.
|Architecture||Baseline test error[%]||PSP test error [%]|
In this work we have proposed a post-synaptic potential regularizer for supervised learning problems. Starting from the observation that weight decay indirectly shrinks the post-synaptic potential to zero, we have formulated the new PSP regularization. Contrarily to weight decay, it uses a global information coming from other parameters affecting the post-synaptic potential. We have also shown that regularization is a special case of our PSP regularization. Looking at the computational complexity, if the autograd  package is used for back-propagation, no significant computational overhead is added.
Empirical results show that PSP regularization improves the generalization on both simple and more complex problems, boosting the performance also on deep architectures. Future work includes the application of PSP to recurrent neural networks, tests on networks using non-linear activation functions and the definition of a proper decay policy for PSP regularization.
-  Calimeri, F., Marzullo, A., Stamile, C., Terracina, G.: Biomedical data augmentation using generative adversarial neural networks. In: International conference on artificial neural networks. pp. 626–634. Springer (2017)
Caruana, R.: Multitask learning. Machine learning28(1), 41–75 (1997)
-  Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., Chayes, J., Sagun, L., Zecchina, R.: Entropy-sgd: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838 (2016)
-  Cui, X., Goel, V., Kingsbury, B.: Data augmentation for deep neural network acoustic modeling. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP) 23(9), 1469–1477 (2015)
-  Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: A survey. arXiv preprint arXiv:1808.05377 (2018)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics. pp. 249–256 (2010)
-  Goldberg, P.W., Williams, C.K., Bishop, C.M.: Regression with input-dependent noise: A gaussian process treatment. In: Advances in neural information processing systems. pp. 493–499 (1998)
-  Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. MIT press (2016)
-  Gulcehre, C., Moczulski, M., Visin, F., Bengio, Y.: Mollifying networks. arXiv preprint arXiv:1608.04980 (2016)
-  Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
-  Kawaguchi, K.: Deep learning without poor local minima. In: Advances in neural information processing systems. pp. 586–594 (2016)
-  Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
-  Kukacka, J., Golkov, V., Cremers, D.: Regularization for deep learning: A taxonomy. CoRR abs/1710.10686 (2017), http://arxiv.org/abs/1710.10686
-  LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)
-  Lin, H.W., Tegmark, M., Rolnick, D.: Why does deep and cheap learning work so well? Journal of Statistical Physics 168(6), 1223–1247 (2017)
-  Maclaurin, D., Duvenaud, D., Adams, R.P.: Autograd: Effortless gradients in numpy. In: ICML 2015 AutoML Workshop (2015)
-  Prechelt, L.: Automatic early stopping using cross validation: quantifying the criteria. Neural Networks 11(4), 761–767 (1998)
-  Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4510–4520 (2018)
-  Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
-  Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014)
-  Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15(1), 1929–1958 (2014)
-  Tartaglione, E., Lepsøy, S., Fiandrotti, A., Francini, G.: Learning sparse neural networks via sensitivity-driven regularization. In: Advances in Neural Information Processing Systems. pp. 3882–3892 (2018)
-  Tikhonov, A.N.: On the stability of inverse problems. In: Dokl. Akad. Nauk SSSR. vol. 39, pp. 195–198 (1943)
-  Wager, S., Wang, S., Liang, P.S.: Dropout training as adaptive regularization. In: Advances in neural information processing systems. pp. 351–359 (2013)
-  Wan, L., Zeiler, M., Zhang, S., Le Cun, Y., Fergus, R.: Regularization of neural networks using dropconnect. In: International conference on machine learning. pp. 1058–1066 (2013)
-  Weigend, A.S., Rumelhart, D.E., Huberman, B.A.: Back-propagation, weight-elimination and time series prediction. In: Connectionist models, pp. 105–116. Elsevier (1991)
-  Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms (2017)