Optimisation of Overparametrized Sum-Product Networks

05/20/2019
by   Martin Trapp, et al.
0

It seems to be a pearl of conventional wisdom that parameter learning in deep sum-product networks is surprisingly fast compared to shallow mixture models. This paper examines the effects of overparameterization in sum-product networks on the speed of parameter optimisation. Using theoretical analysis and empirical experiments, we show that deep sum-product networks exhibit an implicit acceleration compared to their shallow counterpart. In fact, gradient-based optimisation in deep sum-product networks is equal to gradient ascend with adaptive and time-varying learning rates and additional momentum terms.

READ FULL TEXT
10/10/2021

Fitting large mixture models using stochastic component selection

Traditional methods for unsupervised learning of finite mixture models r...
10/19/2018

Leveraging Product as an Activation Function in Deep Networks

Product unit neural networks (PUNNs) are powerful representational model...
11/09/2018

Deep Compression of Sum-Product Networks on Tensor Networks

Sum-product networks (SPNs) represent an emerging class of neural networ...
01/19/2017

Online Structure Learning for Sum-Product Networks with Gaussian Leaves

Sum-product networks have recently emerged as an attractive representati...
01/03/2016

A Unified Approach for Learning the Parameters of Sum-Product Networks

We present a unified approach for learning the parameters of Sum-Product...
04/15/2021

Fast Private Parameter Learning and Evaluation for Sum-Product Networks

A sum-product network (SPN) is a graphical model that allows several typ...
10/19/2021

Explaining Deep Tractable Probabilistic Models: The sum-product network case

We consider the problem of explaining a tractable deep probabilistic mod...