DeepAI AI Chat
Log In Sign Up

Learning Sparse Deep Feedforward Networks via Tree Skeleton Expansion

by   Zhourong Chen, et al.
The Hong Kong University of Science and Technology

Despite the popularity of deep learning, structure learning for deep models remains a relatively under-explored area. In contrast, structure learning has been studied extensively for probabilistic graphical models (PGMs). In particular, an efficient algorithm has been developed for learning a class of tree-structured PGMs called hierarchical latent tree models (HLTMs), where there is a layer of observed variables at the bottom and multiple layers of latent variables on top. In this paper, we propose a simple method for learning the structures of feedforward neural networks (FNNs) based on HLTMs. The idea is to expand the connections in the tree skeletons from HLTMs and to use the resulting structures for FNNs. An important characteristic of FNN structures learned this way is that they are sparse. We present extensive empirical results to show that, compared with standard FNNs tuned-manually, sparse FNNs learned by our method achieve better or comparable classification performance with much fewer parameters. They are also more interpretable.


page 1

page 2

page 3

page 4


Building Sparse Deep Feedforward Networks using Tree Receptive Fields

Sparse connectivity is an important factor behind the success of convolu...

From Boltzmann Machines to Neural Networks and Back Again

Graphical models are powerful tools for modeling high-dimensional data, ...

On the Effectiveness of Interpretable Feedforward Neural Network

Deep learning models have achieved state-of-the-art performance in many ...

Nonparametric Latent Tree Graphical Models: Inference, Estimation, and Structure Learning

Tree structured graphical models are powerful at expressing long range o...

Probabilistic Discriminative Learning with Layered Graphical Models

Probabilistic graphical models are traditionally known for their success...

Strengthening Probabilistic Graphical Models: The Purge-and-merge Algorithm

Probabilistic graphical models (PGMs) are powerful tools for solving sys...