Making Predictive Coding Networks Generative

10/26/2019
by   Jeff Orchard, et al.
0

Predictive coding (PC) networks are a biologically interesting class of neural networks. Their layered hierarchy mimics the reciprocal connectivity pattern observed in the mammalian cortex, and they can be trained using local learning rules that approximate backpropagation [Bogacz, 2017]. However, despite having feedback connections that enable information to flow down the network hierarchy, discriminative PC networks are not generative. Clamping the output class and running the network to equilibrium yields an input sample that typically does not resemble the training input. This paper studies this phenomenon, and proposes a simple solution that promotes the generation of input samples that resemble the training inputs. Simple decay, a technique already in wide use in neural networks, pushes the PC network toward a unique minimum 2-norm solution, and that unique solution provably (for linear networks) matches the training inputs. The method also vastly improves the samples generated for nonlinear networks, as we demonstrate on MNIST.

READ FULL TEXT

page 6

page 13

research
06/20/2021

On the relationship between predictive coding and backpropagation

Artificial neural networks are often interpreted as abstract models of b...
research
05/30/2019

Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections

Quantifying and measuring uncertainty in deep neural networks, despite r...
research
11/12/2021

Predictive coding, precision and natural gradients

There is an increasing convergence between biologically plausible comput...
research
06/10/2014

Generative Adversarial Networks

We propose a new framework for estimating generative models via an adver...
research
08/15/2022

Preventing Deterioration of Classification Accuracy in Predictive Coding Networks

Predictive Coding Networks (PCNs) aim to learn a generative model of the...
research
11/21/2017

Variational Probability Flow for Biologically Plausible Training of Deep Neural Networks

The quest for biologically plausible deep learning is driven, not just b...
research
03/02/2006

The Basic Kak Neural Network with Complex Inputs

The Kak family of neural networks is able to learn patterns quickly, and...

Please sign up or login with your details

Forgot password? Click here to reset