Feedback Networks

12/30/2016
by   Amir R. Zamir, et al.
0

Currently, the most successful learning models in computer vision are based on learning successive representations followed by a decision layer. This is usually actualized through feedforward multilayer neural networks, e.g. ConvNets, where each layer forms one of such successive representations. However, an alternative that can achieve the same goal is a feedback based approach in which the representation is formed in an iterative manner based on a feedback received from previous iteration's output. We establish that a feedback based approach has several fundamental advantages over feedforward: it enables making early predictions at the query time, its output naturally conforms to a hierarchical structure in the label space (e.g. a taxonomy), and it provides a new basis for Curriculum Learning. We observe that feedback networks develop a considerably different representation compared to feedforward counterparts, in line with the aforementioned advantages. We put forth a general feedback based learning architecture with the endpoint results on par or better than existing feedforward networks with the addition of the above advantages. We also investigate several mechanisms in feedback architectures (e.g. skip connections in time) and design choices (e.g. feedback length). We hope this study offers new perspectives in quest for more natural and practical learning models.

READ FULL TEXT

page 1

page 4

page 6

page 7

research
12/22/2019

Recurrent Feedback Improves Feedforward Representations in Deep Neural Networks

The abundant recurrent horizontal and feedback connections in the primat...
research
07/10/2020

Biological credit assignment through dynamic inversion of feedforward networks

Learning depends on changes in synaptic connections deep inside the brai...
research
12/19/2022

Fixed-Weight Difference Target Propagation

Target Propagation (TP) is a biologically more plausible algorithm than ...
research
12/20/2013

Competitive Learning with Feedforward Supervisory Signal for Pre-trained Multilayered Networks

We propose a novel learning method for multilayered neural networks whic...
research
02/21/2020

Accessing Higher-level Representations in Sequential Transformers with Feedback Memory

Transformers are feedforward networks that can process input tokens in p...
research
04/30/2020

Feedback U-net for Cell Image Segmentation

Human brain is a layered structure, and performs not only a feedforward ...
research
12/22/2016

Highway and Residual Networks learn Unrolled Iterative Estimation

The past year saw the introduction of new architectures such as Highway ...

Please sign up or login with your details

Forgot password? Click here to reset