DeepAI AI Chat
Log In Sign Up

Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithms

by   Surbhi Goel, et al.
Harvard University

Neural Networks (NNs) struggle to efficiently learn certain problems, such as parity problems, even when there are simple learning algorithms for those problems. Can NNs discover learning algorithms on their own? We exhibit a NN architecture that, in polynomial time, learns as well as any efficient learning algorithm describable by a constant-sized learning algorithm. For example, on parity problems, the NN learns as well as row reduction, an efficient algorithm that can be succinctly described. Our architecture combines both recurrent weight-sharing between layers and convolutional weight-sharing to reduce the number of parameters down to a constant, even though the network itself may have trillions of nodes. While in practice the constants in our analysis are too large to be directly meaningful, our work suggests that the synergy of Recurrent and Convolutional NNs (RCNNs) may be more powerful than either alone.


page 1

page 2

page 3

page 4


Downsampling for Testing and Learning in Product Distributions

We study the domain reduction problem of eliminating dependence on n fro...

SGD Learns the Conjugate Kernel Class of the Network

We show that the standard stochastic gradient decent (SGD) algorithm is ...

Learning Depth-Three Neural Networks in Polynomial Time

We give a polynomial-time algorithm for learning neural networks with on...

Hybrid-Layers Neural Network Architectures for Modeling the Self-Interference in Full-Duplex Systems

Full-duplex (FD) systems have been introduced to provide high data rates...

Bounding the Complexity of Formally Verifying Neural Networks: A Geometric Approach

In this paper, we consider the computational complexity of formally veri...

WSNet: Compact and Efficient Networks with Weight Sampling

We present a new approach and a novel architecture, termed WSNet, for le...

Priority Promotion with Parysian Flair

We develop an algorithm that combines the advantages of priority promoti...