PAC-learning gains of Turing machines over circuits and neural networks

03/23/2021
by   Brieuc Pinon, et al.
0

A caveat to many applications of the current Deep Learning approach is the need for large-scale data. One improvement suggested by Kolmogorov Complexity results is to apply the minimum description length principle with computationally universal models. We study the potential gains in sample efficiency that this approach can bring in principle. We use polynomial-time Turing machines to represent computationally universal models and Boolean circuits to represent Artificial Neural Networks (ANNs) acting on finite-precision digits. Our analysis unravels direct links between our question and Computational Complexity results. We provide lower and upper bounds on the potential gains in sample efficiency between the MDL applied with Turing machines instead of ANNs. Our bounds depend on the bit-size of the input of the Boolean function to be learned. Furthermore, we highlight close relationships between classical open problems in Circuit Complexity and the tightness of these.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/15/2020

On Partial Differential Encodings, with Application to Boolean Circuits

The present work argues that strong arithmetic circuit lower bounds yiel...
research
07/28/2021

Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers

A common lens to theoretically study neural net architectures is to anal...
research
08/06/2018

Revisiting the simulation of quantum Turing machines by quantum circuits

Yao (1993) proved that quantum Turing machines and uniformly generated q...
research
11/08/2015

(Yet) Another Theoretical Model of Thinking

This paper presents a theoretical, idealized model of the thinking proce...
research
09/18/2021

Separating Circuits : Switching Lemmas and Random Restrictions

This was submitted as a final project for CS254B, taught by Li Yang Tan ...
research
09/06/2019

Mapping finite state machines to zk-SNARKS Using Category Theory

We provide a categorical procedure to turn graphs corresponding to state...
research
09/02/2020

Circuit Satisfiability Problem for circuits of small complexity

The following problem is considered. A Turing machine M, that accepts a ...

Please sign up or login with your details

Forgot password? Click here to reset