Active online learning in the binary perceptron problem

02/21/2019
by   Hai-Jun Zhou, et al.
0

The binary perceptron is the simplest artificial neural network formed by N input units and one output unit, with the neural states and the synaptic weights all restricted to ± 1 values. The task in the teacher--student scenario is to infer the hidden weight vector by training on a set of labeled patterns. Previous efforts on the passive learning mode have shown that learning from independent random patterns is quite inefficient. Here we consider the active online learning mode in which the student designs every new Ising training pattern. We demonstrate that it is mathematically possible to achieve perfect (error-free) inference using only N designed training patterns, but this is computationally unfeasible for large systems. We then investigate two Bayesian statistical designing protocols, which require 2.3 N and 1.9 N training patterns, respectively, to achieve error-free inference. If the training patterns are instead designed through deductive reasoning, perfect inference is achieved using N+_2N samples. The performance gap between Bayesian and deductive designing strategies may be shortened in future work by taking into account the possibility of ergodicity breaking in the version space of the binary perceptron.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/19/2020

A new role for circuit expansion for learning in neural networks

Many sensory pathways in the brain rely on sparsely active populations o...
research
03/24/2023

Online Learning for the Random Feature Model in the Student-Teacher Framework

Deep neural networks are widely used prediction algorithms whose perform...
research
09/20/2020

Expectation propagation for the diluted Bayesian classifier

Efficient feature selection from high-dimensional datasets is a very imp...
research
09/03/2020

Bayesian Perceptron: Towards fully Bayesian Neural Networks

Artificial neural networks (NNs) have become the de facto standard in ma...
research
03/22/2022

Learning curves for the multi-class teacher-student perceptron

One of the most classical results in high-dimensional learning theory pr...
research
11/26/2021

Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems

Binary perceptron is a fundamental model of supervised learning for the ...
research
10/22/2019

From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks

We develop a statistical mechanical approach based on the replica method...

Please sign up or login with your details

Forgot password? Click here to reset