Average-Case Information Complexity of Learning

11/25/2018
by   Ido Nachum, et al.
0

How many bits of information are revealed by a learning algorithm for a concept class of VC-dimension d? Previous works have shown that even for d=1 the amount of information may be unbounded (tend to ∞ with the universe size). Can it be that all concepts in the class require leaking a large amount of information? We show that typically concepts do not require leakage. There exists a proper learning algorithm that reveals O(d) bits of information for most concepts in the class. This result is a special case of a more general phenomenon we explore. If there is a low information learner when the algorithm knows the underlying distribution on inputs, then there is a learner that reveals little information on an average concept without knowing the distribution on inputs.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/16/2018

A Direct Sum Result for the Information Complexity of Learning

How many bits of information are required to PAC learn a class of hypoth...
research
07/11/2023

MAP- and MLE-Based Teaching

Imagine a learner L who tries to infer a hidden concept from a collectio...
research
02/12/2020

Quantum Boosting

Suppose we have a weak learning algorithm A for a Boolean-valued problem...
research
04/21/2023

The simplicity bubble effect as a zemblanitous phenomenon in learning systems

The ubiquity of Big Data and machine learning in society evinces the nee...
research
10/01/2018

Simple Algorithms for Learning from Random Counterexamples

This work describes two simple and efficient algorithms for exactly lear...
research
02/10/2021

Agnostic Proper Learning of Halfspaces under Gaussian Marginals

We study the problem of agnostically learning halfspaces under the Gauss...
research
06/27/2022

Finite Littlestone Dimension Implies Finite Information Complexity

We prove that every online learnable class of functions of Littlestone d...

Please sign up or login with your details

Forgot password? Click here to reset