Optimal Learning

03/30/2022
by   Peter Binev, et al.
0

This paper studies the problem of learning an unknown function f from given data about f. The learning problem is to give an approximation f̂ to f that predicts the values of f away from the data. There are numerous settings for this learning problem depending on (i) what additional information we have about f (known as a model class assumption), (ii) how we measure the accuracy of how well f̂ predicts f, (iii) what is known about the data and data sites, (iv) whether the data observations are polluted by noise. A mathematical description of the optimal performance possible (the smallest possible error of recovery) is known in the presence of a model class assumption. Under standard model class assumptions, it is shown in this paper that a near optimal f̂ can be found by solving a certain discrete over-parameterized optimization problem with a penalty term. Here, near optimal means that the error is bounded by a fixed constant times the optimal error. This explains the advantage of over-parameterization which is commonly used in modern machine learning. The main results of this paper prove that over-parameterized learning with an appropriate loss function gives a near optimal approximation f̂ of the function f from which the data is collected. Quantitative bounds are given for how much over-parameterization needs to be employed and how the penalization needs to be scaled in order to guarantee a near optimal recovery of f. An extension of these results to the case where the data is polluted by additive deterministic noise is also given.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/22/2021

Simple and near-optimal algorithms for hidden stratification and multi-group learning

Multi-group agnostic learning is a formal learning criterion that is con...
research
03/30/2016

Adaptive Maximization of Pointwise Submodular Functions With Budget Constraint

We study the worst-case adaptive optimization problem with budget constr...
research
07/22/2021

Learning Sparse Fixed-Structure Gaussian Bayesian Networks

Gaussian Bayesian networks (a.k.a. linear Gaussian structural equation m...
research
11/02/2020

Complexity of near-optimal robust versions of multilevel optimization problems

Near-optimality robustness extends multilevel optimization with a limite...
research
01/18/2023

Near-Optimal Estimation of Linear Functionals with Log-Concave Observation Errors

This note addresses the question of optimally estimating a linear functi...
research
08/13/2022

A Near-Optimal Algorithm for Univariate Zeroth-Order Budget Convex Optimization

This paper studies a natural generalization of the problem of minimizing...
research
10/10/2020

Noise in Classification

This chapter considers the computational and statistical aspects of lear...

Please sign up or login with your details

Forgot password? Click here to reset