DeepAI AI Chat
Log In Sign Up

Distribution-Dependent Sample Complexity of Large Margin Learning

04/05/2012
by   Sivan Sabato, et al.
Microsoft
Hebrew University of Jerusalem
Toyota Technological Institute at Chicago
0

We obtain a tight distribution-specific characterization of the sample complexity of large-margin classification with L2 regularization: We introduce the margin-adapted dimension, which is a simple function of the second order statistics of the data distribution, and show distribution-specific upper and lower bounds on the sample complexity, both governed by the margin-adapted dimension of the data distribution. The upper bounds are universal, and the lower bounds hold for the rich family of sub-Gaussian distributions with independent features. We conclude that this new quantity tightly characterizes the true sample complexity of large-margin classification. To prove the lower bound, we develop several new tools of independent interest. These include new connections between shattering and hardness of learning, new properties of shattering with linear classifiers, and a new lower bound on the smallest eigenvalue of a random Gram matrix generated by sub-Gaussian variables. Our results can be used to quantitatively compare large margin learning to other learning rules, and to improve the effectiveness of methods that use sample complexity bounds, such as active learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/23/2010

Tight Sample Complexity of Large-Margin Learning

We obtain a tight distribution-specific characterization of the sample c...
02/04/2020

Efficient, Noise-Tolerant, and Private Learning via Boosting

We introduce a simple framework for designing private boosting algorithm...
06/20/2014

Noise-adaptive Margin-based Active Learning and Lower Bounds under Tsybakov Noise Condition

We present a simple noise-robust margin-based active learning algorithm ...
05/11/2015

Sample complexity of learning Mahalanobis distance metrics

Metric learning seeks a transformation of the feature space that enhance...
02/20/2021

Generalization bounds for graph convolutional neural networks via Rademacher complexity

This paper aims at studying the sample complexity of graph convolutional...
06/14/2019

A Distribution Dependent and Independent Complexity Analysis of Manifold Regularization

Manifold regularization is a commonly used technique in semi-supervised ...
03/07/2023

Margin theory for the scenario-based approach to robust optimization in high dimension

This paper deals with the scenario approach to robust optimization. This...