A Generalized Weighted Optimization Method for Computational Learning and Inversion

by   Kui Ren, et al.

The generalization capacity of various machine learning models exhibits different phenomena in the under- and over-parameterized regimes. In this paper, we focus on regression models such as feature regression and kernel regression and analyze a generalized weighted least-squares optimization method for computational learning and inversion with noisy data. The highlight of the proposed framework is that we allow weighting in both the parameter space and the data space. The weighting scheme encodes both a priori knowledge on the object to be learned and a strategy to weight the contribution of different data points in the loss function. Here, we characterize the impact of the weighting scheme on the generalization error of the learning method, where we derive explicit generalization errors for the random Fourier feature model in both the under- and over-parameterized regimes. For more general feature maps, error bounds are provided based on the singular values of the feature matrix. We demonstrate that appropriate weighting from prior knowledge can improve the generalization capability of the learned model.



There are no comments yet.


page 1

page 2

page 3

page 4


Understanding the role of importance weighting for deep learning

The recent paper by Byrd Lipton (2019), based on empirical observati...

Mean Absolute Percentage Error for regression models

We study in this paper the consequences of using the Mean Absolute Perce...

Generalization Error of Generalized Linear Models in High Dimensions

At the heart of machine learning lies the question of generalizability o...

Weighted Optimization: better generalization by smoother interpolation

We provide a rigorous analysis of how implicit bias towards smooth inter...

A generalized meta-loss function for distillation and learning using privileged information for classification and regression

Learning using privileged information and distillation are powerful mach...

A Precise Performance Analysis of Learning with Random Features

We study the problem of learning an unknown function using random featur...

Which Samples Should be Learned First: Easy or Hard?

An effective weighting scheme for training samples is essential for lear...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.