A Multilevel Approach to Training

06/28/2020
by   Vanessa Braglia, et al.
0

We propose a novel training method based on nonlinear multilevel minimization techniques, commonly used for solving discretized large scale partial differential equations. Our multilevel training method constructs a multilevel hierarchy by reducing the number of samples. The training of the original model is then enhanced by internally training surrogate models constructed with fewer samples. We construct the surrogate models using first-order consistency approach. This gives rise to surrogate models, whose gradients are stochastic estimators of the full gradient, but with reduced variance compared to standard stochastic gradient estimators. We illustrate the convergence behavior of the proposed multilevel method to machine learning applications based on logistic regression. A comparison with subsampled Newton's and variance reduction methods demonstrate the efficiency of our multilevel method.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/19/2021

Multilevel Picard approximations for McKean-Vlasov stochastic differential equations

In the literatur there exist approximation methods for McKean-Vlasov sto...
research
01/19/2023

Multi-output multilevel best linear unbiased estimators via semidefinite programming

Multifidelity forward uncertainty quantification (UQ) problems often inv...
research
08/19/2021

Using Multilevel Circulant Matrix Approximate to Speed Up Kernel Logistic Regression

Kernel logistic regression (KLR) is a classical nonlinear classifier in ...
research
04/13/2020

Multilevel Minimization for Deep Residual Networks

We present a new multilevel minimization framework for the training of d...
research
03/09/2020

Multilevel spectral coarsening for graph Laplacian problems with application to reservoir simulation

We extend previously developed two-level coarsening procedures for graph...
research
11/11/2022

Multilevel-in-Layer Training for Deep Neural Network Regression

A common challenge in regression is that for many problems, the degrees ...
research
05/28/2021

A Gradient Method for Multilevel Optimization

Although application examples of multilevel optimization have already be...

Please sign up or login with your details

Forgot password? Click here to reset