An algorithmic view of ℓ_2 regularization and some path-following algorithms

07/07/2021
by   Yunzhang Zhu, et al.
0

We establish an equivalence between the ℓ_2-regularized solution path for a convex loss function, and the solution of an ordinary differentiable equation (ODE). Importantly, this equivalence reveals that the solution path can be viewed as the flow of a hybrid of gradient descent and Newton method applying to the empirical loss, which is similar to a widely used optimization technique called trust region method. This provides an interesting algorithmic view of ℓ_2 regularization, and is in contrast to the conventional view that the ℓ_2 regularization solution path is similar to the gradient flow of the empirical loss.New path-following algorithms based on homotopy methods and numerical ODE solvers are proposed to numerically approximate the solution path. In particular, we consider respectively Newton method and gradient descent method as the basis algorithm for the homotopy method, and establish their approximation error rates over the solution path. Importantly, our theory suggests novel schemes to choose grid points that guarantee an arbitrarily small suboptimality for the solution path. In terms of computational cost, we prove that in order to achieve an ϵ-suboptimality for the entire solution path, the number of Newton steps required for the Newton method is 𝒪(ϵ^-1/2), while the number of gradient steps required for the gradient descent method is 𝒪(ϵ^-1ln(ϵ^-1)). Finally, we use ℓ_2-regularized logistic regression as an illustrating example to demonstrate the effectiveness of the proposed path-following algorithms.

READ FULL TEXT

page 22

page 23

page 24

research
09/08/2019

On the connections between algorithmic regularization and penalization for convex losses

In this work we establish the equivalence of algorithmic regularization ...
research
05/12/2021

An efficient projection neural network for ℓ_1-regularized logistic regression

ℓ_1 regularization has been used for logistic regression to circumvent t...
research
06/19/2020

Gradient descent follows the regularization path for general losses

Recent work across many machine learning disciplines has highlighted tha...
research
05/18/2023

Modified Gauss-Newton Algorithms under Noise

Gauss-Newton methods and their stochastic version have been widely used ...
research
01/06/2019

Dynamic Visualization and Fast Computation for Convex Clustering via Algorithmic Regularization

Convex clustering is a promising new approach to the classical problem o...
research
01/06/2019

Dynamic Visualization and Fast Computation for Convex Clustering and Bi-Clustering

Convex clustering is a promising new approach to the classical problem o...
research
07/13/2022

Iterative Linear Quadratic Optimization for Nonlinear Control: Differentiable Programming Algorithmic Templates

We present the implementation of nonlinear control algorithms based on l...

Please sign up or login with your details

Forgot password? Click here to reset