Efficient Learning with a Family of Nonconvex Regularizers by Redistributing Nonconvexity

06/13/2016
by   Quanming Yao, et al.
0

The use of convex regularizers allows for easy optimization, though they often produce biased estimation and inferior prediction performance. Recently, nonconvex regularizers have attracted a lot of attention and outperformed convex ones. However, the resultant optimization problem is much harder. In this paper, for a large class of nonconvex regularizers, we propose to move the nonconvexity from the regularizer to the loss. The nonconvex regularizer is then transformed to a familiar convex regularizer, while the resultant loss function can still be guaranteed to be smooth. Learning with the convexified regularizer can be performed by existing efficient algorithms originally designed for convex regularizers (such as the proximal algorithm, Frank-Wolfe algorithm, alternating direction method of multipliers and stochastic gradient descent). Extensions are made when the convexified regularizer does not have closed-form proximal step, and when the loss function is nonconvex, nonsmooth. Extensive experiments on a variety of machine learning application scenarios show that optimizing the transformed problem is much faster than running the state-of-the-art on the original problem.

READ FULL TEXT
research
05/23/2016

Fast Stochastic Methods for Nonsmooth Nonconvex Optimization

We analyze stochastic algorithms for optimizing nonconvex, nonsmooth fin...
research
06/27/2012

Sparse Support Vector Infinite Push

In this paper, we address the problem of embedded feature selection for ...
research
06/01/2016

Self-Paced Learning: an Implicit Regularization Perspective

Self-paced learning (SPL) mimics the cognitive mechanism of humans and a...
research
02/27/2017

Multi-Label Segmentation via Residual-Driven Adaptive Regularization

We present a variational multi-label segmentation algorithm based on a r...
research
12/27/2022

Optimal Convex and Nonconvex Regularizers for a Data Source

In optimization-based approaches to inverse problems and to statistical ...
research
10/12/2022

Optimizing Evaluation Metrics for Multi-Task Learning via the Alternating Direction Method of Multipliers

Multi-task learning (MTL) aims to improve the generalization performance...
research
10/01/2021

Update in Unit Gradient

In Machine Learning, optimization mostly has been done by using a gradie...

Please sign up or login with your details

Forgot password? Click here to reset