Log In Sign Up

Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis

by   Ayaka Sakata, et al.

We analyze linear regression problem with a nonconvex regularization called smoothly clipped absolute deviation (SCAD) under overcomplete Gaussian basis for Gaussian random data. We develop a message passing algorithm SCAD-AMP and analytically show that the stability condition is corresponding to the AT condition in spin glass literature. As asymptotic analysis, we show the correspondence between density evolution of SCAD-AMP and replica symmetric solution. Numerical experiments confirm that for sufficiently large system size, SCAD-AMP achieves the optimal performance predicted by replica method. From replica analysis, phase transition between replica symmetric (RS) and replica symmetry breaking (RSB) region is found in the parameter space of SCAD. The appearance of RS region for nonconvex penalty is a great advantage which indicate the region of smooth landscape of the optimization problem. Furthermore, we analytically show that the statistical representation performance of SCAD penalty is improved compared with ℓ_1-based methods, and the minimum representation error under RS assumption is obtained at the edge of RS/RSB phase. The correspondence between the convergence of the existing coordinate descent algorithm and RS/RSB transition is also indicated.


The planted XY model: thermodynamics and inference

In this paper we study a fully connected planted spin glass named the pl...

Reward Shaping Using Convolutional Neural Network

In this paper, we propose Value Iteration Network for Reward Shaping (VI...

Perfect reconstruction of sparse signals with piecewise continuous nonconvex penalties and nonconvexity control

We consider compressed sensing formulated as a minimization problem of n...

Cross validation in sparse linear regression with piecewise continuous nonconvex penalties and its acceleration

We investigate the signal reconstruction performance of sparse linear re...

Estimator of Prediction Error Based on Approximate Message Passing for Penalized Linear Regression

We propose an estimator of prediction error using an approximate message...

Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses

Revealing hidden features in unlabeled data is called unsupervised featu...

Large-deviation Properties of Linear-programming Computational Hardness of the Vertex Cover Problem

The distribution of the computational cost of linear-programming (LP) re...