DeepAI
Log In Sign Up

Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis

11/08/2017
by   Ayaka Sakata, et al.
0

We analyze linear regression problem with a nonconvex regularization called smoothly clipped absolute deviation (SCAD) under overcomplete Gaussian basis for Gaussian random data. We develop a message passing algorithm SCAD-AMP and analytically show that the stability condition is corresponding to the AT condition in spin glass literature. As asymptotic analysis, we show the correspondence between density evolution of SCAD-AMP and replica symmetric solution. Numerical experiments confirm that for sufficiently large system size, SCAD-AMP achieves the optimal performance predicted by replica method. From replica analysis, phase transition between replica symmetric (RS) and replica symmetry breaking (RSB) region is found in the parameter space of SCAD. The appearance of RS region for nonconvex penalty is a great advantage which indicate the region of smooth landscape of the optimization problem. Furthermore, we analytically show that the statistical representation performance of SCAD penalty is improved compared with ℓ_1-based methods, and the minimum representation error under RS assumption is obtained at the edge of RS/RSB phase. The correspondence between the convergence of the existing coordinate descent algorithm and RS/RSB transition is also indicated.

READ FULL TEXT
08/12/2022

The planted XY model: thermodynamics and inference

In this paper we study a fully connected planted spin glass named the pl...
10/30/2022

Reward Shaping Using Convolutional Neural Network

In this paper, we propose Value Iteration Network for Reward Shaping (VI...
02/20/2019

Perfect reconstruction of sparse signals with piecewise continuous nonconvex penalties and nonconvexity control

We consider compressed sensing formulated as a minimization problem of n...
02/27/2019

Cross validation in sparse linear regression with piecewise continuous nonconvex penalties and its acceleration

We investigate the signal reconstruction performance of sparse linear re...
02/20/2018

Estimator of Prediction Error Based on Approximate Message Passing for Penalized Linear Regression

We propose an estimator of prediction error using an approximate message...
12/06/2016

Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses

Revealing hidden features in unlabeled data is called unsupervised featu...
02/07/2018

Large-deviation Properties of Linear-programming Computational Hardness of the Vertex Cover Problem

The distribution of the computational cost of linear-programming (LP) re...