Efficient Relaxed Gradient Support Pursuit for Sparsity Constrained Non-convex Optimization

12/02/2019
by   Fanhua Shang, et al.
0

Large-scale non-convex sparsity-constrained problems have recently gained extensive attention. Most existing deterministic optimization methods (e.g., GraSP) are not suitable for large-scale and high-dimensional problems, and thus stochastic optimization methods with hard thresholding (e.g., SVRGHT) become more attractive. Inspired by GraSP, this paper proposes a new general relaxed gradient support pursuit (RGraSP) framework, in which the sub-algorithm only requires to satisfy a slack descent condition. We also design two specific semi-stochastic gradient hard thresholding algorithms. In particular, our algorithms have much less hard thresholding operations than SVRGHT, and their average per-iteration cost is much lower (i.e., O(d) vs. O(d log(d)) for SVRGHT), which leads to faster convergence. Our experimental results on both synthetic and real-world datasets show that our algorithms are superior to the state-of-the-art gradient hard thresholding methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/22/2013

Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization

Hard Thresholding Pursuit (HTP) is an iterative greedy selection procedu...
research
05/09/2019

Stochastic Iterative Hard Thresholding for Graph-structured Sparsity Optimization

Stochastic optimization algorithms update models with cheap per-iteratio...
research
10/11/2022

Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivity

ℓ_0 constrained optimization is prevalent in machine learning, particula...
research
02/19/2016

Structured Sparse Regression via Greedy Hard-Thresholding

Several learning applications require solving high-dimensional regressio...
research
04/11/2022

Iterative Hard Thresholding with Adaptive Regularization: Sparser Solutions Without Sacrificing Runtime

We propose a simple modification to the iterative hard thresholding (IHT...
research
01/26/2023

Learning Large Scale Sparse Models

In this work, we consider learning sparse models in large scale settings...
research
04/11/2012

Least Absolute Gradient Selector: Statistical Regression via Pseudo-Hard Thresholding

Variable selection in linear models plays a pivotal role in modern stati...

Please sign up or login with your details

Forgot password? Click here to reset