Nonconvex Optimization via MM Algorithms: Convergence Theory

06/05/2021
by   Kenneth Lange, et al.
0

The majorization-minimization (MM) principle is an extremely general framework for deriving optimization algorithms. It includes the expectation-maximization (EM) algorithm, proximal gradient algorithm, concave-convex procedure, quadratic lower bound algorithm, and proximal distance algorithm as special cases. Besides numerous applications in statistics, optimization, and imaging, the MM principle finds wide applications in large scale machine learning problems such as matrix completion, discriminant analysis, and nonnegative matrix factorizations. When applied to nonconvex optimization problems, MM algorithms enjoy the advantages of convexifying the objective function, separating variables, numerical stability, and ease of implementation. However, compared to the large body of literature on other optimization algorithms, the convergence analysis of MM algorithms is scattered and problem specific. This survey presents a unified treatment of the convergence of MM algorithms. With modern applications in mind, the results encompass non-smooth objective functions and non-asymptotic analysis.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/25/2015

Generalized Majorization-Minimization

Non-convex optimization is ubiquitous in machine learning. The Majorizat...
research
02/10/2021

A Framework of Inertial Alternating Direction Method of Multipliers for Non-Convex Non-Smooth Optimization

In this paper, we propose an algorithmic framework dubbed inertial alter...
research
01/15/2022

Quasi-Newton acceleration of EM and MM algorithms via Broyden's method

The principle of majorization-minimization (MM) provides a general frame...
research
11/12/2016

An Introduction to MM Algorithms for Machine Learning and Statistical

MM (majorization--minimization) algorithms are an increasingly popular t...
research
02/24/2023

Asymptotic convergence of iterative optimization algorithms

This paper introduces a general framework for iterative optimization alg...
research
11/20/2018

Sampling Can Be Faster Than Optimization

Optimization algorithms and Monte Carlo sampling algorithms have provide...
research
11/16/2012

Distance Majorization and Its Applications

The problem of minimizing a continuously differentiable convex function ...

Please sign up or login with your details

Forgot password? Click here to reset