Fast Gradient Methods with Alignment for Symmetric Linear Systems without Using Cauchy Step

09/03/2019
by   Qinmeng Zou, et al.
0

The performance of gradient methods has been considerably improved by the introduction of delayed parameters. After two and a half decades, the revealing of second-order information has recently given rise to the Cauchy-based methods with alignment, which reduce asymptotically the search spaces in smaller and smaller dimensions. They are generally considered as the state of the art of gradient methods. This paper reveals the spectral properties of minimal gradient and asymptotically optimal steps, and then suggests three fast methods with alignment without using the Cauchy step. The convergence results are provided, and numerical experiments show that the new methods provide competitive and more stable alternatives to the classical Cauchy-based methods. In particular, alignment gradient methods present advantages over the Krylov subspace methods in some situations, which makes them attractive in practice.

READ FULL TEXT
research
10/26/2020

Convergence Acceleration via Chebyshev Step: Plausible Interpretation of Deep-Unfolded Gradient Descent

Deep unfolding is a promising deep-learning technique, whose network arc...
research
06/17/2023

Gradient-type subspace iteration methods for the symmetric eigenvalue problem

This paper explores variants of the subspace iteration algorithm for com...
research
02/20/2020

Second Order Optimization Made Practical

Optimization in machine learning, both theoretical and applied, is prese...
research
08/23/2022

A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information

In this paper, we consider to improve the stochastic variance reduce gra...
research
02/05/2021

In-Loop Meta-Learning with Gradient-Alignment Reward

At the heart of the standard deep learning training loop is a greedy gra...
research
05/30/2023

Flexible Enlarged Conjugate Gradient Methods

Enlarged Krylov subspace methods and their s-step versions were introduc...
research
07/08/2014

Regression-Based Image Alignment for General Object Categories

Gradient-descent methods have exhibited fast and reliable performance fo...

Please sign up or login with your details

Forgot password? Click here to reset