Anderson Acceleration as a Krylov Method with Application to Asymptotic Convergence Analysis

09/29/2021
by   Hans De Sterck, et al.
0

Anderson acceleration is widely used for accelerating the convergence of fixed-point methods x_k+1=q(x_k), x_k ∈ℝ^n. We consider the case of linear fixed-point methods x_k+1=M x_k+b and obtain polynomial residual update formulas for AA(m), i.e., Anderson acceleration with window size m. We find that the standard AA(m) method with initial iterates x_k, k=0, …, m defined recursively using AA(k), is a Krylov space method. This immediately implies that k iterations of AA(m) cannot produce a smaller residual than k iterations of GMRES without restart (but without implying anything about the relative convergence speed of (windowed) AA(m) versus restarted GMRES(m)). We introduce the notion of multi-Krylov method and show that AA(m) with general initial iterates {x_0, …, x_m} is a multi-Krylov method. We find that the AA(m) residual polynomials observe a periodic memory effect where increasing powers of the error iteration matrix M act on the initial residual as the iteration number increases. We derive several further results based on these polynomial residual update formulas, including orthogonality relations, a lower bound on the AA(1) acceleration coefficient β_k, and explicit nonlinear recursions for the AA(1) residuals and residual polynomials that do not include the acceleration coefficient β_k. We apply these results to study the influence of the initial guess on the asymptotic convergence factor of AA(1).

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 15

page 16

page 20

07/04/2020

On the Asymptotic Linear Convergence Speed of Anderson Acceleration, Nesterov Acceleration, and Nonlinear GMRES

We consider nonlinear convergence acceleration methods for fixed-point i...
09/29/2021

Linear Asymptotic Convergence of Anderson Acceleration: Fixed-Point Analysis

We study the asymptotic convergence of AA(m), i.e., Anderson acceleratio...
02/10/2020

Anderson Acceleration Using the H^-s Norm

Anderson acceleration (AA) is a technique for accelerating the convergen...
04/14/2020

Acceleration of nonlinear solvers for natural convection problems

This paper develops an efficient and robust solution technique for the s...
10/26/2021

Stable Anderson Acceleration for Deep Learning

Anderson acceleration (AA) is an extrapolation technique designed to spe...
02/28/2020

Convergence analysis of adaptive DIIS algorithms with application to electronic ground state calculations

This paper deals with a general class of algorithms for the solution of ...
12/24/2020

Secant acceleration of sequential residual methods for solving large-scale nonlinear systems of equations

Sequential Residual Methods try to solve nonlinear systems of equations ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.