Anderson Acceleration as a Krylov Method with Application to Asymptotic Convergence Analysis

by   Hans De Sterck, et al.

Anderson acceleration is widely used for accelerating the convergence of fixed-point methods x_k+1=q(x_k), x_k ∈ℝ^n. We consider the case of linear fixed-point methods x_k+1=M x_k+b and obtain polynomial residual update formulas for AA(m), i.e., Anderson acceleration with window size m. We find that the standard AA(m) method with initial iterates x_k, k=0, …, m defined recursively using AA(k), is a Krylov space method. This immediately implies that k iterations of AA(m) cannot produce a smaller residual than k iterations of GMRES without restart (but without implying anything about the relative convergence speed of (windowed) AA(m) versus restarted GMRES(m)). We introduce the notion of multi-Krylov method and show that AA(m) with general initial iterates {x_0, …, x_m} is a multi-Krylov method. We find that the AA(m) residual polynomials observe a periodic memory effect where increasing powers of the error iteration matrix M act on the initial residual as the iteration number increases. We derive several further results based on these polynomial residual update formulas, including orthogonality relations, a lower bound on the AA(1) acceleration coefficient β_k, and explicit nonlinear recursions for the AA(1) residuals and residual polynomials that do not include the acceleration coefficient β_k. We apply these results to study the influence of the initial guess on the asymptotic convergence factor of AA(1).



There are no comments yet.


page 15

page 16

page 20


On the Asymptotic Linear Convergence Speed of Anderson Acceleration, Nesterov Acceleration, and Nonlinear GMRES

We consider nonlinear convergence acceleration methods for fixed-point i...

Linear Asymptotic Convergence of Anderson Acceleration: Fixed-Point Analysis

We study the asymptotic convergence of AA(m), i.e., Anderson acceleratio...

Anderson Acceleration Using the H^-s Norm

Anderson acceleration (AA) is a technique for accelerating the convergen...

Acceleration of nonlinear solvers for natural convection problems

This paper develops an efficient and robust solution technique for the s...

Stable Anderson Acceleration for Deep Learning

Anderson acceleration (AA) is an extrapolation technique designed to spe...

Convergence analysis of adaptive DIIS algorithms with application to electronic ground state calculations

This paper deals with a general class of algorithms for the solution of ...

Secant acceleration of sequential residual methods for solving large-scale nonlinear systems of equations

Sequential Residual Methods try to solve nonlinear systems of equations ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.